mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 21:38:07 +09:00
Compare commits
145
Commits
No files matched your search
@@ -0,0 +1,393 @@
|
||||
# Plan: Admin Job Details Page
|
||||
|
||||
**Created:** 2026-01-09
|
||||
**Branch:** feat/sess-20260109-1738-admin-job-details
|
||||
**Status:** In Progress (Phase 1 Complete)
|
||||
|
||||
## Overview
|
||||
|
||||
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
|
||||
|
||||
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
|
||||
|
||||
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Must Have
|
||||
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
|
||||
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
|
||||
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
|
||||
- [ ] Download links for all existing files (generate signed URLs)
|
||||
- [ ] View worker logs with filtering by level/worker
|
||||
- [ ] Restart job from beginning (reset to PENDING)
|
||||
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
|
||||
|
||||
### Should Have
|
||||
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
|
||||
- [ ] View/edit raw JSON fields (state_data, file_urls)
|
||||
- [ ] Quick actions: send completion email, clear cache, impersonate user
|
||||
|
||||
### Nice to Have
|
||||
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
|
||||
- [ ] Diff view when editing fields
|
||||
- [ ] Audit log of admin changes
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Routing Fix (COMPLETED)
|
||||
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
|
||||
|
||||
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
|
||||
- Uses `useSearchParams()` to detect `?id=` parameter
|
||||
- Shows detail view when `selectedJobId` is present, list view otherwise
|
||||
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
|
||||
- Deleted old `/admin/jobs/detail/` directory
|
||||
|
||||
### Backend Enhancements
|
||||
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
|
||||
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
|
||||
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
|
||||
|
||||
### Frontend Structure
|
||||
The page will have collapsible sections:
|
||||
1. **Header**: Job ID, status badge, artist/title, quick actions
|
||||
2. **Overview Cards**: User, dates, progress, source info
|
||||
3. **Stage Timeline**: Visual breakdown of each stage with times
|
||||
4. **Files**: Downloadable links for all GCS files
|
||||
5. **Logs**: Filterable worker logs
|
||||
6. **Configuration**: Editable job settings
|
||||
7. **Raw Data**: JSON views of state_data, file_urls, full job
|
||||
|
||||
### Job Reset Logic
|
||||
Reset to specific states requires:
|
||||
- Clearing downstream state_data keys
|
||||
- Resetting status
|
||||
- Optionally clearing associated files
|
||||
- Adding timeline event for the reset
|
||||
|
||||
Valid reset targets:
|
||||
| Target State | Clears | Use Case |
|
||||
|--------------|--------|----------|
|
||||
| PENDING | Everything except input | Full restart |
|
||||
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
|
||||
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
|
||||
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
|
||||
|
||||
### Phase 1: Fix Routing (COMPLETED)
|
||||
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
|
||||
2. [x] Combined list and detail views in single component
|
||||
3. [x] Use `useSearchParams()` to detect `?id=` parameter
|
||||
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
|
||||
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
|
||||
6. [x] Verified: All 9 E2E tests pass, build succeeds
|
||||
|
||||
### Phase 2: Backend - File Downloads with Signed URLs
|
||||
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
|
||||
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
|
||||
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
|
||||
10. [ ] Traverse job.file_urls recursively
|
||||
11. [ ] Generate signed URLs for each GCS path
|
||||
12. [ ] Return structured response with file metadata
|
||||
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
|
||||
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
|
||||
|
||||
### Phase 3: Frontend - Enhanced File Display
|
||||
15. [ ] Create "Files" accordion section with grouped downloads
|
||||
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
|
||||
17. [ ] Show file name, type, download button
|
||||
18. [ ] Handle missing/unavailable files gracefully
|
||||
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
|
||||
20. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 4: Backend - Job Update Endpoint
|
||||
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
|
||||
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
|
||||
23. [ ] Define editable fields whitelist
|
||||
24. [ ] Validate field values (e.g., status must be valid enum)
|
||||
25. [ ] Log admin changes for audit trail
|
||||
26. [ ] Verify: Backend tests pass
|
||||
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
|
||||
|
||||
### Phase 5: Frontend - Editable Fields
|
||||
28. [ ] Create "Configuration" section with editable fields
|
||||
29. [ ] Use inline edit pattern (click to edit, enter to save)
|
||||
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
|
||||
31. [ ] Show validation errors inline
|
||||
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
|
||||
33. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 6: Backend - Job Reset Endpoint
|
||||
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
|
||||
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
|
||||
36. [ ] Accept `target_state` parameter
|
||||
37. [ ] Implement reset logic for each target state
|
||||
38. [ ] Clear appropriate state_data keys
|
||||
39. [ ] Optionally trigger workers after reset
|
||||
40. [ ] Verify: Backend tests pass
|
||||
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
|
||||
|
||||
### Phase 7: Frontend - Reset Actions
|
||||
42. [ ] Add "Reset Job" dropdown/menu in header
|
||||
43. [ ] Show available reset targets based on current state
|
||||
44. [ ] Confirm dialog explaining what will be cleared
|
||||
45. [ ] Show success/error toast after reset
|
||||
46. [ ] **E2E Test**: Add reset confirmation dialog test
|
||||
47. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 8: Enhanced Stage Timeline
|
||||
48. [ ] Parse timeline events to calculate stage durations
|
||||
49. [ ] Display stages as cards or timeline items
|
||||
50. [ ] Show duration for each stage
|
||||
51. [ ] Highlight current/blocking stages
|
||||
52. [ ] **E2E Test**: Add timeline display test
|
||||
|
||||
### Phase 9: Final Verification
|
||||
53. [ ] Run full backend test suite: `make test-backend`
|
||||
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
|
||||
55. [ ] Manual production testing checklist (see Testing Strategy)
|
||||
56. [ ] Verify CI passes on PR
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
|
||||
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
|
||||
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
|
||||
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
|
||||
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
|
||||
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Based on `docs/TESTING.md` guidelines, this feature requires:
|
||||
|
||||
### Test Types Required
|
||||
|
||||
| Type | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
|
||||
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
|
||||
| Manual Production E2E | N/A | Final validation on staging/prod |
|
||||
|
||||
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
|
||||
|
||||
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
|
||||
|
||||
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
|
||||
```python
|
||||
def test_get_job_files_returns_signed_urls_for_all_files():
|
||||
"""Test that all files in file_urls get signed URLs."""
|
||||
|
||||
def test_get_job_files_handles_nested_file_urls():
|
||||
"""Test recursive traversal of stems.instrumental_clean, etc."""
|
||||
|
||||
def test_get_job_files_handles_missing_files_gracefully():
|
||||
"""Test files that don't exist in GCS are omitted or marked."""
|
||||
|
||||
def test_get_job_files_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_get_job_files_returns_404_for_missing_job():
|
||||
"""Test invalid job_id returns 404."""
|
||||
```
|
||||
|
||||
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
|
||||
```python
|
||||
def test_update_job_allows_editable_fields():
|
||||
"""Test artist, title, user_email, theme_id can be updated."""
|
||||
|
||||
def test_update_job_rejects_non_editable_fields():
|
||||
"""Test job_id, created_at, etc. cannot be updated."""
|
||||
|
||||
def test_update_job_validates_status_enum():
|
||||
"""Test invalid status value returns 422."""
|
||||
|
||||
def test_update_job_validates_theme_id_exists():
|
||||
"""Test non-existent theme_id returns 422."""
|
||||
|
||||
def test_update_job_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_update_job_returns_updated_job():
|
||||
"""Test response includes updated job data."""
|
||||
```
|
||||
|
||||
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
|
||||
```python
|
||||
def test_reset_to_pending_clears_all_state_data():
|
||||
"""Test full reset clears everything except input."""
|
||||
|
||||
def test_reset_to_audio_selection_clears_search_results():
|
||||
"""Test audio selection reset clears audio_search_results."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_review_data():
|
||||
"""Test review reset clears screens, videos, review data."""
|
||||
|
||||
def test_reset_to_instrumental_selection_clears_finals():
|
||||
"""Test instrumental reset clears encoding results."""
|
||||
|
||||
def test_reset_adds_timeline_event():
|
||||
"""Test reset adds admin action to timeline."""
|
||||
|
||||
def test_reset_invalid_target_returns_422():
|
||||
"""Test invalid target_state returns validation error."""
|
||||
|
||||
def test_reset_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_reset_from_complete_to_pending_works():
|
||||
"""Test can reset completed job back to start."""
|
||||
```
|
||||
|
||||
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
|
||||
|
||||
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
|
||||
|
||||
test.describe('Admin Job Detail Page', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await setAdminAuthToken(page);
|
||||
});
|
||||
|
||||
test('navigates from jobs list to detail page', async ({ page }) => {
|
||||
// Mock jobs list and single job endpoints
|
||||
await page.route('**/api/admin/jobs', route => route.fulfill({
|
||||
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
|
||||
}));
|
||||
await page.route('**/api/jobs/test123', route => route.fulfill({
|
||||
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
|
||||
}));
|
||||
|
||||
await page.goto('/admin/jobs');
|
||||
await page.click('text=test123');
|
||||
await expect(page).toHaveURL('/admin/jobs/test123');
|
||||
await expect(page.getByText('Job test123')).toBeVisible();
|
||||
});
|
||||
|
||||
test('displays all job metadata sections', async ({ page }) => {
|
||||
// Test overview cards, timeline, logs accordion, etc.
|
||||
});
|
||||
|
||||
test('files section shows downloadable links', async ({ page }) => {
|
||||
// Mock files endpoint, verify download buttons appear
|
||||
});
|
||||
|
||||
test('edit field inline and save', async ({ page }) => {
|
||||
// Click edit, change value, press Enter, verify PATCH called
|
||||
});
|
||||
|
||||
test('reset job shows confirmation dialog', async ({ page }) => {
|
||||
// Click reset, verify dialog, confirm, verify POST called
|
||||
});
|
||||
|
||||
test('handles API errors gracefully', async ({ page }) => {
|
||||
// Mock 500 response, verify error toast shown
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test File Organization
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── test_admin_job_detail.py # NEW - all backend tests for this feature
|
||||
├── test_admin_api.py # Existing admin tests (keep separate)
|
||||
└── conftest.py # Shared fixtures
|
||||
|
||||
frontend/e2e/
|
||||
├── fixtures/
|
||||
│ ├── test-helper.ts # Add setAdminAuthToken helper
|
||||
│ └── mock-responses.ts # Add admin job detail mocks
|
||||
└── regression/
|
||||
└── admin-job-detail.spec.ts # NEW - E2E regression tests
|
||||
```
|
||||
|
||||
### Mocking Guidelines
|
||||
|
||||
**Backend - What to mock:**
|
||||
- `StorageService.generate_signed_url()` - return predictable test URLs
|
||||
- `StorageService.file_exists()` - control which files "exist"
|
||||
- Firestore operations via existing `mock_firestore` fixture
|
||||
|
||||
**Backend - What NOT to mock:**
|
||||
- `JobManager` logic - test the real reset/update logic
|
||||
- Validation logic - test real Pydantic validation
|
||||
- Authorization checks - test real `require_admin` dependency
|
||||
|
||||
**Frontend - What to mock:**
|
||||
- All API responses via Playwright route interception
|
||||
- Time (if testing token expiry)
|
||||
|
||||
**Frontend - What NOT to mock:**
|
||||
- React component behavior
|
||||
- User interactions (click, type, etc.)
|
||||
|
||||
### CI Requirements
|
||||
|
||||
This feature will be validated by existing CI checks:
|
||||
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
|
||||
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
|
||||
- `frontend-build` - Must pass (verifies new page compiles)
|
||||
|
||||
### Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Backend tests only (for this feature)
|
||||
poetry run pytest backend/tests/test_admin_job_detail.py -v
|
||||
|
||||
# All backend tests
|
||||
make test-backend
|
||||
|
||||
# Frontend E2E regression (includes new tests)
|
||||
cd frontend && npm run test:e2e
|
||||
|
||||
# Single E2E test file
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
|
||||
|
||||
# With browser visible for debugging
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
|
||||
```
|
||||
|
||||
### Manual Production Testing Checklist
|
||||
|
||||
After deployment, manually verify on production:
|
||||
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
|
||||
- [ ] All file download links work (valid signed URLs)
|
||||
- [ ] Edit artist/title → refresh → changes persisted
|
||||
- [ ] Reset job to pending → job status shows "pending"
|
||||
- [ ] Logs section loads and filters work
|
||||
- [ ] Error states handled (invalid job ID shows friendly error)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
|
||||
- **Recommendation**: Delete it. No external links depend on it.
|
||||
|
||||
2. [ ] Should job reset delete GCS files or just clear state_data references?
|
||||
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
|
||||
|
||||
3. [ ] Should edits be immediately saved or require explicit "Save" button?
|
||||
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
|
||||
2. Backend: New endpoints are additive, won't affect existing functionality
|
||||
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
|
||||
|
||||
## Notes
|
||||
|
||||
- The existing detail page has good foundational code - we're enhancing rather than replacing
|
||||
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
|
||||
- Job reset is powerful - consider adding confirmation dialogs and logging
|
||||
@@ -0,0 +1,321 @@
|
||||
# Plan: Consolidate YouTube Download Paths
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Status:** Planning
|
||||
|
||||
## Problem Statement
|
||||
|
||||
YouTube downloads currently have **3 different code paths** with different behaviors:
|
||||
|
||||
1. **Audio Search → Select YouTube** (`audio_search.py`)
|
||||
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
|
||||
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
|
||||
|
||||
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
|
||||
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
|
||||
- Runs on Cloud Run, blocked by YouTube bot detection
|
||||
- Never uses remote flacfetch even when configured
|
||||
|
||||
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
|
||||
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
|
||||
- Works fine locally, not relevant to cloud issues
|
||||
|
||||
### Current Issues
|
||||
|
||||
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
|
||||
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
|
||||
3. **Inconsistent error handling** - Different paths handle errors differently
|
||||
4. **Testing gaps** - No integration tests verifying the full flow
|
||||
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
|
||||
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
|
||||
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
|
||||
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
|
||||
5. **Remove dead code** - Clean up redundant implementations
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Entry Points │
|
||||
├──────────────────────┬──────────────────────┬───────────────────────┤
|
||||
│ Audio Search Select │ Direct URL Submit │ File Upload │
|
||||
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
|
||||
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
┌──────────────────────────────────────────────┐ │
|
||||
│ YouTubeDownloadService │ │
|
||||
│ (NEW: backend/services/youtube_service.py) │ │
|
||||
│ │ │
|
||||
│ - download_by_url(url, job_id) → gcs_path │ │
|
||||
│ - download_by_id(video_id, job_id) → gcs_path│ │
|
||||
│ │ │
|
||||
│ Internally: │ │
|
||||
│ 1. Check if remote flacfetch enabled │ │
|
||||
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
|
||||
│ 3. If no: Use local yt_dlp with clear error │ │
|
||||
│ messaging about bot detection risk │ │
|
||||
└──────────────────┬───────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GCS │
|
||||
│ uploads/{job_id}/audio/ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Create YouTubeDownloadService
|
||||
|
||||
**File:** `backend/services/youtube_download_service.py`
|
||||
|
||||
```python
|
||||
class YouTubeDownloadService:
|
||||
"""
|
||||
Single point of entry for all YouTube downloads in the cloud backend.
|
||||
|
||||
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
|
||||
on the flacfetch VM which has YouTube cookies and avoids bot detection.
|
||||
|
||||
When remote is not configured, falls back to local yt_dlp with a warning
|
||||
that downloads may be blocked.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._flacfetch_client = get_flacfetch_client()
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio and upload to GCS.
|
||||
|
||||
Args:
|
||||
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (not gs:// prefix, just the path portion)
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
video_id = self._extract_video_id(url)
|
||||
|
||||
if self._flacfetch_client:
|
||||
return await self._download_remote(video_id, job_id, artist, title)
|
||||
else:
|
||||
logger.warning(
|
||||
"Remote flacfetch not configured. Attempting local download, "
|
||||
"but this may fail due to YouTube bot detection on Cloud Run IPs."
|
||||
)
|
||||
return await self._download_local(url, job_id, artist, title)
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Download by video ID instead of URL."""
|
||||
url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
return await self.download(url, job_id, artist, title)
|
||||
```
|
||||
|
||||
### Phase 2: Refactor Audio Search Path
|
||||
|
||||
**File:** `backend/api/routes/audio_search.py`
|
||||
|
||||
Changes to `_download_and_start_processing()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Complex branching with multiple paths
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Torrent path...
|
||||
elif is_remote_enabled and source_id and source_name:
|
||||
# YouTube remote path (buggy source_id extraction)...
|
||||
else:
|
||||
# Local path with another YouTube branch...
|
||||
|
||||
# AFTER: Clean separation by source type
|
||||
if is_torrent_source:
|
||||
result = await _download_torrent(source_name, source_id, target_file, job_id)
|
||||
elif source_name == 'YouTube':
|
||||
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
|
||||
elif source_name == 'Spotify':
|
||||
result = await _download_spotify(source_id, job_id, artist, title)
|
||||
else:
|
||||
raise DownloadError(f"Unknown source: {source_name}")
|
||||
```
|
||||
|
||||
### Phase 3: Refactor Direct URL Path
|
||||
|
||||
**File:** `backend/api/routes/file_upload.py`
|
||||
|
||||
Changes to `create_job_from_url()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
|
||||
# This always runs on Cloud Run and gets blocked
|
||||
|
||||
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
|
||||
if is_youtube_url(body.url):
|
||||
youtube_service = get_youtube_download_service()
|
||||
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
|
||||
# Update job with audio already downloaded
|
||||
job.input_media_gcs_path = gcs_path
|
||||
# Skip audio download in worker
|
||||
```
|
||||
|
||||
**File:** `backend/workers/audio_worker.py`
|
||||
|
||||
Changes to `download_audio()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Has YouTube download logic with download_from_url()
|
||||
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
|
||||
|
||||
async def download_audio(job_id: str, job: Job) -> str:
|
||||
"""Download audio from GCS (already uploaded by service layer)."""
|
||||
if not job.input_media_gcs_path:
|
||||
raise Exception("No audio path - this shouldn't happen for URL jobs")
|
||||
|
||||
# Download from GCS to local for processing
|
||||
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
|
||||
storage_service.download_file(job.input_media_gcs_path, local_path)
|
||||
return local_path
|
||||
```
|
||||
|
||||
### Phase 4: Clean Up Dead Code
|
||||
|
||||
Remove or consolidate:
|
||||
|
||||
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
|
||||
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
|
||||
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
|
||||
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
|
||||
|
||||
### Phase 5: Comprehensive Testing
|
||||
|
||||
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadService:
|
||||
def test_download_uses_remote_when_configured(self):
|
||||
"""When FLACFETCH_API_URL is set, should use remote client."""
|
||||
|
||||
def test_download_falls_back_to_local_when_remote_disabled(self):
|
||||
"""When remote not configured, should use local yt_dlp."""
|
||||
|
||||
def test_download_extracts_video_id_from_various_url_formats(self):
|
||||
"""Should handle youtube.com, youtu.be, shorts, etc."""
|
||||
|
||||
def test_download_returns_correct_gcs_path_format(self):
|
||||
"""Should return path without gs:// prefix."""
|
||||
|
||||
def test_download_handles_remote_failure_gracefully(self):
|
||||
"""If remote fails, should raise clear error, not fall back silently."""
|
||||
```
|
||||
|
||||
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadFlow:
|
||||
@pytest.mark.integration
|
||||
async def test_audio_search_youtube_selection_downloads_correctly(self):
|
||||
"""Full flow: search → select YouTube → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_direct_url_submission_downloads_correctly(self):
|
||||
"""Full flow: submit YouTube URL → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_youtube_download_with_remote_flacfetch(self):
|
||||
"""Verify remote flacfetch integration works end-to-end."""
|
||||
```
|
||||
|
||||
#### Flacfetch Tests (in flacfetch repo)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadByIdEndpoint:
|
||||
def test_download_by_id_with_youtube_source(self):
|
||||
"""POST /download-by-id with source_name=YouTube works."""
|
||||
|
||||
def test_download_by_id_uploads_to_gcs_when_requested(self):
|
||||
"""When gcs_path provided, should upload and return gs:// path."""
|
||||
|
||||
def test_download_by_id_handles_bot_detection_error(self):
|
||||
"""Should return clear error if YouTube blocks the request."""
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
|
||||
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
|
||||
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
|
||||
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
|
||||
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
|
||||
2. **Phase 3** changes the URL submission flow - may need careful rollout
|
||||
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
|
||||
4. **Phase 5** tests should be written alongside each phase
|
||||
|
||||
## Quick Fix vs Full Refactor
|
||||
|
||||
### Quick Fix (do now)
|
||||
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
|
||||
- This unblocks the "search → select YouTube" flow
|
||||
|
||||
### Full Refactor (this plan)
|
||||
- Consolidate all YouTube downloads into `YouTubeDownloadService`
|
||||
- Fix the "direct URL submission" flow to use remote flacfetch
|
||||
- Remove dead code and simplify branching
|
||||
- Add comprehensive tests
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing flows | Comprehensive tests before deployment |
|
||||
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
|
||||
| YouTube bot detection | Remote flacfetch with cookies is the solution |
|
||||
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **One code path** for YouTube downloads in cloud backend
|
||||
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
|
||||
3. **All tests pass** with >80% coverage on new code
|
||||
4. **Production verified** with real YouTube downloads
|
||||
5. **No dead code** remaining in the codebase
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
|
||||
2. Should direct URL submission be synchronous (wait for download) or async (background)?
|
||||
3. How do we handle YouTube cookies management for remote flacfetch?
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
|
||||
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
|
||||
@@ -0,0 +1,204 @@
|
||||
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
|
||||
|
||||
**Created:** 2026-01-19
|
||||
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
|
||||
|
||||
This plan covers:
|
||||
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
|
||||
2. Archiving the GitHub repo
|
||||
3. Deprecating the PyPI package (via README update published as final version)
|
||||
4. Reorganizing vendored code to a permanent location in karaoke-gen
|
||||
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
|
||||
- [ ] python-lyrics-transcriber GitHub repo archived
|
||||
- [ ] Final PyPI release with deprecation notice in README
|
||||
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
|
||||
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
|
||||
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
|
||||
- [ ] Tests pass after reorganization
|
||||
- [ ] karaoke-gen pyproject.toml updated for new path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### New Path Structure
|
||||
|
||||
Move the vendored code to `libs/lyrics_transcriber/`:
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── libs/
|
||||
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
|
||||
│ ├── __init__.py
|
||||
│ ├── cli/
|
||||
│ ├── core/
|
||||
│ ├── correction/
|
||||
│ ├── lyrics/
|
||||
│ ├── output/
|
||||
│ ├── review/
|
||||
│ ├── storage/
|
||||
│ ├── transcribers/
|
||||
│ ├── types.py
|
||||
│ └── utils/
|
||||
├── karaoke_gen/
|
||||
├── backend/
|
||||
├── frontend/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Why `libs/lyrics_transcriber/` instead of alternatives?
|
||||
|
||||
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
|
||||
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
|
||||
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
|
||||
|
||||
### pyproject.toml Changes
|
||||
|
||||
```toml
|
||||
packages = [
|
||||
{ include = "karaoke_gen" },
|
||||
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
|
||||
{ include = "backend" }
|
||||
]
|
||||
include = [
|
||||
"karaoke_gen/nextjs_frontend/out",
|
||||
"karaoke_gen/nextjs_frontend/out/**/*",
|
||||
"libs/lyrics_transcriber/output/fonts/*", # Updated path
|
||||
"libs/lyrics_transcriber/output/cdgmaker/images/*",
|
||||
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
|
||||
]
|
||||
```
|
||||
|
||||
### Files to Remove from Vendored Code
|
||||
|
||||
From current `lyrics_transcriber_temp/`:
|
||||
- `pyproject.toml` - Package config for standalone release
|
||||
- `poetry.lock` - Lock file for standalone package
|
||||
- `README.md` - Will be replaced by deprecation notice
|
||||
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
|
||||
- `Dockerfile` - Standalone container build
|
||||
- `.github/` - CI/CD for standalone package
|
||||
- `.cursor/` - Cursor AI config
|
||||
- `.specify/` - Specify config
|
||||
- `.gitignore` - Not needed (karaoke-gen has its own)
|
||||
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
|
||||
- `scripts/` - Development scripts for standalone package
|
||||
- `specs/` - Specification files
|
||||
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
|
||||
|
||||
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update and Archive External Project (Manual Steps)
|
||||
|
||||
1. [ ] Clone python-lyrics-transcriber repo locally
|
||||
2. [ ] Create deprecation notice README:
|
||||
```markdown
|
||||
# python-lyrics-transcriber (DEPRECATED)
|
||||
|
||||
**This project has been deprecated and archived.**
|
||||
|
||||
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
|
||||
|
||||
## Migration
|
||||
|
||||
For karaoke video generation with synchronized lyrics:
|
||||
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
|
||||
- Web app: https://gen.nomadkaraoke.com
|
||||
|
||||
## Historical Context
|
||||
|
||||
This library was originally developed as a standalone tool for creating synchronized
|
||||
lyrics files. It has now been integrated into the karaoke-gen platform which provides
|
||||
a complete end-to-end solution for karaoke video generation.
|
||||
|
||||
The last standalone version was 0.81.0. No further releases will be made to PyPI.
|
||||
```
|
||||
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
|
||||
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
|
||||
|
||||
### Phase 2: Reorganize Vendored Code in karaoke-gen
|
||||
|
||||
5. [ ] Create `libs/` directory
|
||||
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
|
||||
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
|
||||
8. [ ] Update `pyproject.toml` packages and include paths
|
||||
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
|
||||
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
|
||||
|
||||
### Phase 3: Verify and Test
|
||||
|
||||
10. [ ] Run `poetry install` to verify package configuration
|
||||
11. [ ] Run `make test` to ensure all tests pass
|
||||
12. [ ] Verify imports work:
|
||||
```python
|
||||
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
```
|
||||
|
||||
### Phase 4: Handle Tests from lyrics_transcriber
|
||||
|
||||
13. [ ] Evaluate lyrics_transcriber tests:
|
||||
- **74 test files** in `lyrics_transcriber_temp/tests/`
|
||||
- These test the internal lyrics_transcriber library functionality
|
||||
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
|
||||
- Skip integration/performance/manual tests that were for standalone development
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
|
||||
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
|
||||
| `pyproject.toml` | Modify | Update package paths |
|
||||
| `scripts/benchmark_encoding.py` | Modify | Update import path |
|
||||
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Immediate Testing
|
||||
- Run `make test` after reorganization - all existing tests must pass
|
||||
- Verify Poetry can build/install the package correctly
|
||||
|
||||
### Test Migration Decision
|
||||
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
|
||||
- `unit/correction/` - Correction logic tests (important)
|
||||
- `unit/lyrics/` - Lyrics provider tests
|
||||
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
|
||||
- `integration/` - Integration tests
|
||||
- `performance/` - Performance benchmarks
|
||||
|
||||
**Recommendation:** Migrate a subset of critical unit tests, particularly:
|
||||
- `test_corrector.py` - Core correction logic
|
||||
- `test_anchor_sequence*.py` - Anchor sequence matching
|
||||
- Handler tests in `handlers/`
|
||||
|
||||
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
|
||||
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
|
||||
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Git revert the consolidation commits
|
||||
2. `lyrics_transcriber_temp/` can be restored from git history
|
||||
3. The external python-lyrics-transcriber repo can be unarchived if needed
|
||||
|
||||
## Notes
|
||||
|
||||
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
|
||||
- Only the internal project structure changes, not the API
|
||||
- This is primarily a cleanup/organization task with some external repo admin work
|
||||
@@ -0,0 +1,110 @@
|
||||
# Plan: Disable Auto-Correction (Agentic and Heuristic)
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1914-disable-auto-correction
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Disable agentic AI correction (LLM-based via Gemini)
|
||||
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
|
||||
- [x] Preserve the transcription → human review → video generation flow
|
||||
- [x] Minimal code changes (configuration-based if possible)
|
||||
- [x] Easy to re-enable in the future if correction quality improves
|
||||
|
||||
## Technical Approach
|
||||
|
||||
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
|
||||
|
||||
**Current behavior:**
|
||||
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
|
||||
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
|
||||
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
|
||||
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
|
||||
|
||||
**Proposed changes:**
|
||||
|
||||
### Option A: Add a new `skip_correction` config (Recommended)
|
||||
Add a single boolean `skip_correction` setting that:
|
||||
- When `true`: Skips the entire correction process, returning raw transcription
|
||||
- When `false` (future): Falls back to existing agentic/heuristic behavior
|
||||
|
||||
This is cleaner than disabling both systems separately because:
|
||||
1. Single config controls both
|
||||
2. Clear intent ("skip correction" vs "disable two different systems")
|
||||
3. Easy to understand for future maintainers
|
||||
|
||||
### Option B: Just disable both individually
|
||||
- Set `use_agentic_ai` default to `false`
|
||||
- Disable default heuristic handlers by setting `enabled_handlers=[]`
|
||||
|
||||
Option A is cleaner, but Option B requires fewer code changes.
|
||||
|
||||
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Add `skip_correction` config to backend/config.py**
|
||||
- Add `skip_correction: bool` setting, default `true`
|
||||
- Comment explaining this disables both agentic and heuristic correction
|
||||
|
||||
2. [x] **Update lyrics_worker.py to pass skip_correction**
|
||||
- Pass the config value through to LyricsProcessor/LyricsTranscriber
|
||||
|
||||
3. [x] **Update corrector.py to support skip_correction**
|
||||
- When `skip_correction=True`, return immediately with uncorrected segments
|
||||
- No handler processing, no agentic AI
|
||||
- Still need to do anchor/gap finding? No - skip entirely
|
||||
|
||||
4. [x] **Update LyricsProcessor to support skip_correction**
|
||||
- Pass through to LyricsTranscriber
|
||||
|
||||
5. [x] **Update LyricsTranscriber controller to support skip_correction**
|
||||
- Pass through to corrector or skip corrector.run() entirely
|
||||
|
||||
6. [x] **Update docs/README.md**
|
||||
- Note that auto-correction is disabled
|
||||
- Update "Agentic AI correction" status to "Disabled"
|
||||
|
||||
7. [x] **Run tests to verify no regressions**
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
|
||||
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
|
||||
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
|
||||
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
|
||||
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
|
||||
| `docs/README.md` | Modify | Update status to reflect disabled correction |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: No new tests needed - this is a config change
|
||||
- **Existing tests**: Run `make test` to ensure no regressions
|
||||
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the approach is straightforward.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
To re-enable auto-correction:
|
||||
1. Set `SKIP_CORRECTION=false` environment variable, OR
|
||||
2. Change the default back to `false` in `backend/config.py`
|
||||
|
||||
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
|
||||
|
||||
## Alternative Considered
|
||||
|
||||
Instead of adding a new config, we could:
|
||||
1. Set `USE_AGENTIC_AI=false` (disables agentic)
|
||||
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
|
||||
|
||||
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Plan: Encoding Worker Immutable Deployment Pattern
|
||||
|
||||
## Status: In Progress
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The GCE encoding worker has recurring version mismatch issues because:
|
||||
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
|
||||
2. Wheel selection uses `sort -V` which can fail if image is outdated
|
||||
3. CI only restarts service - doesn't verify version strictly
|
||||
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
|
||||
5. 51 old wheel files accumulate in GCS bucket
|
||||
|
||||
## Solution: Immutable Deployment Pattern
|
||||
|
||||
### Core Principles
|
||||
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
|
||||
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
|
||||
3. **Version Manifest**: CI writes expected version, service MUST match
|
||||
4. **Strict Verification**: CI FAILS if deployed version doesn't match
|
||||
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: GCS Structure & Scripts
|
||||
|
||||
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
|
||||
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
|
||||
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
|
||||
- [x] **Step 1.4**: Create README.md with documentation
|
||||
|
||||
### Phase 2: CI Workflow Updates
|
||||
|
||||
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
|
||||
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
|
||||
- [x] **Step 2.3**: Update CI to write version.txt manifest
|
||||
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
|
||||
|
||||
### Phase 3: Packer Image Updates
|
||||
|
||||
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
|
||||
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
|
||||
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
|
||||
|
||||
### Phase 4: Documentation & Cleanup
|
||||
|
||||
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
|
||||
- [x] **Step 4.2**: Update infrastructure/packer/README.md
|
||||
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
|
||||
|
||||
## New GCS Structure
|
||||
|
||||
```
|
||||
gs://karaoke-gen-storage-nomadkaraoke/
|
||||
├── wheels/
|
||||
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
|
||||
│ └── karaoke_gen-{version}.whl # Versioned for rollback
|
||||
│
|
||||
└── encoding-worker/
|
||||
├── bootstrap.sh # Reference copy
|
||||
├── startup.sh # CI-managed, downloaded on boot
|
||||
└── version.txt # Expected version number
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Purpose |
|
||||
|------|--------|---------|
|
||||
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
|
||||
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
|
||||
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
|
||||
| `.github/workflows/ci.yml` | Modify | New deployment flow |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit test**: Verify startup.sh logic locally
|
||||
2. **Integration test**: After merge, verify:
|
||||
- CI uploads files to correct GCS paths
|
||||
- Service starts with correct version
|
||||
- Health endpoint returns expected version
|
||||
3. **Manual verification**: Check VM logs after first deployment
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
|
||||
2. Previous versioned wheels remain in GCS for manual rollback
|
||||
3. Old Packer images remain for VM rollback
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - design is complete.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bootstrap.sh is ~10 lines and should never need changes
|
||||
- All logic changes go through startup.sh in GCS
|
||||
- Version verification is strict - CI fails on mismatch
|
||||
@@ -0,0 +1,166 @@
|
||||
# Plan: Fix YouTube Remote Download Missing GCS Path
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
|
||||
|
||||
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
|
||||
|
||||
## Root Cause
|
||||
|
||||
In `backend/api/routes/audio_search.py` lines 354-363:
|
||||
|
||||
```python
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
# <-- MISSING: gcs_path parameter
|
||||
)
|
||||
```
|
||||
|
||||
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
|
||||
- [x] Backend should receive a `gs://` path from flacfetch
|
||||
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
|
||||
- [x] Maintain backwards compatibility (local-only deployments should still work)
|
||||
- [x] Add test coverage for this code path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
|
||||
|
||||
The current code structure has two branches:
|
||||
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
|
||||
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
|
||||
|
||||
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
|
||||
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
|
||||
- If remote disabled: download locally and upload to GCS from Cloud Run
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
|
||||
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
|
||||
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
|
||||
4. [x] Update logging to clarify the download path being used
|
||||
5. [x] Bump version in `pyproject.toml`
|
||||
6. [x] Run tests to verify no regressions
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
|
||||
| `pyproject.toml` | Modify | Bump version |
|
||||
|
||||
## Detailed Code Changes
|
||||
|
||||
### audio_search.py (lines 349-393)
|
||||
|
||||
Current structure:
|
||||
```python
|
||||
else:
|
||||
# Local download (YouTube or fallback) <-- WRONG: not always local!
|
||||
temp_dir = tempfile.mkdtemp(...)
|
||||
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
result = audio_search_service.download_by_id(
|
||||
...
|
||||
output_dir=temp_dir, # <-- BUG: unused for remote
|
||||
# Missing gcs_path!
|
||||
)
|
||||
```
|
||||
|
||||
Fixed structure:
|
||||
```python
|
||||
else:
|
||||
# Non-torrent download (YouTube or fallback)
|
||||
|
||||
if is_remote_enabled and source_id and source_name:
|
||||
# Remote download via flacfetch VM - upload directly to GCS
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination, # <-- FIX: Add this!
|
||||
)
|
||||
|
||||
# Handle GCS path response (same logic as torrent branch)
|
||||
if result.filepath.startswith("gs://"):
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: upload manually
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (remote disabled or no source_id)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
# ... existing local download logic ...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
|
||||
|
||||
```python
|
||||
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads via remote flacfetch include GCS path.
|
||||
|
||||
This is the same bug pattern as the torrent download bug fixed in 2025-12:
|
||||
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
|
||||
"""
|
||||
# Setup: YouTube source with remote enabled
|
||||
# Verify: download_by_id called with gcs_path parameter
|
||||
```
|
||||
|
||||
### Manual Testing (post-deploy)
|
||||
1. Create a job and select a YouTube audio source
|
||||
2. Verify job completes successfully
|
||||
3. Check GCS path is properly set in job data
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the fix is straightforward and follows the pattern established for torrent sources.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
This is a minor code change. If issues occur:
|
||||
1. Revert the PR
|
||||
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
@@ -0,0 +1,218 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -0,0 +1,317 @@
|
||||
# Plan: White Label B2B Karaoke Portals
|
||||
|
||||
**Created:** 2026-01-08
|
||||
**Branch:** feat/sess-20260108-0358-white-label-portal
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
|
||||
|
||||
**First customer: Vocal Star**
|
||||
- Subdomain: `vocalstar.nomadkaraoke.com`
|
||||
- Workflow: Upload original + instrumental audio (no audio search)
|
||||
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
|
||||
- Distribution: Download only (no Dropbox/GDrive/YouTube)
|
||||
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
|
||||
- [ ] Per-tenant frontend branding (logo, colors, metadata)
|
||||
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
|
||||
- [ ] Per-tenant default theme (always use Vocal Star theme)
|
||||
- [ ] Per-tenant allowed email domains for magic link auth
|
||||
- [ ] Per-tenant fixed API token support (for automation)
|
||||
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
|
||||
- [ ] Users scoped to tenant
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Minimal code duplication - single codebase serves all tenants
|
||||
- [ ] Easy to add new tenants via config, not code changes
|
||||
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
|
||||
- [ ] No performance impact on existing single-tenant flow
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decision: Config-Driven Multi-Tenancy
|
||||
|
||||
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
|
||||
|
||||
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
|
||||
2. **Frontend detects subdomain** and fetches tenant config from API
|
||||
3. **Backend middleware** extracts tenant from request and applies config
|
||||
4. **Firestore collections** get `tenant_id` field for isolation
|
||||
|
||||
This approach:
|
||||
- Requires no database migrations for new tenants
|
||||
- Configs can be managed via simple JSON files
|
||||
- Easy to test locally with query param override (`?tenant=vocalstar`)
|
||||
- Can evolve to admin UI later if needed
|
||||
|
||||
### Tenant Config Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vocalstar",
|
||||
"name": "Vocal Star",
|
||||
"subdomain": "vocalstar.nomadkaraoke.com",
|
||||
"branding": {
|
||||
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
|
||||
"primary_color": "#ffff00",
|
||||
"secondary_color": "#006CF9",
|
||||
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
|
||||
"site_title": "Vocal Star Karaoke Generator"
|
||||
},
|
||||
"features": {
|
||||
"audio_search": false,
|
||||
"file_upload": true,
|
||||
"youtube_upload": false,
|
||||
"dropbox_upload": false,
|
||||
"gdrive_upload": false,
|
||||
"theme_selection": false
|
||||
},
|
||||
"defaults": {
|
||||
"theme_id": "vocalstar",
|
||||
"distribution_mode": "download_only"
|
||||
},
|
||||
"auth": {
|
||||
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
|
||||
"fixed_tokens": ["<will be in Secret Manager>"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### 1. Subdomain Detection (Frontend)
|
||||
- Next.js middleware checks `request.headers.host`
|
||||
- If subdomain matches tenant, store tenant ID in cookie/context
|
||||
- Fetch `/api/tenant/config` to get branding/features
|
||||
- Apply CSS variables dynamically for colors
|
||||
- Swap logo component based on config
|
||||
|
||||
#### 2. Tenant Middleware (Backend)
|
||||
- FastAPI middleware extracts tenant from:
|
||||
- `X-Tenant-ID` header (set by frontend)
|
||||
- Or subdomain from `Host` header
|
||||
- Or query param `?tenant=` (dev only)
|
||||
- Attaches `tenant_id` to request state
|
||||
- Tenant config loaded from GCS (cached 5 min like themes)
|
||||
|
||||
#### 3. Auth Enhancement
|
||||
- `AuthService.validate_token_full()` already returns `user_email`
|
||||
- Add tenant validation: check if user's email domain is in tenant's allowed list
|
||||
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
|
||||
- Magic links include tenant context for proper redirect
|
||||
|
||||
#### 4. Data Isolation
|
||||
- Add `tenant_id: Optional[str]` to User and Job models
|
||||
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
|
||||
- Query filters: `where('tenant_id', '==', tenant_id or None)`
|
||||
- No schema migration needed - new field, optional
|
||||
|
||||
#### 5. Theme Integration
|
||||
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
|
||||
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
|
||||
- Tenant config specifies `default_theme_id: "vocalstar"`
|
||||
- When `theme_selection: false`, UI hides theme picker and uses default
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Tenant Foundation (Day 1-2)
|
||||
|
||||
1. [ ] **Create tenant model and config loader**
|
||||
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
|
||||
- Create `backend/services/tenant_service.py` with GCS config loading and caching
|
||||
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
|
||||
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
|
||||
|
||||
2. [ ] **Add tenant middleware**
|
||||
- Create `backend/api/middleware/tenant.py`
|
||||
- Extract tenant from subdomain or header
|
||||
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
|
||||
- Add `get_tenant()` dependency for routes
|
||||
|
||||
3. [ ] **Update data models**
|
||||
- Add `tenant_id: Optional[str] = None` to `User` model
|
||||
- Add `tenant_id: Optional[str] = None` to `Job` model
|
||||
- Update Firestore indexes in Pulumi for tenant-scoped queries
|
||||
|
||||
4. [ ] **Update auth for tenant awareness**
|
||||
- Modify `AuthService` to validate email domain against tenant config
|
||||
- Add tenant context to magic link generation
|
||||
- Support per-tenant fixed tokens in `auth_tokens` collection
|
||||
|
||||
### Phase 2: Upload Vocal Star Theme (Day 2)
|
||||
|
||||
5. [ ] **Prepare Vocal Star theme for GCS**
|
||||
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
|
||||
- Upload background images to `themes/vocalstar/assets/`
|
||||
- Upload Oswald font to `themes/vocalstar/assets/`
|
||||
- Upload `style_params.json` to `themes/vocalstar/`
|
||||
- Create preview image and upload to `themes/vocalstar/preview.png`
|
||||
- Add Vocal Star to `themes/_metadata.json`
|
||||
|
||||
6. [ ] **Create Vocal Star tenant config**
|
||||
- Create `tenants/vocalstar/config.json` in GCS
|
||||
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
|
||||
- Upload favicon to `tenants/vocalstar/favicon.ico`
|
||||
|
||||
### Phase 3: Frontend Tenant Support (Day 3-4)
|
||||
|
||||
7. [ ] **Add tenant context provider**
|
||||
- Create `frontend/lib/tenant.ts` with tenant config types
|
||||
- Create `frontend/hooks/useTenant.ts` for tenant state
|
||||
- Create `TenantProvider` component that fetches config on mount
|
||||
- Detect subdomain in `_app.tsx` or middleware
|
||||
|
||||
8. [ ] **Dynamic branding**
|
||||
- Create `TenantLogo` component that renders tenant or default logo
|
||||
- Update `globals.css` to support CSS variable overrides
|
||||
- Add `applyTenantColors()` function to set CSS variables from config
|
||||
- Update page metadata (title, favicon) from tenant config
|
||||
|
||||
9. [ ] **Feature-flagged UI**
|
||||
- Wrap audio search in `{tenant.features.audio_search && ...}`
|
||||
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
|
||||
- Wrap distribution options based on tenant features
|
||||
- Show simplified form for Vocal Star (just file upload fields)
|
||||
|
||||
10. [ ] **Update magic link flow**
|
||||
- Pass tenant context when requesting magic link
|
||||
- Magic link redirect goes to tenant subdomain
|
||||
- Verify endpoint extracts tenant from link context
|
||||
|
||||
### Phase 4: Backend Feature Enforcement (Day 4-5)
|
||||
|
||||
11. [ ] **Enforce tenant features in API**
|
||||
- Audio search endpoint: return 403 if `audio_search: false`
|
||||
- File upload: check `file_upload: true`
|
||||
- Distribution settings: ignore/error if disabled features requested
|
||||
- Theme selection: use default if `theme_selection: false`
|
||||
|
||||
12. [ ] **Job creation with tenant context**
|
||||
- `JobCreate` handler sets `tenant_id` from request state
|
||||
- Distribution config respects tenant defaults
|
||||
- Theme defaults to tenant's default theme
|
||||
|
||||
13. [ ] **Query scoping**
|
||||
- Update `JobManager.get_jobs()` to filter by tenant_id
|
||||
- Update `UserService` queries to include tenant_id
|
||||
- Admin endpoints bypass tenant filter (see all tenants)
|
||||
|
||||
### Phase 5: Infrastructure & Deployment (Day 5-6)
|
||||
|
||||
14. [ ] **DNS setup**
|
||||
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
|
||||
- Configure Cloudflare Pages to accept wildcard subdomain
|
||||
- Add SSL certificate for subdomain
|
||||
|
||||
15. [ ] **Backend CORS update**
|
||||
- Add `vocalstar.nomadkaraoke.com` to allowed origins
|
||||
- Or use pattern matching for `*.nomadkaraoke.com`
|
||||
|
||||
16. [ ] **Pulumi updates**
|
||||
- Add Firestore indexes for tenant_id queries
|
||||
- Add any new secrets (tenant fixed tokens)
|
||||
|
||||
### Phase 6: Testing & Documentation (Day 6-7)
|
||||
|
||||
17. [ ] **Testing**
|
||||
- Unit tests for TenantService config loading
|
||||
- Unit tests for tenant middleware
|
||||
- E2E test for Vocal Star flow (upload audio, generate, download)
|
||||
- Test magic link works for @vocal-star.com email
|
||||
|
||||
18. [ ] **Documentation**
|
||||
- Update CLAUDE.md with tenant info
|
||||
- Create `docs/WHITE-LABEL.md` with tenant setup guide
|
||||
- Document how to add new tenants
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/tenant.py` | Create | Tenant config models |
|
||||
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
|
||||
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
|
||||
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
|
||||
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
|
||||
| `backend/models/user.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/models/job.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
|
||||
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
|
||||
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/main.py` | Modify | Add tenant middleware |
|
||||
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
|
||||
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
|
||||
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
|
||||
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
|
||||
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
|
||||
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
|
||||
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
|
||||
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
|
||||
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `TenantService`: Config loading, caching, validation
|
||||
- `TenantMiddleware`: Subdomain extraction, header extraction
|
||||
- `AuthService`: Tenant email domain validation
|
||||
- `JobManager`: Tenant-scoped queries
|
||||
|
||||
### Integration Tests (Emulator)
|
||||
- Create job with tenant context
|
||||
- Query jobs filtered by tenant
|
||||
- Auth with tenant-allowed email domain
|
||||
- Auth rejection for wrong email domain
|
||||
|
||||
### E2E Tests
|
||||
- Vocal Star happy path: Upload audio files, review lyrics, download video
|
||||
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
|
||||
- Feature hiding: Verify audio search not shown on Vocal Star portal
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] **Q: Should tenant configs be in GCS or Firestore?**
|
||||
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
|
||||
|
||||
- [x] **Q: How to handle localhost development for specific tenant?**
|
||||
A: Query param override `?tenant=vocalstar` for local testing
|
||||
|
||||
- [x] **Q: Should admin portal be tenant-aware?**
|
||||
A: No - admins see all tenants. Admin routes bypass tenant filter.
|
||||
|
||||
- [x] **Q: Do we need tenant-specific email sender identity?**
|
||||
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
|
||||
|
||||
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
|
||||
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
|
||||
|
||||
1. Revert frontend to not check tenant
|
||||
2. Revert backend middleware to not extract tenant
|
||||
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
|
||||
4. No schema migration to reverse
|
||||
|
||||
For DNS: Simply remove CNAME record if needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
|
||||
- [ ] Portal shows Vocal Star branding (logo, colors)
|
||||
- [ ] Audio search is hidden; only file upload available
|
||||
- [ ] Theme is locked to Vocal Star style
|
||||
- [ ] Distribution options hidden; download only
|
||||
- [ ] Magic link works for @vocal-star.com emails
|
||||
- [ ] Fixed API token works for automation
|
||||
- [ ] Generated videos use Vocal Star theme
|
||||
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
|
||||
@@ -0,0 +1,115 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
+342
-60
@@ -67,6 +67,29 @@ jobs:
|
||||
# FRONTEND TESTS (PRs only - main skips tests)
|
||||
# ============================================
|
||||
|
||||
frontend-lint:
|
||||
name: "Frontend - Lint (Rules of Hooks)"
|
||||
runs-on: ubuntu-latest
|
||||
needs: changes
|
||||
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js with caching
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Run ESLint
|
||||
working-directory: frontend
|
||||
run: npm run lint
|
||||
|
||||
frontend-unit-tests:
|
||||
name: "Frontend - Unit Tests"
|
||||
runs-on: ubuntu-latest
|
||||
@@ -229,6 +252,7 @@ jobs:
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test against Production"
|
||||
echo "Time: $(date)"
|
||||
@@ -655,12 +679,14 @@ jobs:
|
||||
runs-on: ubuntu-latest
|
||||
# Note: frontend-e2e-full is optional (requires 'e2e' label) so not included here
|
||||
# frontend-e2e-smoke IS required - fast smoke tests catch critical regressions
|
||||
needs: [frontend-unit-tests, frontend-e2e-smoke, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
# frontend-lint IS required - catches React hooks violations that cause runtime crashes
|
||||
needs: [frontend-lint, frontend-unit-tests, frontend-e2e-smoke, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
|
||||
if: github.ref != 'refs/heads/main' && always()
|
||||
steps:
|
||||
- name: Check results
|
||||
run: |
|
||||
results=(
|
||||
"${{ needs.frontend-lint.result }}"
|
||||
"${{ needs.frontend-unit-tests.result }}"
|
||||
"${{ needs.frontend-e2e-smoke.result }}"
|
||||
"${{ needs.frontend-build.result }}"
|
||||
@@ -840,51 +866,23 @@ jobs:
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Enable Yarn (for lyrics frontend)
|
||||
run: corepack enable
|
||||
|
||||
# ===== Build Main Frontend (Next.js) =====
|
||||
- name: Install main frontend dependencies
|
||||
# ===== Build Unified Frontend (Next.js) =====
|
||||
# NOTE: Lyrics review and instrumental review UIs were consolidated into
|
||||
# the main Next.js frontend (PR #267, Jan 2025). The unified frontend
|
||||
# serves all three UIs from a single codebase at:
|
||||
# - / (main app)
|
||||
# - /app/jobs/{id}/lyrics (lyrics review)
|
||||
# - /app/jobs/{id}/instrumental (instrumental review)
|
||||
- name: Install frontend dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Build main frontend (Next.js static export)
|
||||
- name: Build frontend (Next.js static export)
|
||||
working-directory: frontend
|
||||
env:
|
||||
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
|
||||
run: npm run build
|
||||
|
||||
# ===== Build Lyrics Review Frontend (Vite) =====
|
||||
- name: Cache Yarn dependencies (lyrics frontend)
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
|
||||
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
|
||||
restore-keys: |
|
||||
yarn-lyrics-${{ runner.os }}-
|
||||
|
||||
- name: Install lyrics frontend dependencies
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
run: yarn install --frozen-lockfile
|
||||
|
||||
- name: Build lyrics frontend (Vite)
|
||||
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
|
||||
env:
|
||||
# Set base path for subpath deployment
|
||||
VITE_BASE_PATH: /lyrics/
|
||||
run: yarn build --base=/lyrics/
|
||||
|
||||
- name: Copy lyrics frontend to main output
|
||||
run: |
|
||||
mkdir -p frontend/out/lyrics
|
||||
cp -r lyrics_transcriber_temp/lyrics_transcriber/frontend/dist/* frontend/out/lyrics/
|
||||
|
||||
# ===== Copy Instrumental Review UI (Static HTML) =====
|
||||
- name: Copy instrumental review UI to main output
|
||||
run: |
|
||||
mkdir -p frontend/out/instrumental
|
||||
cp karaoke_gen/instrumental_review/static/index.html frontend/out/instrumental/
|
||||
|
||||
- name: Setup Pages
|
||||
uses: actions/configure-pages@v4
|
||||
|
||||
@@ -903,11 +901,14 @@ jobs:
|
||||
|
||||
deploy-backend:
|
||||
name: "Deploy - Backend (Cloud Run)"
|
||||
needs: [package-build]
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
outputs:
|
||||
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
@@ -965,6 +966,134 @@ jobs:
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
# ============================================
|
||||
# Deploy to GCE Encoding Worker (Immutable Deployment Pattern)
|
||||
#
|
||||
# This pattern ensures version consistency:
|
||||
# 1. Upload wheel to FIXED path (no version sorting)
|
||||
# 2. Upload startup.sh (logic can change without image rebuild)
|
||||
# 3. Write version manifest for verification
|
||||
# 4. Restart service (triggers bootstrap.sh -> startup.sh)
|
||||
# 5. STRICTLY verify deployed version (fail CI on mismatch)
|
||||
#
|
||||
# See infrastructure/encoding-worker/README.md for details
|
||||
# ============================================
|
||||
|
||||
- name: Download wheel artifact
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: package-dist
|
||||
path: dist/
|
||||
|
||||
- name: Deploy to GCE Encoding Worker
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
BUCKET="gs://karaoke-gen-storage-${{ env.PROJECT_ID }}"
|
||||
|
||||
echo "🚀 Deploying encoding worker v${VERSION}..."
|
||||
|
||||
# 1. Find the wheel file
|
||||
WHEEL_FILE=$(ls dist/*.whl | head -1)
|
||||
if [ -z "$WHEEL_FILE" ]; then
|
||||
echo "::error::No wheel file found in dist/"
|
||||
exit 1
|
||||
fi
|
||||
echo "📦 Found wheel: $WHEEL_FILE"
|
||||
|
||||
# 2. Upload wheel to FIXED path (atomic via temp file)
|
||||
# This eliminates version sorting - always use karaoke_gen-current.whl
|
||||
echo "📤 Uploading wheel to fixed path..."
|
||||
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/karaoke_gen-current.whl.tmp"
|
||||
gsutil mv "${BUCKET}/wheels/karaoke_gen-current.whl.tmp" \
|
||||
"${BUCKET}/wheels/karaoke_gen-current.whl"
|
||||
|
||||
# Also keep versioned copy for rollback/debugging
|
||||
gsutil cp "$WHEEL_FILE" "${BUCKET}/wheels/"
|
||||
|
||||
# 3. Upload startup.sh (allows logic changes without image rebuild)
|
||||
echo "📤 Uploading startup.sh..."
|
||||
gsutil cp infrastructure/encoding-worker/startup.sh \
|
||||
"${BUCKET}/encoding-worker/startup.sh"
|
||||
|
||||
# 4. Write version manifest for verification
|
||||
echo "📝 Writing version manifest..."
|
||||
echo "${VERSION}" | gsutil cp - "${BUCKET}/encoding-worker/version.txt"
|
||||
|
||||
echo "✅ GCS deployment complete"
|
||||
|
||||
- name: Restart GCE Encoding Worker
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
|
||||
echo "🔄 Restarting encoding worker..."
|
||||
|
||||
# Use IAP tunneling - works directly with service account credentials
|
||||
gcloud compute ssh encoding-worker \
|
||||
--zone=us-central1-c \
|
||||
--project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
echo "⏳ Waiting for service to start (up to 60s)..."
|
||||
sleep 20
|
||||
|
||||
- name: Verify Encoding Worker Version
|
||||
run: |
|
||||
VERSION="${{ steps.version_check.outputs.version }}"
|
||||
|
||||
echo "🔍 Verifying encoding worker version..."
|
||||
|
||||
# Retry loop - service may take time to fully start
|
||||
MAX_RETRIES=6
|
||||
RETRY_DELAY=10
|
||||
|
||||
for i in $(seq 1 $MAX_RETRIES); do
|
||||
echo "Attempt $i/$MAX_RETRIES..."
|
||||
|
||||
NEW_VERSION=$(gcloud compute ssh encoding-worker \
|
||||
--zone=us-central1-c \
|
||||
--project=nomadkaraoke \
|
||||
--tunnel-through-iap \
|
||||
--command="curl -s http://localhost:8080/health 2>/dev/null | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4" 2>/dev/null || echo "")
|
||||
|
||||
if [ "$NEW_VERSION" = "$VERSION" ]; then
|
||||
echo "✅ Encoding worker version verified: v$NEW_VERSION"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -n "$NEW_VERSION" ]; then
|
||||
echo "⚠️ Version: $NEW_VERSION (expected: $VERSION)"
|
||||
else
|
||||
echo "⚠️ Health endpoint not responding yet"
|
||||
fi
|
||||
|
||||
if [ $i -lt $MAX_RETRIES ]; then
|
||||
echo " Retrying in ${RETRY_DELAY}s..."
|
||||
sleep $RETRY_DELAY
|
||||
fi
|
||||
done
|
||||
|
||||
# STRICT VERIFICATION: Fail CI on version mismatch
|
||||
echo ""
|
||||
echo "=============================================="
|
||||
echo "❌ ENCODING WORKER DEPLOYMENT FAILED"
|
||||
echo "=============================================="
|
||||
echo "Expected version: $VERSION"
|
||||
echo "Deployed version: $NEW_VERSION"
|
||||
echo ""
|
||||
echo "Possible causes:"
|
||||
echo " - GCS upload failed (check 'Deploy to GCE' step)"
|
||||
echo " - startup.sh download failed (check VM logs)"
|
||||
echo " - Wheel installation failed (check VM logs)"
|
||||
echo ""
|
||||
echo "To debug, SSH to VM and check logs:"
|
||||
echo " gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke --tunnel-through-iap"
|
||||
echo " sudo journalctl -u encoding-worker -n 100"
|
||||
echo " cat /var/log/encoding-worker-startup.log"
|
||||
echo "=============================================="
|
||||
|
||||
exit 1
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
@@ -990,32 +1119,32 @@ jobs:
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Try to get the existing base image's hash label
|
||||
# First check if image exists locally (fast, no network call)
|
||||
# Only pull from registry if not cached locally
|
||||
# Fetch hash label directly from registry (10KB config blob, not full image)
|
||||
# This avoids issues with stale local Docker cache on GitHub runners
|
||||
EXISTING_HASH=""
|
||||
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
|
||||
echo "✅ Base image found in local cache (skipping slow registry pull)"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
elif docker pull $BASE_IMAGE 2>/dev/null; then
|
||||
echo "📥 Base image pulled from registry"
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
|
||||
# Fallback to old label name for backwards compatibility
|
||||
if [ -z "$EXISTING_HASH" ]; then
|
||||
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
|
||||
fi
|
||||
echo "Existing base image hash: $EXISTING_HASH"
|
||||
|
||||
# Get the manifest (small JSON, ~1KB) - includes config digest
|
||||
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
|
||||
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
|
||||
|
||||
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
|
||||
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
|
||||
|
||||
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
|
||||
TOKEN=$(gcloud auth print-access-token)
|
||||
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
|
||||
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
|
||||
|
||||
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
|
||||
echo "📋 Registry base image hash: $EXISTING_HASH"
|
||||
else
|
||||
echo "⚠️ No existing base image found - will build"
|
||||
echo "⚠️ No existing base image found in registry - will build"
|
||||
fi
|
||||
|
||||
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
|
||||
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
|
||||
echo " Current: ${{ steps.base-hash.outputs.hash }}"
|
||||
echo " Registry: $EXISTING_HASH"
|
||||
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "✅ Base image is up to date"
|
||||
@@ -1043,6 +1172,31 @@ jobs:
|
||||
# Provenance disabled for smaller image and faster push
|
||||
provenance: false
|
||||
|
||||
- name: Ensure local base image is current (if not rebuilt)
|
||||
if: steps.check-base.outputs.needs_rebuild == 'false'
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
|
||||
# Get the config digest from registry manifest (identifies the image content)
|
||||
# This is the same value as Docker's Image ID for the same image
|
||||
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
|
||||
|
||||
# Get local image's ID (which is the config digest)
|
||||
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
|
||||
echo "Local image ID: $LOCAL_IMAGE_ID"
|
||||
|
||||
if [ -z "$LOCAL_IMAGE_ID" ]; then
|
||||
echo "📥 Base image not in local cache - pulling from registry"
|
||||
docker pull $BASE_IMAGE
|
||||
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
|
||||
echo "🔄 Local cache has different image - pulling latest"
|
||||
docker pull $BASE_IMAGE
|
||||
else
|
||||
echo "✅ Local cache matches registry - skipping pull"
|
||||
fi
|
||||
|
||||
- name: Build and push app image
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
@@ -1080,13 +1234,16 @@ jobs:
|
||||
--memory 16Gi \
|
||||
--cpu 8 \
|
||||
--cpu-boost \
|
||||
--no-cpu-throttling \
|
||||
--execution-environment gen2 \
|
||||
--timeout 1800 \
|
||||
--concurrency 1 \
|
||||
--max-instances 50 \
|
||||
--min-instances 4 \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
|
||||
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
|
||||
--vpc-egress private-ranges-only \
|
||||
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true" \
|
||||
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest"
|
||||
|
||||
- name: Verify deployment and version
|
||||
run: |
|
||||
@@ -1139,3 +1296,128 @@ jobs:
|
||||
echo ""
|
||||
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
|
||||
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
|
||||
|
||||
# ============================================
|
||||
# PROPAGATE BASE IMAGE TO ALL RUNNERS
|
||||
# When base image is rebuilt, pull it to all 20 runners
|
||||
# so subsequent app builds don't wait 8 min for the pull
|
||||
#
|
||||
# NOTE: This is best-effort distribution. GitHub Actions doesn't
|
||||
# support targeting specific runners by name, so we spawn 20 jobs
|
||||
# and rely on the scheduler to distribute them across idle runners.
|
||||
# In practice, with all runners idle after deploy-backend, distribution
|
||||
# is good. Runners that miss this pull will get the image on their
|
||||
# next app build (with the 8 min delay).
|
||||
# ============================================
|
||||
|
||||
propagate-base-image:
|
||||
name: "Propagate Base Image (${{ matrix.runner_index }})"
|
||||
needs: deploy-backend
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
strategy:
|
||||
matrix:
|
||||
# 20 parallel jobs - GitHub distributes across available runners
|
||||
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
|
||||
max-parallel: 20
|
||||
fail-fast: false # Don't cancel other runners if one fails
|
||||
env:
|
||||
REGISTRY: us-central1-docker.pkg.dev
|
||||
PROJECT_ID: nomadkaraoke
|
||||
REPOSITORY: karaoke-repo
|
||||
BASE_IMAGE_NAME: karaoke-backend-base
|
||||
steps:
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Configure Docker for Artifact Registry
|
||||
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
|
||||
|
||||
- name: Pull latest base image
|
||||
run: |
|
||||
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
|
||||
RUNNER_NAME=$(hostname)
|
||||
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
|
||||
docker pull $BASE_IMAGE
|
||||
echo "✅ Base image cached on $RUNNER_NAME"
|
||||
|
||||
deploy-infrastructure:
|
||||
name: "Deploy - Infrastructure (Pulumi)"
|
||||
runs-on: [self-hosted, linux, gcp]
|
||||
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.13'
|
||||
|
||||
- name: Install Pulumi
|
||||
run: |
|
||||
curl -fsSL https://get.pulumi.com | sh
|
||||
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
|
||||
|
||||
- name: Install Python dependencies
|
||||
working-directory: infrastructure
|
||||
run: |
|
||||
pip install pulumi pulumi-gcp pulumi-docker
|
||||
|
||||
- name: Configure Pulumi
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
pulumi login
|
||||
|
||||
- name: Refresh Infrastructure State
|
||||
working-directory: infrastructure
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
echo "🔄 Refreshing Pulumi state to sync with actual cloud resources..."
|
||||
# Refresh detects resources deleted outside of Pulumi (e.g., via gcloud CLI)
|
||||
# This ensures Pulumi recreates missing resources
|
||||
pulumi refresh --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod || true
|
||||
|
||||
- name: Deploy Infrastructure
|
||||
working-directory: infrastructure
|
||||
env:
|
||||
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
|
||||
run: |
|
||||
echo "🚀 Deploying infrastructure changes..."
|
||||
# Skip preview to avoid needing broad read permissions on all GCP resources
|
||||
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
|
||||
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
|
||||
|
||||
- name: Summary
|
||||
if: success()
|
||||
run: |
|
||||
echo "🎉 Infrastructure deployment successful!"
|
||||
echo "Commit: ${{ github.sha }}"
|
||||
echo "Branch: ${{ github.ref_name }}"
|
||||
@@ -74,6 +74,7 @@ jobs:
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
|
||||
+1
-2
@@ -58,8 +58,7 @@ frontend/.eslintcache
|
||||
# Storybook
|
||||
frontend/storybook-static/
|
||||
|
||||
# PWA files
|
||||
frontend/**/public/sw.js
|
||||
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
|
||||
frontend/**/public/workbox-*.js
|
||||
frontend/**/public/swe-worker-*.js
|
||||
|
||||
|
||||
@@ -16,8 +16,11 @@
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
@@ -34,22 +37,62 @@
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
### Testing (Required)
|
||||
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
|
||||
- Which test types are needed (unit, integration, E2E)
|
||||
- Where tests should be placed
|
||||
- What mocking approach to use
|
||||
- Coverage expectations
|
||||
- **Production E2E tests for user-facing features** - see below
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
|
||||
|
||||
**Before committing**, run all tests:
|
||||
```bash
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
This single command:
|
||||
- Installs dependencies automatically (backend + frontend)
|
||||
- Runs all backend tests (unit, integration, emulator)
|
||||
- Runs all frontend tests (Jest unit + Playwright E2E)
|
||||
- Takes ~5-10 minutes
|
||||
|
||||
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
|
||||
|
||||
### Testing in Production (for Agents)
|
||||
|
||||
When asked to "test it yourself in prod" or verify a fix in production:
|
||||
|
||||
```bash
|
||||
# Get admin token
|
||||
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
|
||||
|
||||
# Option 1: Use the debug template directly
|
||||
cd frontend && node e2e/helpers/debug-prod-template.mjs
|
||||
|
||||
# Option 2: Copy template for customization (gitignored)
|
||||
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
|
||||
# Edit the script, then run it
|
||||
node test-my-issue.local.mjs
|
||||
|
||||
# Option 3: Run existing production E2E tests
|
||||
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
|
||||
```
|
||||
|
||||
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
|
||||
|
||||
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
@@ -86,11 +129,11 @@ EOF
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
|
||||
@@ -1,34 +1,50 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
.PHONY: help install install-backend install-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Install dependencies (only if needed)
|
||||
install-backend:
|
||||
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
|
||||
echo "=== Installing backend dependencies ==="; \
|
||||
poetry install; \
|
||||
fi
|
||||
|
||||
install-frontend:
|
||||
@if [ ! -d "frontend/node_modules" ]; then \
|
||||
echo "=== Installing frontend dependencies ==="; \
|
||||
cd frontend && npm install; \
|
||||
fi
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit:
|
||||
test-unit: install-backend
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e:
|
||||
test-e2e: install-backend
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
@@ -42,8 +58,18 @@ test-e2e:
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
# Run all backend tests (unit + emulator)
|
||||
test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend tests (unit + E2E)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend tests ==="
|
||||
cd frontend && npm run test:all
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
|
||||
@@ -159,6 +159,9 @@ karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# Skip video encoding (CDG/TXT only, faster)
|
||||
karaoke-gen --no-video --enable_cdg "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
@@ -403,26 +406,8 @@ gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
|
||||
# Get outputs from Pulumi
|
||||
SA_EMAIL=$(pulumi stack output service_account_email)
|
||||
BUCKET_NAME=$(pulumi stack output bucket_name)
|
||||
|
||||
# Deploy Cloud Run service
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--service-account $SA_EMAIL \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
|
||||
```
|
||||
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
|
||||
See `.github/workflows/ci.yml` for the full deployment workflow.
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
|
||||
+2
-3
@@ -22,10 +22,9 @@ WORKDIR /app
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules (like style_loader.py) are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
|
||||
# This ensures new modules are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
|
||||
@@ -54,6 +54,7 @@ WORKDIR /app
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
|
||||
+1232
-94
File diff suppressed because it is too large.
Load diff
+170
-136
@@ -13,14 +13,17 @@ The flow is:
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import mimetypes
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from karaoke_gen.utils import normalize_text
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
@@ -32,10 +35,17 @@ from backend.services.audio_search_service import (
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.services.youtube_download_service import (
|
||||
get_youtube_download_service,
|
||||
YouTubeDownloadError,
|
||||
)
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -95,7 +105,7 @@ class AudioSearchRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -107,6 +117,10 @@ class AudioSearchRequest(BaseModel):
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Display overrides (optional - if empty, search values are used for display)
|
||||
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
|
||||
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
@@ -120,6 +134,13 @@ class AudioSearchRequest(BaseModel):
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
|
||||
def normalize_text_fields(cls, v):
|
||||
"""Normalize text fields to standardize Unicode characters."""
|
||||
if v is not None and isinstance(v, str):
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
@@ -187,35 +208,6 @@ class AudioSelectResponse(BaseModel):
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Explicit values override defaults, None uses default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
@@ -252,6 +244,24 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _extract_gcs_path(filepath: str) -> str:
|
||||
"""
|
||||
Extract the path portion from a GCS path or gs:// URL.
|
||||
|
||||
Args:
|
||||
filepath: Either a gs:// URL or a plain path
|
||||
|
||||
Returns:
|
||||
The path portion (e.g., "uploads/job123/audio/file.flac")
|
||||
"""
|
||||
if filepath.startswith("gs://"):
|
||||
# Format: gs://bucket/path/to/file
|
||||
parts = filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
return parts[1]
|
||||
return filepath
|
||||
|
||||
|
||||
async def _download_and_start_processing(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
@@ -302,129 +312,127 @@ async def _download_and_start_processing(
|
||||
)
|
||||
|
||||
try:
|
||||
# Determine if this is a remote torrent download
|
||||
is_torrent_source = selected.get('provider') in ['RED', 'OPS']
|
||||
is_remote_enabled = audio_search_service.is_remote_enabled()
|
||||
|
||||
# Check if we can use download_by_id (preferred - avoids re-searching)
|
||||
source_id = selected.get('source_id')
|
||||
# Get source info from selected result
|
||||
source_name = selected.get('provider')
|
||||
source_id = selected.get('source_id')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
# For remote torrent downloads, have flacfetch VM upload directly to GCS
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Generate GCS path for remote upload
|
||||
# Route to appropriate download handler based on source type
|
||||
if source_name == 'YouTube':
|
||||
# Use YouTubeDownloadService for all YouTube downloads
|
||||
# This ensures consistent handling whether remote is configured or not
|
||||
youtube_service = get_youtube_download_service()
|
||||
|
||||
if not source_id and download_url:
|
||||
# Extract video ID from URL if not in source_id
|
||||
source_id = youtube_service._extract_video_id(download_url)
|
||||
|
||||
if not source_id:
|
||||
raise DownloadError(f"No video ID available for YouTube download")
|
||||
|
||||
logger.info(f"YouTube download via YouTubeDownloadService: video_id={source_id}")
|
||||
|
||||
try:
|
||||
audio_gcs_path = await youtube_service.download_by_id(
|
||||
video_id=source_id,
|
||||
job_id=job_id,
|
||||
artist=selected.get('artist'),
|
||||
title=selected.get('title'),
|
||||
)
|
||||
filename = os.path.basename(audio_gcs_path)
|
||||
except YouTubeDownloadError as e:
|
||||
raise DownloadError(f"YouTube download failed: {e}")
|
||||
|
||||
elif source_name in ['RED', 'OPS']:
|
||||
# Torrent sources - must use remote flacfetch
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
)
|
||||
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
# Use download_by_id if we have source_id (preferred - no re-search needed)
|
||||
if source_id and source_name:
|
||||
logger.info(f"Using download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
if source_id:
|
||||
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
output_dir="",
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
# Fallback to search-based download (requires re-search)
|
||||
logger.info(f"No source_id available, falling back to search-based download to: {gcs_destination}")
|
||||
|
||||
logger.info(f"Torrent download via search-based download")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir="", # Not used for remote
|
||||
output_dir="",
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# For remote downloads, filepath is already the GCS path
|
||||
if result.filepath.startswith("gs://"):
|
||||
# Extract the path portion after the bucket name
|
||||
# Format: gs://bucket/uploads/job_id/audio/filename.flac
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: treat as local path (shouldn't happen for remote)
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
# Extract GCS path from result
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
logger.info(f"Torrent download complete: {audio_gcs_path}")
|
||||
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
elif source_name == 'Spotify':
|
||||
# Spotify downloads - use audio_search_service
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
raise DownloadError(
|
||||
f"Cannot download from Spotify without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
)
|
||||
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Spotify download: source_id={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="",
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
logger.info(f"Spotify download complete: {audio_gcs_path}")
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (YouTube or fallback)
|
||||
# Unknown source - try generic download
|
||||
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
elif source_name == 'YouTube' and download_url:
|
||||
# For YouTube without remote, download directly using URL from state_data
|
||||
# This avoids relying on in-memory cache which doesn't persist across instances
|
||||
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
local_path = await download_from_url(
|
||||
download_url,
|
||||
temp_dir,
|
||||
selected.get('artist'),
|
||||
selected.get('title')
|
||||
)
|
||||
|
||||
if not local_path or not os.path.exists(local_path):
|
||||
raise DownloadError(f"Failed to download from YouTube: {download_url}")
|
||||
|
||||
# Create a result-like object
|
||||
class DownloadResult:
|
||||
def __init__(self, filepath):
|
||||
self.filepath = filepath
|
||||
|
||||
result = DownloadResult(local_path)
|
||||
else:
|
||||
# Fallback to search-based download (may fail if cache is empty)
|
||||
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
|
||||
try:
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id, # Pass for potential fallback scenarios
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type='audio/flac' # flacfetch typically returns FLAC
|
||||
)
|
||||
content_type, _ = mimetypes.guess_type(result.filepath)
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type=content_type or 'application/octet-stream'
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded audio to GCS: {audio_gcs_path}")
|
||||
|
||||
# Clean up temp file
|
||||
try:
|
||||
os.remove(result.filepath)
|
||||
os.rmdir(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
logger.info(f"Generic download uploaded to GCS: {audio_gcs_path}")
|
||||
finally:
|
||||
# Clean up temp files
|
||||
try:
|
||||
if os.path.exists(temp_dir):
|
||||
import shutil
|
||||
shutil.rmtree(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
job_manager.update_job(job_id, {
|
||||
@@ -466,17 +474,25 @@ async def search_audio(
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
|
||||
|
||||
This endpoint:
|
||||
1. Creates a job in PENDING state
|
||||
2. Searches for audio using flacfetch
|
||||
3. Either returns search results for user selection, or
|
||||
4. If auto_download=True, automatically selects best and starts processing
|
||||
|
||||
|
||||
Use cases:
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.audio_search:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Audio search is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Apply default distribution settings
|
||||
settings = get_settings()
|
||||
@@ -485,7 +501,8 @@ async def search_audio(
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
@@ -521,19 +538,34 @@ async def search_audio(
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Determine display values - use display_* if provided, otherwise fall back to search values
|
||||
# Display values are used for title screens, filenames, YouTube, etc.
|
||||
# Search values (body.artist, body.title) are used for audio search
|
||||
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
|
||||
effective_display_title = body.display_title.strip() if body.display_title else body.title
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
theme_id=body.theme_id,
|
||||
artist=effective_display_artist, # Display value for title screens, filenames
|
||||
title=effective_display_title, # Display value for title screens, filenames
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
@@ -556,8 +588,10 @@ async def search_audio(
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
@@ -572,11 +606,11 @@ async def search_audio(
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if body.theme_id and not body.style_files:
|
||||
if effective_theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
@@ -585,9 +619,9 @@ async def search_audio(
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
|
||||
+207
-124
@@ -15,7 +15,6 @@ import json
|
||||
import logging
|
||||
import tempfile
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body, Depends
|
||||
from pathlib import Path
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
@@ -29,13 +28,30 @@ from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.job_defaults_service import (
|
||||
get_effective_distribution_settings,
|
||||
resolve_cdg_txt_defaults,
|
||||
EffectiveDistributionSettings,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.services.metrics import metrics
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from backend.services.youtube_download_service import (
|
||||
get_youtube_download_service,
|
||||
YouTubeDownloadError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _is_youtube_url(url: str) -> bool:
|
||||
"""Check if a URL is a YouTube URL."""
|
||||
return any(domain in url.lower() for domain in [
|
||||
'youtube.com', 'youtu.be', 'youtube-nocookie.com'
|
||||
])
|
||||
router = APIRouter(tags=["jobs"])
|
||||
|
||||
|
||||
@@ -69,7 +85,7 @@ class CreateJobFromUrlRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -123,7 +139,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
|
||||
@@ -263,75 +279,6 @@ async def _trigger_audio_worker_only(job_id: str) -> None:
|
||||
await worker_service.trigger_audio_worker(job_id)
|
||||
|
||||
|
||||
def _resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool],
|
||||
enable_txt: Optional[bool]
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
When a theme is selected, CDG and TXT are enabled by default.
|
||||
Explicit True/False values always override the default.
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
# Default based on whether theme is set
|
||||
default_enabled = theme_id is not None
|
||||
|
||||
# Resolve with explicit override taking precedence
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_enabled
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
@dataclass
|
||||
class EffectiveDistributionSettings:
|
||||
"""Settings with defaults applied from environment variables."""
|
||||
dropbox_path: Optional[str]
|
||||
gdrive_folder_id: Optional[str]
|
||||
discord_webhook_url: Optional[str]
|
||||
brand_prefix: Optional[str]
|
||||
|
||||
|
||||
def _get_effective_distribution_settings(
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
discord_webhook_url: Optional[str] = None,
|
||||
brand_prefix: Optional[str] = None,
|
||||
) -> EffectiveDistributionSettings:
|
||||
"""
|
||||
Get distribution settings with defaults applied from environment variables.
|
||||
|
||||
This ensures consistent handling of defaults across all job creation endpoints.
|
||||
Each parameter, if not provided (None), falls back to the corresponding
|
||||
environment variable configured in settings.
|
||||
|
||||
Args:
|
||||
dropbox_path: Explicit Dropbox path or None for default
|
||||
gdrive_folder_id: Explicit Google Drive folder ID or None for default
|
||||
discord_webhook_url: Explicit Discord webhook URL or None for default
|
||||
brand_prefix: Explicit brand prefix or None for default
|
||||
|
||||
Returns:
|
||||
EffectiveDistributionSettings with defaults applied
|
||||
"""
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=dropbox_path or settings.default_dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
|
||||
brand_prefix=brand_prefix or settings.default_brand_prefix,
|
||||
)
|
||||
|
||||
|
||||
def _prepare_theme_for_job(
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
@@ -406,7 +353,7 @@ async def upload_and_create_job(
|
||||
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
|
||||
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
|
||||
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
|
||||
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
|
||||
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
|
||||
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
|
||||
@@ -446,6 +393,14 @@ async def upload_and_create_job(
|
||||
The style_params JSON can reference the uploaded images/fonts by their original
|
||||
filenames, and the backend will update the paths to GCS locations.
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.file_upload:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="File upload is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Validate main audio file type
|
||||
file_ext = Path(file.filename).suffix.lower()
|
||||
@@ -494,7 +449,7 @@ async def upload_and_create_job(
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
dist = _get_effective_distribution_settings(
|
||||
dist = get_effective_distribution_settings(
|
||||
dropbox_path=dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url,
|
||||
@@ -510,12 +465,19 @@ async def upload_and_create_job(
|
||||
if dist.brand_prefix and not brand_prefix:
|
||||
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
if effective_enable_youtube_upload and enable_youtube_upload is None:
|
||||
logger.info("Using default enable_youtube_upload: True (from env)")
|
||||
|
||||
# Validate credentials for requested distribution services (including defaults)
|
||||
# This prevents accepting jobs that will fail later due to missing credentials
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -555,9 +517,18 @@ async def upload_and_create_job(
|
||||
detail=f"Invalid color_overrides JSON: {e}"
|
||||
)
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
theme_id, enable_cdg, enable_txt
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, enable_cdg, enable_txt
|
||||
)
|
||||
|
||||
# Check if any custom style files are being uploaded (overrides theme)
|
||||
@@ -589,12 +560,12 @@ async def upload_and_create_job(
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=file.filename,
|
||||
theme_id=theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=parsed_color_overrides,
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=webhook_url,
|
||||
@@ -616,10 +587,12 @@ async def upload_and_create_job(
|
||||
request_metadata=request_metadata,
|
||||
# Non-interactive mode
|
||||
non_interactive=non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="upload")
|
||||
|
||||
@@ -631,16 +604,16 @@ async def upload_and_create_job(
|
||||
theme_style_params_path = None
|
||||
theme_style_assets = {}
|
||||
theme_youtube_desc = None
|
||||
if theme_id and not has_custom_style_files:
|
||||
if effective_theme_id and not has_custom_style_files:
|
||||
try:
|
||||
theme_style_params_path, theme_style_assets, theme_youtube_desc = _prepare_theme_for_job(
|
||||
job_id, theme_id, parsed_color_overrides or None
|
||||
job_id, effective_theme_id, parsed_color_overrides or None
|
||||
)
|
||||
logger.info(f"Applied theme '{theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except HTTPException:
|
||||
raise # Re-raise validation errors (e.g., theme not found)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{theme_id}' for job {job_id}: {e}")
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Upload main audio file to GCS
|
||||
@@ -815,11 +788,12 @@ async def upload_and_create_job(
|
||||
"using_default": gdrive_folder_id is None,
|
||||
}
|
||||
|
||||
if enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
youtube_result = credential_manager.check_youtube_credentials()
|
||||
distribution_services["youtube"] = {
|
||||
"enabled": True,
|
||||
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
|
||||
"using_default": enable_youtube_upload is None,
|
||||
}
|
||||
|
||||
if dist.discord_webhook_url:
|
||||
@@ -908,7 +882,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -1012,6 +986,14 @@ async def create_job_with_upload_urls(
|
||||
- Works with any HTTP client (no HTTP/2 required)
|
||||
- Resumable uploads possible with GCS
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.file_upload:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="File upload is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Validate files list
|
||||
if not body.files:
|
||||
@@ -1041,18 +1023,23 @@ async def create_job_with_upload_urls(
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
dist = _get_effective_distribution_settings(
|
||||
dist = get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1084,9 +1071,18 @@ async def create_job_with_upload_urls(
|
||||
# Get original audio filename
|
||||
audio_file = audio_files[0]
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
@@ -1100,12 +1096,12 @@ async def create_job_with_upload_urls(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename=audio_file.filename,
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
youtube_description_template=body.youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
@@ -1122,8 +1118,10 @@ async def create_job_with_upload_urls(
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
@@ -1132,9 +1130,9 @@ async def create_job_with_upload_urls(
|
||||
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
if effective_theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1144,7 +1142,7 @@ async def create_job_with_upload_urls(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
@@ -1450,6 +1448,14 @@ async def create_job_from_url(
|
||||
Note: YouTube rate limiting may cause occasional download failures.
|
||||
The backend will retry automatically.
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.youtube_url:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="URL-based job creation is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Validate URL
|
||||
if not _validate_url(body.url):
|
||||
@@ -1459,18 +1465,23 @@ async def create_job_from_url(
|
||||
)
|
||||
|
||||
# Apply default distribution settings from environment if not provided
|
||||
dist = _get_effective_distribution_settings(
|
||||
dist = get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate credentials for requested distribution services
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1503,9 +1514,18 @@ async def create_job_from_url(
|
||||
artist = body.artist
|
||||
title = body.title
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Prefer authenticated user's email over request body
|
||||
@@ -1517,12 +1537,12 @@ async def create_job_from_url(
|
||||
artist=artist,
|
||||
title=title,
|
||||
filename=None, # No file uploaded
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
webhook_url=body.webhook_url,
|
||||
@@ -1538,17 +1558,19 @@ async def create_job_from_url(
|
||||
other_stems_models=body.other_stems_models,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job_id, source="url")
|
||||
|
||||
# If theme is set, prepare theme style now
|
||||
if body.theme_id:
|
||||
if effective_theme_id:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1558,14 +1580,14 @@ async def create_job_from_url(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
|
||||
if artist:
|
||||
logger.info(f" Artist: {artist}")
|
||||
if title:
|
||||
logger.info(f" Title: {title}")
|
||||
|
||||
|
||||
# Transition job to DOWNLOADING state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1574,11 +1596,48 @@ async def create_job_from_url(
|
||||
message="Starting audio download from URL"
|
||||
)
|
||||
|
||||
# For URL jobs, trigger ONLY audio worker first
|
||||
# The audio worker will trigger lyrics worker after download completes
|
||||
# This prevents race condition where lyrics worker times out waiting for audio
|
||||
background_tasks.add_task(_trigger_audio_worker_only, job_id)
|
||||
|
||||
# For YouTube URLs, download audio NOW using YouTubeDownloadService
|
||||
# This uses remote flacfetch (if configured) to avoid bot detection on Cloud Run
|
||||
if _is_youtube_url(body.url):
|
||||
logger.info(f"YouTube URL detected, downloading via YouTubeDownloadService")
|
||||
try:
|
||||
youtube_service = get_youtube_download_service()
|
||||
audio_gcs_path = await youtube_service.download(
|
||||
url=body.url,
|
||||
job_id=job_id,
|
||||
artist=artist,
|
||||
title=title,
|
||||
)
|
||||
|
||||
# Update job with the downloaded audio path
|
||||
job_manager.update_job(job_id, {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': os.path.basename(audio_gcs_path),
|
||||
})
|
||||
|
||||
logger.info(f"YouTube audio downloaded to GCS: {audio_gcs_path}")
|
||||
|
||||
# Now trigger both workers in parallel (audio already downloaded)
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=10,
|
||||
message="Audio downloaded, starting processing"
|
||||
)
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
except YouTubeDownloadError as e:
|
||||
logger.error(f"YouTube download failed: {e}")
|
||||
job_manager.fail_job(job_id, f"YouTube download failed: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"YouTube download failed: {e}"
|
||||
)
|
||||
else:
|
||||
# For non-YouTube URLs, trigger ONLY audio worker first
|
||||
# The audio worker will download the URL and trigger lyrics worker after
|
||||
background_tasks.add_task(_trigger_audio_worker_only, job_id)
|
||||
|
||||
return CreateJobFromUrlResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
@@ -1658,6 +1717,14 @@ async def create_finalise_only_job(
|
||||
|
||||
The endpoint returns signed URLs for uploading all the prep files.
|
||||
"""
|
||||
# Check tenant feature flag - finalise-only requires file upload capability
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.file_upload:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="File upload is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Validate files list
|
||||
if not body.files:
|
||||
@@ -1705,18 +1772,23 @@ async def create_finalise_only_job(
|
||||
)
|
||||
|
||||
# Apply default distribution settings
|
||||
dist = _get_effective_distribution_settings(
|
||||
dist = get_effective_distribution_settings(
|
||||
dropbox_path=body.dropbox_path,
|
||||
gdrive_folder_id=body.gdrive_folder_id,
|
||||
discord_webhook_url=body.discord_webhook_url,
|
||||
brand_prefix=body.brand_prefix,
|
||||
)
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Validate distribution credentials if services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if body.enable_youtube_upload:
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
@@ -1745,9 +1817,18 @@ async def create_finalise_only_job(
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="finalise_only_upload")
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
body.theme_id, body.enable_cdg, body.enable_txt
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Check if style_params is being uploaded (overrides theme)
|
||||
@@ -1761,12 +1842,12 @@ async def create_finalise_only_job(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
filename="finalise_only", # No single audio file - using prep outputs
|
||||
theme_id=body.theme_id,
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
enable_youtube_upload=body.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=body.youtube_description,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
@@ -1775,8 +1856,10 @@ async def create_finalise_only_job(
|
||||
finalise_only=True,
|
||||
keep_brand_code=body.keep_brand_code,
|
||||
request_metadata=request_metadata,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
# Record job creation metric
|
||||
@@ -1785,9 +1868,9 @@ async def create_finalise_only_job(
|
||||
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
|
||||
|
||||
# If theme is set and no style_params uploaded, prepare theme style now
|
||||
if body.theme_id and not has_style_params_upload:
|
||||
if effective_theme_id and not has_style_params_upload:
|
||||
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, body.theme_id, body.color_overrides
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
# Update job with theme style data
|
||||
update_data = {
|
||||
@@ -1797,7 +1880,7 @@ async def create_finalise_only_job(
|
||||
if youtube_desc and not body.youtube_description:
|
||||
update_data['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, update_data)
|
||||
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
|
||||
# Generate signed upload URLs for each file
|
||||
upload_urls = []
|
||||
|
||||
@@ -13,6 +13,9 @@ from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -227,6 +230,7 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
@@ -237,6 +241,39 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -300,6 +337,68 @@ async def detailed_health_check() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
async def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
|
||||
@@ -316,13 +316,124 @@ async def trigger_render_video_worker(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
|
||||
if getattr(job, 'made_for_you', False):
|
||||
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
|
||||
add_span_event("made_for_you_skip")
|
||||
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
|
||||
+41
-19
@@ -11,7 +11,7 @@ import asyncio
|
||||
import logging
|
||||
import httpx
|
||||
from typing import List, Optional, Dict, Any, Tuple
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
|
||||
from backend.models.job import Job, JobCreate, JobResponse, JobStatus
|
||||
from backend.models.requests import (
|
||||
@@ -25,10 +25,13 @@ from backend.models.requests import (
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.services.metrics import metrics
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -87,20 +90,36 @@ async def create_job(
|
||||
user_email = request.user_email
|
||||
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Apply default theme - all jobs require a theme
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if not effective_theme_id:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="No default theme configured. Please contact support or specify a theme_id."
|
||||
)
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Create job with all preferences
|
||||
job_create = JobCreate(
|
||||
url=str(request.url),
|
||||
artist=request.artist,
|
||||
title=request.title,
|
||||
theme_id=effective_theme_id, # Required - all jobs must have a theme
|
||||
enable_cdg=request.enable_cdg,
|
||||
enable_txt=request.enable_txt,
|
||||
enable_youtube_upload=request.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=request.youtube_description,
|
||||
webhook_url=request.webhook_url,
|
||||
user_email=user_email
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job.job_id, source="url")
|
||||
|
||||
@@ -172,11 +191,13 @@ def _check_job_ownership(job: Job, auth_result: AuthResult) -> bool:
|
||||
|
||||
@router.get("", response_model=List[Job])
|
||||
async def list_jobs(
|
||||
request: Request,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[str] = None,
|
||||
created_before: Optional[str] = None,
|
||||
exclude_test: bool = True,
|
||||
limit: int = 100,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> List[Job]:
|
||||
@@ -184,6 +205,7 @@ async def list_jobs(
|
||||
List jobs with optional filters.
|
||||
|
||||
Regular users only see their own jobs. Admins see all jobs.
|
||||
Users on tenant portals only see jobs from their tenant.
|
||||
|
||||
Args:
|
||||
status: Filter by job status (pending, complete, failed, etc.)
|
||||
@@ -191,6 +213,7 @@ async def list_jobs(
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this ISO datetime (e.g., 2024-01-01T00:00:00Z)
|
||||
created_before: Filter jobs created before this ISO datetime
|
||||
exclude_test: If True (default), exclude jobs from test users (admin only)
|
||||
limit: Maximum number of jobs to return (default 100)
|
||||
|
||||
Returns:
|
||||
@@ -227,6 +250,10 @@ async def list_jobs(
|
||||
logger.warning("Non-admin auth without user_email, returning empty job list")
|
||||
return []
|
||||
|
||||
# Get tenant_id from request for portal scoping
|
||||
# Tenant users only see jobs from their tenant
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
|
||||
jobs = job_manager.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
@@ -234,9 +261,14 @@ async def list_jobs(
|
||||
created_after=created_after_dt,
|
||||
created_before=created_before_dt,
|
||||
user_email=user_email_filter,
|
||||
tenant_id=tenant_id,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
# Filter out test user jobs if exclude_test is True (admin only)
|
||||
if exclude_test and auth_result.is_admin:
|
||||
jobs = [j for j in jobs if not is_test_email(j.user_email or "")]
|
||||
|
||||
logger.debug(f"Listed {len(jobs)} jobs for user={auth_result.user_email}, admin={auth_result.is_admin}")
|
||||
return jobs
|
||||
except HTTPException:
|
||||
@@ -1159,21 +1191,11 @@ async def download_file(
|
||||
content_type = content_types.get(ext, 'application/octet-stream')
|
||||
|
||||
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
|
||||
# Sanitize artist/title to remove filesystem-unsafe characters
|
||||
def sanitize_filename_part(s: str) -> str:
|
||||
if not s:
|
||||
return ""
|
||||
# Remove/replace characters unsafe for filesystems: / \ : * ? " < > |
|
||||
unsafe_chars = r'[/\\:*?"<>|]'
|
||||
import re
|
||||
sanitized = re.sub(unsafe_chars, '', s)
|
||||
# Collapse multiple spaces and trim
|
||||
sanitized = ' '.join(sanitized.split())
|
||||
# Limit length to prevent overly long filenames
|
||||
return sanitized[:100].strip()
|
||||
|
||||
artist_clean = sanitize_filename_part(job.artist) if job.artist else None
|
||||
title_clean = sanitize_filename_part(job.title) if job.title else None
|
||||
# Use sanitize_filename to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that cause HTTP header encoding issues (Content-Disposition uses latin-1)
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
artist_clean = sanitize_filename(job.artist) if job.artist else None
|
||||
title_clean = sanitize_filename(job.title) if job.title else None
|
||||
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
|
||||
|
||||
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Push Notification API routes.
|
||||
|
||||
Provides endpoints for managing Web Push notification subscriptions:
|
||||
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
|
||||
- POST /api/push/subscribe: Register a push subscription
|
||||
- POST /api/push/unsubscribe: Remove a push subscription
|
||||
- GET /api/push/subscriptions: List user's subscriptions
|
||||
- POST /api/push/test: Send a test notification (admin only)
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/push", tags=["push"])
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
|
||||
class VapidPublicKeyResponse(BaseModel):
|
||||
"""Response containing VAPID public key."""
|
||||
enabled: bool
|
||||
vapid_public_key: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeRequest(BaseModel):
|
||||
"""Request to subscribe to push notifications."""
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
"""Response after subscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnsubscribeRequest(BaseModel):
|
||||
"""Request to unsubscribe from push notifications."""
|
||||
endpoint: str
|
||||
|
||||
|
||||
class UnsubscribeResponse(BaseModel):
|
||||
"""Response after unsubscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class SubscriptionInfo(BaseModel):
|
||||
"""Information about a push subscription."""
|
||||
endpoint: str
|
||||
device_name: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_used_at: Optional[str] = None
|
||||
|
||||
|
||||
class SubscriptionsListResponse(BaseModel):
|
||||
"""Response containing user's subscriptions."""
|
||||
subscriptions: List[SubscriptionInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
"""Request to send a test notification."""
|
||||
title: Optional[str] = "Test Notification"
|
||||
body: Optional[str] = "This is a test push notification from Karaoke Generator"
|
||||
|
||||
|
||||
class TestNotificationResponse(BaseModel):
|
||||
"""Response after sending test notification."""
|
||||
status: str
|
||||
sent_count: int
|
||||
message: str
|
||||
|
||||
|
||||
# Routes
|
||||
|
||||
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
|
||||
async def get_vapid_public_key():
|
||||
"""
|
||||
Get the VAPID public key for push subscription.
|
||||
|
||||
This endpoint is public - no authentication required.
|
||||
Returns the public key needed for client-side PushManager.subscribe().
|
||||
"""
|
||||
settings = get_settings()
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
if not settings.enable_push_notifications:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
public_key = push_service.get_public_key()
|
||||
if not public_key:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
return VapidPublicKeyResponse(
|
||||
enabled=True,
|
||||
vapid_public_key=public_key
|
||||
)
|
||||
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
request: SubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Register a push notification subscription for the current user.
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in request.keys or "auth" not in request.keys:
|
||||
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint,
|
||||
keys=request.keys,
|
||||
device_name=request.device_name
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail="Failed to save subscription")
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription registered successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Remove a push notification subscription.
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
success = await push_service.remove_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint
|
||||
)
|
||||
|
||||
if not success:
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Subscription removed (or was not found)"
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription removed successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
List all push notification subscriptions for the current user.
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
|
||||
|
||||
return SubscriptionsListResponse(
|
||||
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
|
||||
count=len(subscriptions)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Send a test push notification to the current user's devices.
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
sent_count = await push_service.send_push(
|
||||
user_email=auth_result.user_email,
|
||||
title=request.title or "Test Notification",
|
||||
body=request.body or "This is a test push notification",
|
||||
url="/app/",
|
||||
tag="test"
|
||||
)
|
||||
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=f"Test notification sent to {sent_count} device(s)"
|
||||
)
|
||||
@@ -0,0 +1,428 @@
|
||||
"""
|
||||
Admin API routes for rate limit management.
|
||||
|
||||
Handles:
|
||||
- Rate limit statistics and monitoring
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- User override management (whitelist/bypass permissions)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class RateLimitStatsResponse(BaseModel):
|
||||
"""Current rate limit statistics."""
|
||||
# Configuration
|
||||
jobs_per_day_limit: int
|
||||
youtube_uploads_per_day_limit: int
|
||||
beta_ip_per_day_limit: int
|
||||
rate_limiting_enabled: bool
|
||||
|
||||
# Current usage
|
||||
youtube_uploads_today: int
|
||||
youtube_uploads_remaining: int
|
||||
|
||||
# Blocklist stats
|
||||
disposable_domains_count: int
|
||||
blocked_emails_count: int
|
||||
blocked_ips_count: int
|
||||
|
||||
# User override stats
|
||||
total_overrides: int
|
||||
|
||||
|
||||
class UserRateLimitStatusResponse(BaseModel):
|
||||
"""Rate limit status for a specific user."""
|
||||
email: str
|
||||
jobs_today: int
|
||||
jobs_limit: int
|
||||
jobs_remaining: int
|
||||
has_bypass: bool
|
||||
custom_limit: Optional[int]
|
||||
bypass_reason: Optional[str]
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data."""
|
||||
disposable_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
updated_at: Optional[datetime]
|
||||
updated_by: Optional[str]
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
"""Request to add/remove a domain."""
|
||||
domain: str
|
||||
|
||||
|
||||
class EmailRequest(BaseModel):
|
||||
"""Request to add/remove an email."""
|
||||
email: str
|
||||
|
||||
|
||||
class IPRequest(BaseModel):
|
||||
"""Request to add/remove an IP."""
|
||||
ip_address: str
|
||||
|
||||
|
||||
class UserOverride(BaseModel):
|
||||
"""User override configuration."""
|
||||
email: str
|
||||
bypass_job_limit: bool
|
||||
custom_daily_job_limit: Optional[int]
|
||||
reason: str
|
||||
created_by: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class UserOverrideRequest(BaseModel):
|
||||
"""Request to set/update a user override."""
|
||||
bypass_job_limit: bool = False
|
||||
custom_daily_job_limit: Optional[int] = None
|
||||
reason: str
|
||||
|
||||
|
||||
class UserOverridesListResponse(BaseModel):
|
||||
"""List of all user overrides."""
|
||||
overrides: List[UserOverride]
|
||||
total: int
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Statistics Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats", response_model=RateLimitStatsResponse)
|
||||
async def get_rate_limit_stats(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns configuration values, current usage counts, and blocklist stats.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Get YouTube uploads today
|
||||
youtube_today = rate_limit_service.get_youtube_uploads_today()
|
||||
youtube_limit = settings.rate_limit_youtube_uploads_per_day
|
||||
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
|
||||
|
||||
# Get blocklist stats
|
||||
blocklist_stats = email_validation.get_blocklist_stats()
|
||||
|
||||
# Get override count
|
||||
overrides = rate_limit_service.get_all_overrides()
|
||||
|
||||
return RateLimitStatsResponse(
|
||||
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
|
||||
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
|
||||
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
|
||||
rate_limiting_enabled=settings.enable_rate_limiting,
|
||||
youtube_uploads_today=youtube_today,
|
||||
youtube_uploads_remaining=youtube_remaining,
|
||||
disposable_domains_count=blocklist_stats["disposable_domains_count"],
|
||||
blocked_emails_count=blocklist_stats["blocked_emails_count"],
|
||||
blocked_ips_count=blocklist_stats["blocked_ips_count"],
|
||||
total_overrides=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
|
||||
async def get_user_rate_limit_status(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get rate limit status for a specific user.
|
||||
|
||||
Returns their current usage and any override settings.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
jobs_today = rate_limit_service.get_user_job_count_today(email)
|
||||
override = rate_limit_service.get_user_override(email)
|
||||
|
||||
if override:
|
||||
has_bypass = override.get("bypass_job_limit", False)
|
||||
custom_limit = override.get("custom_daily_job_limit")
|
||||
bypass_reason = override.get("reason")
|
||||
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
|
||||
else:
|
||||
has_bypass = False
|
||||
custom_limit = None
|
||||
bypass_reason = None
|
||||
jobs_limit = settings.rate_limit_jobs_per_day
|
||||
|
||||
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
|
||||
|
||||
return UserRateLimitStatusResponse(
|
||||
email=email,
|
||||
jobs_today=jobs_today,
|
||||
jobs_limit=jobs_limit,
|
||||
jobs_remaining=jobs_remaining,
|
||||
has_bypass=has_bypass,
|
||||
custom_limit=custom_limit,
|
||||
bypass_reason=bypass_reason,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/blocklists", response_model=BlocklistsResponse)
|
||||
async def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get all blocklist data.
|
||||
|
||||
Returns disposable domains, blocked emails, and blocked IPs.
|
||||
"""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Force refresh to get latest data
|
||||
config = email_validation.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Get the raw document for metadata
|
||||
from google.cloud import firestore
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
|
||||
|
||||
db = get_firestore_client()
|
||||
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
|
||||
|
||||
updated_at = None
|
||||
updated_by = None
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
updated_at = data.get("updated_at")
|
||||
updated_by = data.get("updated_by")
|
||||
|
||||
return BlocklistsResponse(
|
||||
disposable_domains=sorted(list(config["disposable_domains"])),
|
||||
blocked_emails=sorted(list(config["blocked_emails"])),
|
||||
blocked_ips=sorted(list(config["blocked_ips"])),
|
||||
updated_at=updated_at,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
async def add_disposable_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail="Invalid domain format")
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' added to disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_disposable_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' removed from disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
|
||||
async def add_blocked_email(
|
||||
request: EmailRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an email to the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' added to blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
|
||||
async def remove_blocked_email(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Email not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' removed from blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
|
||||
async def add_blocked_ip(
|
||||
request: IPRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail="Invalid IP address")
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' added to blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
|
||||
async def remove_blocked_ip(
|
||||
ip_address: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="IP not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' removed from blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# User Override Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/overrides", response_model=UserOverridesListResponse)
|
||||
async def get_all_overrides(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all user overrides."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
overrides_data = rate_limit_service.get_all_overrides()
|
||||
|
||||
overrides = [
|
||||
UserOverride(
|
||||
email=email,
|
||||
bypass_job_limit=data.get("bypass_job_limit", False),
|
||||
custom_daily_job_limit=data.get("custom_daily_job_limit"),
|
||||
reason=data.get("reason", ""),
|
||||
created_by=data.get("created_by", "unknown"),
|
||||
created_at=data.get("created_at", datetime.now(timezone.utc)),
|
||||
)
|
||||
for email, data in overrides_data.items()
|
||||
]
|
||||
|
||||
return UserOverridesListResponse(
|
||||
overrides=overrides,
|
||||
total=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def set_user_override(
|
||||
email: str,
|
||||
request: UserOverrideRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Set or update a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
if not request.reason or len(request.reason.strip()) < 3:
|
||||
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
|
||||
|
||||
rate_limit_service.set_user_override(
|
||||
user_email=email,
|
||||
bypass_job_limit=request.bypass_job_limit,
|
||||
custom_daily_job_limit=request.custom_daily_job_limit,
|
||||
reason=request.reason,
|
||||
admin_email=auth_result.user_email,
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override set for user '{email}'"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def remove_user_override(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Override not found for user")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override removed for user '{email}'"
|
||||
)
|
||||
+147
-49
@@ -27,13 +27,15 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
from lyrics_transcriber.core.config import OutputConfig
|
||||
from lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
|
||||
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
|
||||
# Import from the unified style loader
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
@@ -230,9 +232,18 @@ async def complete_review(
|
||||
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
storage.upload_json(corrections_gcs_path, updated_data)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Saved updated corrections")
|
||||
|
||||
|
||||
# Clear worker progress keys to ensure workers will run fresh (not skip due to idempotency)
|
||||
# This handles cases where a job is re-reviewed after completion or where state was inconsistent.
|
||||
# The idempotency check in workers looks at {worker}_progress.stage == 'complete' to skip,
|
||||
# so we must clear these before triggering the render-video worker.
|
||||
progress_keys = ['render_progress', 'screens_progress', 'video_progress', 'encoding_progress']
|
||||
cleared = job_manager.delete_state_data_keys(job_id, progress_keys)
|
||||
if cleared:
|
||||
logger.info(f"Job {job_id}: Cleared worker progress keys for re-processing: {cleared}")
|
||||
|
||||
# Transition to REVIEW_COMPLETE
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -240,18 +251,18 @@ async def complete_review(
|
||||
progress=70,
|
||||
message="Review complete, rendering video with corrected lyrics"
|
||||
)
|
||||
|
||||
|
||||
# Trigger render video worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
|
||||
|
||||
# Run in background, keep reference to prevent garbage collection
|
||||
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Review complete, triggered render video worker")
|
||||
|
||||
|
||||
return {"status": "success"}
|
||||
|
||||
except Exception as e:
|
||||
@@ -385,28 +396,38 @@ async def generate_preview_video(
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
|
||||
# Check if user wants theme background image (slower) or black background (faster, default)
|
||||
use_background_image = updated_data.get("use_background_image", False)
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE: {use_gce_preview}, background image: {use_background_image})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
@@ -416,19 +437,19 @@ async def generate_preview_video(
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
@@ -439,54 +460,131 @@ async def generate_preview_video(
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
# Only use background image if user explicitly requested it
|
||||
# Default is black background for faster preview generation (~10s vs ~30-60s)
|
||||
background_image_gcs_path = None
|
||||
if use_background_image:
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
gce_span.set_attribute("use_background_image", use_background_image)
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""
|
||||
Tenant API routes for white-label portal configuration.
|
||||
|
||||
These endpoints are public (no auth required) since the frontend needs
|
||||
to fetch tenant branding before the user logs in.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
|
||||
|
||||
|
||||
@router.get("/config", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config(
|
||||
request: Request,
|
||||
tenant: Optional[str] = Query(
|
||||
None,
|
||||
description="Tenant ID override (for development). If not provided, detected from subdomain.",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Get tenant configuration for the current portal.
|
||||
|
||||
This endpoint detects the tenant from:
|
||||
1. Query parameter `tenant` (for development/testing)
|
||||
2. X-Tenant-ID header (set by frontend)
|
||||
3. Host header subdomain
|
||||
|
||||
Returns the public tenant configuration (branding, features, defaults)
|
||||
or indicates this is the default Nomad Karaoke portal.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
|
||||
# Priority 1: Query parameter (for dev/testing)
|
||||
tenant_id = tenant
|
||||
|
||||
# Priority 2: X-Tenant-ID header
|
||||
if not tenant_id:
|
||||
tenant_id = request.headers.get("X-Tenant-ID")
|
||||
|
||||
# Priority 3: Detect from Host header
|
||||
if not tenant_id:
|
||||
host = request.headers.get("Host", "")
|
||||
# Check if this is a tenant subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
if host and "nomadkaraoke.com" in host.lower():
|
||||
parts = host.lower().split(".")
|
||||
# Skip known non-tenant subdomains
|
||||
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
|
||||
potential_tenant = parts[0]
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
tenant_id = potential_tenant
|
||||
|
||||
# If no tenant detected, return default config
|
||||
if not tenant_id:
|
||||
logger.debug("No tenant detected, returning default config")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Load tenant config
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
logger.warning(f"Tenant not found: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
logger.info(f"Returning config for tenant: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config_by_id(tenant_id: str):
|
||||
"""
|
||||
Get tenant configuration by explicit tenant ID.
|
||||
|
||||
This is useful for admin tools or debugging.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Check if tenant is active
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/asset/{tenant_id}/{asset_name}")
|
||||
async def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
This redirects to the signed GCS URL for the asset.
|
||||
"""
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
tenant_service = get_tenant_service()
|
||||
url = tenant_service.get_asset_url(tenant_id, asset_name)
|
||||
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail="Asset not found")
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
+543
-51
@@ -8,8 +8,28 @@ Handles:
|
||||
- Stripe checkout and webhooks
|
||||
- Admin user management
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
from typing import Optional, Tuple
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to avoid PII exposure.
|
||||
|
||||
Example: test@example.com -> te***@ex***.com
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
masked_local = local[:2] + "***" if len(local) > 2 else "***"
|
||||
domain_parts = domain.split(".")
|
||||
if len(domain_parts) >= 2:
|
||||
masked_domain = domain_parts[0][:2] + "***." + domain_parts[-1]
|
||||
else:
|
||||
masked_domain = "***"
|
||||
return f"{masked_local}@{masked_domain}"
|
||||
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Depends, Request, Header
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
@@ -32,8 +52,15 @@ from backend.models.user import (
|
||||
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
|
||||
from backend.services.email_service import get_email_service, EmailService
|
||||
from backend.services.stripe_service import get_stripe_service, StripeService, CREDIT_PACKAGES
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.job_defaults_service import (
|
||||
get_effective_distribution_settings,
|
||||
resolve_cdg_txt_defaults,
|
||||
)
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -57,6 +84,16 @@ class CreateCheckoutResponse(BaseModel):
|
||||
message: str
|
||||
|
||||
|
||||
class MadeForYouCheckoutRequest(BaseModel):
|
||||
"""Request to create a made-for-you karaoke video order."""
|
||||
email: EmailStr
|
||||
artist: str
|
||||
title: str
|
||||
source_type: str = "search" # search, youtube, or upload
|
||||
youtube_url: Optional[str] = None
|
||||
notes: Optional[str] = None
|
||||
|
||||
|
||||
class CreditPackage(BaseModel):
|
||||
"""Credit package information."""
|
||||
id: str
|
||||
@@ -99,7 +136,11 @@ async def send_magic_link(
|
||||
|
||||
The user will receive an email with a link that logs them in.
|
||||
Links expire after 15 minutes and can only be used once.
|
||||
|
||||
For white-label tenants, validates email domain against tenant config.
|
||||
"""
|
||||
from backend.middleware.tenant import get_tenant_from_request, get_tenant_config_from_request
|
||||
|
||||
# Check if email service is configured
|
||||
if not email_service.is_configured():
|
||||
logger.error("Email service not configured - cannot send magic links")
|
||||
@@ -110,19 +151,39 @@ async def send_magic_link(
|
||||
|
||||
email = request.email.lower()
|
||||
|
||||
# Get tenant context from middleware
|
||||
tenant_id = get_tenant_from_request(http_request)
|
||||
tenant_config = get_tenant_config_from_request(http_request)
|
||||
|
||||
# Validate email domain for tenant if configured
|
||||
if tenant_config and tenant_config.auth.allowed_email_domains:
|
||||
if not tenant_config.is_email_allowed(email):
|
||||
logger.warning(f"Email domain not allowed for tenant {tenant_id}: {_mask_email(email)}")
|
||||
# Return success anyway to prevent email enumeration
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
|
||||
# Get client info for security logging
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
user_agent = http_request.headers.get("user-agent")
|
||||
|
||||
# Create magic link token
|
||||
# Create magic link token with tenant context
|
||||
magic_link = user_service.create_magic_link(
|
||||
email,
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id
|
||||
)
|
||||
|
||||
# Send email
|
||||
sent = email_service.send_magic_link(email, magic_link.token)
|
||||
# Get tenant-specific sender email
|
||||
sender_email = None
|
||||
if tenant_config:
|
||||
sender_email = tenant_config.get_sender_email()
|
||||
|
||||
# Send email (with tenant-specific sender if configured)
|
||||
sent = email_service.send_magic_link(email, magic_link.token, sender_email=sender_email)
|
||||
|
||||
if not sent:
|
||||
logger.error(f"Failed to send magic link email to {email}")
|
||||
@@ -146,18 +207,26 @@ async def verify_magic_link(
|
||||
Verify a magic link token and create a session.
|
||||
|
||||
Returns a session token that should be stored and used for subsequent requests.
|
||||
The session will be associated with the tenant from the magic link.
|
||||
"""
|
||||
# Reject empty tokens early to avoid invalid Firestore document paths
|
||||
if not token or not token.strip():
|
||||
raise HTTPException(status_code=401, detail="Invalid token")
|
||||
|
||||
# Get client info
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
user_agent = http_request.headers.get("user-agent")
|
||||
|
||||
# Check if this is a first login BEFORE verification (which sets last_login_at)
|
||||
# We need to get the user's state before verify_magic_link updates it
|
||||
# Also extract tenant_id from the magic link for session creation
|
||||
from backend.services.user_service import MAGIC_LINKS_COLLECTION
|
||||
magic_link_doc = user_service.db.collection(MAGIC_LINKS_COLLECTION).document(token).get()
|
||||
is_first_login = False
|
||||
magic_link_tenant_id = None
|
||||
if magic_link_doc.exists:
|
||||
magic_link_data = magic_link_doc.to_dict()
|
||||
magic_link_tenant_id = magic_link_data.get('tenant_id')
|
||||
pre_verify_user = user_service.get_user(magic_link_data.get('email', ''))
|
||||
if pre_verify_user:
|
||||
is_first_login = pre_verify_user.total_jobs_created == 0 and not pre_verify_user.last_login_at
|
||||
@@ -168,18 +237,19 @@ async def verify_magic_link(
|
||||
if not success or not user:
|
||||
raise HTTPException(status_code=401, detail=message)
|
||||
|
||||
# Create session
|
||||
# Create session with tenant context from the magic link
|
||||
session = user_service.create_session(
|
||||
user.email,
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent
|
||||
user_agent=user_agent,
|
||||
tenant_id=magic_link_tenant_id
|
||||
)
|
||||
|
||||
# Send welcome email to first-time users
|
||||
if is_first_login:
|
||||
email_service.send_welcome_email(user.email, user.credits)
|
||||
|
||||
# Return user info
|
||||
# Return user info with tenant_id
|
||||
user_public = UserPublic(
|
||||
email=user.email,
|
||||
role=user.role,
|
||||
@@ -187,6 +257,7 @@ async def verify_magic_link(
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
tenant_id=user.tenant_id,
|
||||
)
|
||||
|
||||
return VerifyMagicLinkResponse(
|
||||
@@ -327,10 +398,328 @@ async def create_checkout(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/made-for-you/checkout", response_model=CreateCheckoutResponse)
|
||||
async def create_made_for_you_checkout(
|
||||
request: MadeForYouCheckoutRequest,
|
||||
stripe_service: StripeService = Depends(get_stripe_service),
|
||||
):
|
||||
"""
|
||||
Create a Stripe checkout session for a made-for-you karaoke video order.
|
||||
|
||||
This is the full-service option where Nomad Karaoke handles everything:
|
||||
- Finding or processing the audio
|
||||
- Reviewing and correcting lyrics
|
||||
- Selecting the best instrumental
|
||||
- Generating the final video
|
||||
|
||||
$15 with 24-hour delivery guarantee.
|
||||
No authentication required - customer email is provided in the request.
|
||||
"""
|
||||
if not stripe_service.is_configured():
|
||||
raise HTTPException(status_code=503, detail="Payment processing is not available")
|
||||
|
||||
success, checkout_url, message = stripe_service.create_made_for_you_checkout_session(
|
||||
customer_email=request.email,
|
||||
artist=request.artist,
|
||||
title=request.title,
|
||||
source_type=request.source_type,
|
||||
youtube_url=request.youtube_url,
|
||||
notes=request.notes,
|
||||
)
|
||||
|
||||
if not success or not checkout_url:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return CreateCheckoutResponse(
|
||||
status="success",
|
||||
checkout_url=checkout_url,
|
||||
message=message,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Stripe Webhooks
|
||||
# =============================================================================
|
||||
|
||||
# Admin email for made-for-you order notifications
|
||||
ADMIN_EMAIL = "madeforyou@nomadkaraoke.com"
|
||||
|
||||
|
||||
async def _handle_made_for_you_order(
|
||||
session_id: str,
|
||||
metadata: dict,
|
||||
user_service: UserService,
|
||||
email_service: EmailService,
|
||||
) -> None:
|
||||
"""
|
||||
Handle a completed made-for-you order by creating a job and notifying Andrew.
|
||||
|
||||
The made-for-you flow:
|
||||
1. Job is created with made_for_you=True, owned by admin during processing
|
||||
2. For search orders: audio search runs, results stored, job pauses at AWAITING_AUDIO_SELECTION
|
||||
3. Admin receives notification email with link to select audio source
|
||||
4. Customer receives order confirmation email
|
||||
5. Admin selects audio in UI, job proceeds through normal pipeline
|
||||
6. On completion, ownership transfers to customer
|
||||
|
||||
For orders with a YouTube URL, workers are triggered immediately (no search needed).
|
||||
|
||||
Args:
|
||||
session_id: Stripe checkout session ID
|
||||
metadata: Order metadata from Stripe session
|
||||
user_service: User service for marking session processed
|
||||
email_service: Email service for notifications
|
||||
"""
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.audio_search_service import (
|
||||
get_audio_search_service,
|
||||
NoResultsError,
|
||||
AudioSearchError,
|
||||
)
|
||||
from backend.services.storage_service import StorageService
|
||||
import asyncio
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
customer_email = metadata.get("customer_email", "")
|
||||
artist = metadata.get("artist", "Unknown Artist")
|
||||
title = metadata.get("title", "Unknown Title")
|
||||
source_type = metadata.get("source_type", "search")
|
||||
youtube_url = metadata.get("youtube_url")
|
||||
notes = metadata.get("notes", "")
|
||||
|
||||
logger.info(
|
||||
f"Processing made-for-you order: {artist} - {title} for {customer_email} "
|
||||
f"(session: {session_id}, source_type: {source_type})"
|
||||
)
|
||||
|
||||
try:
|
||||
job_manager = JobManager()
|
||||
worker_service = get_worker_service()
|
||||
storage_service = StorageService()
|
||||
|
||||
# Apply default theme (Nomad) - same as audio_search endpoint
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme '{effective_theme_id}' for made-for-you order")
|
||||
|
||||
# Get distribution defaults using centralized service
|
||||
dist = get_effective_distribution_settings()
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme (uses centralized service)
|
||||
# This ensures made-for-you jobs get the same CDG/TXT defaults as regular jobs
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(effective_theme_id)
|
||||
|
||||
# Create job with admin ownership during processing
|
||||
# CRITICAL: made_for_you=True, user_email=ADMIN_EMAIL, customer_email for delivery
|
||||
# auto_download=False to pause at audio selection for admin to choose
|
||||
job_create = JobCreate(
|
||||
url=youtube_url if youtube_url else None,
|
||||
artist=artist,
|
||||
title=title,
|
||||
user_email=ADMIN_EMAIL, # Admin owns during processing
|
||||
theme_id=effective_theme_id, # Apply default theme
|
||||
non_interactive=False, # Admin will review lyrics/instrumental
|
||||
# Made-for-you specific fields
|
||||
made_for_you=True, # Flag for ownership transfer on completion
|
||||
customer_email=customer_email, # Customer email for final delivery
|
||||
customer_notes=notes if notes else None, # Customer's special requests
|
||||
# Audio search fields for search-based orders
|
||||
audio_search_artist=artist if not youtube_url else None,
|
||||
audio_search_title=title if not youtube_url else None,
|
||||
auto_download=False, # Pause at audio selection for admin to choose
|
||||
# CDG/TXT settings (resolved via centralized service)
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
# Distribution settings from centralized defaults
|
||||
enable_youtube_upload=dist.enable_youtube_upload,
|
||||
dropbox_path=dist.dropbox_path,
|
||||
gdrive_folder_id=dist.gdrive_folder_id,
|
||||
brand_prefix=dist.brand_prefix,
|
||||
discord_webhook_url=dist.discord_webhook_url,
|
||||
youtube_description=dist.youtube_description,
|
||||
)
|
||||
# Made-for-you jobs are created by admin (via Stripe webhook) - bypass rate limits
|
||||
job = job_manager.create_job(job_create, is_admin=True)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created made-for-you job {job_id} for {_mask_email(customer_email)} (owned by {_mask_email(ADMIN_EMAIL)})")
|
||||
|
||||
# Prepare theme style assets for the job (same as audio_search endpoint)
|
||||
if effective_theme_id:
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, effective_theme_id, None # No color overrides for made-for-you
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to made-for-you job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme for made-for-you job {job_id}: {e}")
|
||||
|
||||
# Mark session as processed for idempotency
|
||||
# Note: Using internal method since this isn't a credit transaction
|
||||
user_service._mark_stripe_session_processed(
|
||||
stripe_session_id=session_id,
|
||||
email=customer_email,
|
||||
amount=0 # No credits, just tracking the session
|
||||
)
|
||||
|
||||
# Initialize search_results for later use in email notification
|
||||
search_results = None
|
||||
|
||||
# Handle based on whether we have a YouTube URL or need to search
|
||||
if youtube_url:
|
||||
# URL provided - trigger workers directly (no audio selection needed)
|
||||
logger.info(f"Job {job_id}: YouTube URL provided, triggering workers")
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
else:
|
||||
# No URL - use audio search flow, pause for admin selection
|
||||
# Made-for-you jobs require admin to select audio source
|
||||
logger.info(f"Job {job_id}: No URL, using audio search for '{artist} - {title}'")
|
||||
|
||||
# Update job with audio search fields
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_search_artist': artist,
|
||||
'audio_search_title': title,
|
||||
'auto_download': False, # Admin must select
|
||||
})
|
||||
|
||||
# Transition to searching state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.SEARCHING_AUDIO,
|
||||
progress=5,
|
||||
message=f"Searching for audio: {artist} - {title}"
|
||||
)
|
||||
|
||||
# Perform audio search
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
try:
|
||||
search_results = audio_search_service.search(artist, title)
|
||||
except NoResultsError as e:
|
||||
# No results found - still transition to AWAITING_AUDIO_SELECTION
|
||||
# Admin can manually provide audio
|
||||
logger.warning(f"Job {job_id}: No audio sources found for '{artist} - {title}'")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"No automatic audio sources found. Manual intervention required."
|
||||
)
|
||||
search_results = None
|
||||
except AudioSearchError as e:
|
||||
logger.error(f"Job {job_id}: Audio search failed: {e}")
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"Audio search error. Manual intervention required."
|
||||
)
|
||||
search_results = None
|
||||
|
||||
if search_results:
|
||||
# Store search results in state_data for admin to review
|
||||
results_dicts = [r.to_dict() for r in search_results]
|
||||
state_data_update = {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
if audio_search_service.last_remote_search_id:
|
||||
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
|
||||
job_manager.update_job(job_id, {'state_data': state_data_update})
|
||||
|
||||
# Transition to AWAITING_AUDIO_SELECTION for admin to choose
|
||||
# Do NOT auto-select or download - admin must review and select
|
||||
logger.info(f"Job {job_id}: Found {len(results_dicts)} audio sources, awaiting admin selection")
|
||||
|
||||
# Transition to AWAITING_AUDIO_SELECTION for admin to choose
|
||||
# Admin will select from results in the UI, then job proceeds
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"Found {len(results_dicts)} audio sources. Awaiting admin selection."
|
||||
)
|
||||
|
||||
# Get audio source count for admin notification
|
||||
# (search_results may be set from the search flow above, or None for YouTube URL orders)
|
||||
audio_source_count = len(search_results) if search_results else 0
|
||||
|
||||
# Generate admin login token for one-click email access (24hr expiry)
|
||||
admin_login = user_service.create_admin_login_token(
|
||||
email=ADMIN_EMAIL,
|
||||
expiry_hours=24,
|
||||
)
|
||||
|
||||
# Send confirmation email to customer using professional template
|
||||
email_service.send_made_for_you_order_confirmation(
|
||||
to_email=customer_email,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
notes=notes,
|
||||
)
|
||||
|
||||
# Send notification email to admin using professional template
|
||||
email_service.send_made_for_you_admin_notification(
|
||||
to_email=ADMIN_EMAIL,
|
||||
customer_email=customer_email,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
admin_login_token=admin_login.token,
|
||||
notes=notes,
|
||||
audio_source_count=audio_source_count,
|
||||
)
|
||||
|
||||
logger.info(f"Sent made-for-you order notifications for job {job_id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error processing made-for-you order: {e}", exc_info=True)
|
||||
# Still try to notify Andrew of the failure
|
||||
try:
|
||||
email_service.send_email(
|
||||
to_email=ADMIN_EMAIL,
|
||||
subject=f"[FAILED] Made For You Order: {artist} - {title}",
|
||||
html_content=f"""
|
||||
<h2>Made-For-You Order Failed</h2>
|
||||
<p>An error occurred processing this order:</p>
|
||||
<ul>
|
||||
<li><strong>Customer:</strong> {customer_email}</li>
|
||||
<li><strong>Artist:</strong> {artist}</li>
|
||||
<li><strong>Title:</strong> {title}</li>
|
||||
<li><strong>Error:</strong> {str(e)}</li>
|
||||
</ul>
|
||||
<p>Please manually create this job and notify the customer.</p>
|
||||
""",
|
||||
text_content=f"""
|
||||
Made-For-You Order Failed
|
||||
|
||||
Customer: {customer_email}
|
||||
Artist: {artist}
|
||||
Title: {title}
|
||||
Error: {str(e)}
|
||||
|
||||
Please manually create this job and notify the customer.
|
||||
""".strip(),
|
||||
)
|
||||
except Exception as email_error:
|
||||
logger.error(f"Failed to send error notification: {email_error}")
|
||||
|
||||
|
||||
@router.post("/webhooks/stripe")
|
||||
async def stripe_webhook(
|
||||
request: Request,
|
||||
@@ -365,30 +754,41 @@ async def stripe_webhook(
|
||||
if event_type == "checkout.session.completed":
|
||||
session = event["data"]["object"]
|
||||
session_id = session.get("id")
|
||||
metadata = session.get("metadata", {})
|
||||
|
||||
# Idempotency check: Skip if this session was already processed
|
||||
if session_id and user_service.is_stripe_session_processed(session_id):
|
||||
logger.info(f"Skipping already processed session: {session_id}")
|
||||
return {"status": "received", "type": event_type, "note": "already_processed"}
|
||||
|
||||
# Process the completed checkout
|
||||
success, user_email, credits, _ = stripe_service.handle_checkout_completed(session)
|
||||
|
||||
if success and user_email and credits > 0:
|
||||
# Add credits to user account
|
||||
ok, new_balance, credit_msg = user_service.add_credits(
|
||||
email=user_email,
|
||||
amount=credits,
|
||||
reason="stripe_purchase",
|
||||
stripe_session_id=session_id,
|
||||
# Check if this is a made-for-you order
|
||||
if metadata.get("order_type") == "made_for_you":
|
||||
# Handle made-for-you order - create a job
|
||||
await _handle_made_for_you_order(
|
||||
session_id=session_id,
|
||||
metadata=metadata,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
)
|
||||
else:
|
||||
# Handle regular credit purchase
|
||||
success, user_email, credits, _ = stripe_service.handle_checkout_completed(session)
|
||||
|
||||
if ok:
|
||||
# Send confirmation email
|
||||
email_service.send_credits_added(user_email, credits, new_balance)
|
||||
logger.info(f"Added {credits} credits to {user_email}, new balance: {new_balance}")
|
||||
else:
|
||||
logger.error(f"Failed to add credits: {credit_msg}")
|
||||
if success and user_email and credits > 0:
|
||||
# Add credits to user account
|
||||
ok, new_balance, credit_msg = user_service.add_credits(
|
||||
email=user_email,
|
||||
amount=credits,
|
||||
reason="stripe_purchase",
|
||||
stripe_session_id=session_id,
|
||||
)
|
||||
|
||||
if ok:
|
||||
# Send confirmation email
|
||||
email_service.send_credits_added(user_email, credits, new_balance)
|
||||
logger.info(f"Added {credits} credits to {user_email}, new balance: {new_balance}")
|
||||
else:
|
||||
logger.error(f"Failed to add credits: {credit_msg}")
|
||||
|
||||
elif event_type == "checkout.session.expired":
|
||||
logger.info(f"Checkout session expired: {event['data']['object'].get('id')}")
|
||||
@@ -423,6 +823,9 @@ async def enroll_beta_tester(
|
||||
|
||||
Returns free credits and optionally a session token for new users.
|
||||
"""
|
||||
from backend.services.email_validation_service import get_email_validation_service
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
|
||||
# Check if email service is configured
|
||||
if not email_service.is_configured():
|
||||
logger.error("Email service not configured - cannot send beta welcome emails")
|
||||
@@ -432,6 +835,63 @@ async def enroll_beta_tester(
|
||||
)
|
||||
|
||||
email = request.email.lower()
|
||||
email_validation = get_email_validation_service()
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
# ----- ABUSE PREVENTION CHECKS -----
|
||||
|
||||
# 1. Validate email (disposable domain, blocked email checks)
|
||||
is_valid, error_message = email_validation.validate_email_for_beta(email)
|
||||
if not is_valid:
|
||||
logger.warning(f"Beta enrollment rejected - email validation failed: {email} - {error_message}")
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=error_message
|
||||
)
|
||||
|
||||
# 2. Check IP blocking
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"Beta enrollment rejected - IP blocked: {ip_address}")
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Access denied from this location"
|
||||
)
|
||||
|
||||
# 3. Check for E2E test bypass (allows automated testing to skip IP rate limit)
|
||||
from backend.config import settings
|
||||
e2e_bypass_key = http_request.headers.get("X-E2E-Bypass-Key")
|
||||
skip_ip_rate_limit = False
|
||||
if e2e_bypass_key and settings.e2e_bypass_key:
|
||||
if e2e_bypass_key == settings.e2e_bypass_key:
|
||||
logger.info(f"Beta enrollment: E2E bypass key validated for {_mask_email(email)}")
|
||||
skip_ip_rate_limit = True
|
||||
else:
|
||||
logger.warning(f"Beta enrollment: Invalid E2E bypass key attempted for {_mask_email(email)}")
|
||||
|
||||
# 4. Check IP-based enrollment rate limit (1 per 24h per IP)
|
||||
if ip_address and not skip_ip_rate_limit:
|
||||
allowed, remaining, message = rate_limit_service.check_beta_ip_limit(ip_address)
|
||||
if not allowed:
|
||||
logger.warning(f"Beta enrollment rejected - IP rate limit: {ip_address} - {message}")
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail="Too many beta enrollments from your location. Please try again tomorrow."
|
||||
)
|
||||
|
||||
# 5. Check for duplicate enrollment via normalized email
|
||||
normalized_email = email_validation.normalize_email(email)
|
||||
if normalized_email != email:
|
||||
# Check if normalized version is already enrolled
|
||||
normalized_user = user_service.get_user(normalized_email)
|
||||
if normalized_user and normalized_user.is_beta_tester:
|
||||
logger.warning(f"Beta enrollment rejected - normalized email already enrolled: {email} -> {normalized_email}")
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="An account with this email address is already enrolled in the beta program"
|
||||
)
|
||||
|
||||
# ----- END ABUSE PREVENTION CHECKS -----
|
||||
|
||||
# Validate acceptance
|
||||
if not request.accept_corrections_work:
|
||||
@@ -456,8 +916,7 @@ async def enroll_beta_tester(
|
||||
detail="You are already enrolled in the beta program"
|
||||
)
|
||||
|
||||
# Get client info
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
# Get client info (ip_address already set above in abuse prevention)
|
||||
user_agent = http_request.headers.get("user-agent")
|
||||
|
||||
# Enroll as beta tester
|
||||
@@ -477,6 +936,13 @@ async def enroll_beta_tester(
|
||||
reason="beta_tester_enrollment",
|
||||
)
|
||||
|
||||
# Record IP enrollment for rate limiting
|
||||
if ip_address:
|
||||
try:
|
||||
rate_limit_service.record_beta_enrollment(ip_address, email)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to record beta IP enrollment: {e}")
|
||||
|
||||
# Create session for the user (so they can start using the service immediately)
|
||||
session = user_service.create_session(email, ip_address=ip_address, user_agent=user_agent)
|
||||
|
||||
@@ -663,6 +1129,7 @@ async def list_users(
|
||||
sort_by: str = "created_at",
|
||||
sort_order: str = "desc",
|
||||
include_inactive: bool = False,
|
||||
exclude_test: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
@@ -676,6 +1143,7 @@ async def list_users(
|
||||
sort_by: Field to sort by (created_at, last_login_at, credits, email)
|
||||
sort_order: Sort direction (asc, desc)
|
||||
include_inactive: Include disabled users
|
||||
exclude_test: If True (default), exclude test users (e.g., @inbox.testmail.app)
|
||||
"""
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
@@ -704,9 +1172,14 @@ async def list_users(
|
||||
else:
|
||||
query = query.order_by("created_at", direction=direction)
|
||||
|
||||
# Get total count (without pagination) for has_more calculation
|
||||
# Get all docs and filter in Python
|
||||
# Note: This is expensive for large datasets, consider caching
|
||||
all_docs = list(query.stream())
|
||||
|
||||
# Filter out test users if exclude_test is True
|
||||
if exclude_test:
|
||||
all_docs = [d for d in all_docs if not is_test_email(d.to_dict().get('email', ''))]
|
||||
|
||||
total_count = len(all_docs)
|
||||
|
||||
# Apply pagination manually (Firestore doesn't support offset well)
|
||||
@@ -942,44 +1415,64 @@ async def list_beta_feedback(
|
||||
|
||||
@router.get("/admin/beta/stats")
|
||||
async def get_beta_stats(
|
||||
exclude_test: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
):
|
||||
"""
|
||||
Get beta tester program statistics (admin only).
|
||||
|
||||
Args:
|
||||
exclude_test: If True (default), exclude test users from beta stats
|
||||
"""
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import aggregation
|
||||
|
||||
# Count beta testers by status using efficient aggregation queries
|
||||
users_collection = user_service.db.collection(USERS_COLLECTION)
|
||||
|
||||
# Helper function to get count using aggregation
|
||||
def get_count(query) -> int:
|
||||
agg_query = aggregation.AggregationQuery(query)
|
||||
agg_query.count(alias="count")
|
||||
results = agg_query.get()
|
||||
return results[0][0].value if results else 0
|
||||
if exclude_test:
|
||||
# Stream and filter in Python since Firestore doesn't support "not ends with"
|
||||
all_beta_users = []
|
||||
for doc in users_collection.where(filter=FieldFilter("is_beta_tester", "==", True)).stream():
|
||||
data = doc.to_dict()
|
||||
if not is_test_email(data.get("email", "")):
|
||||
all_beta_users.append(data)
|
||||
|
||||
total_beta_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
|
||||
)
|
||||
total_beta_testers = len(all_beta_users)
|
||||
active_testers = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "active")
|
||||
pending_feedback = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "pending_feedback")
|
||||
completed_feedback = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "completed")
|
||||
|
||||
active_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "active"))
|
||||
)
|
||||
# Filter feedback by non-test users
|
||||
all_feedback = []
|
||||
for doc in user_service.db.collection("beta_feedback").stream():
|
||||
data = doc.to_dict()
|
||||
if not is_test_email(data.get("user_email", "")):
|
||||
all_feedback.append(data)
|
||||
feedback_docs = all_feedback
|
||||
else:
|
||||
# Use efficient aggregation queries when including test data
|
||||
def get_count(query) -> int:
|
||||
agg_query = aggregation.AggregationQuery(query)
|
||||
agg_query.count(alias="count")
|
||||
results = agg_query.get()
|
||||
return results[0][0].value if results else 0
|
||||
|
||||
pending_feedback = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "pending_feedback"))
|
||||
)
|
||||
|
||||
completed_feedback = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "completed"))
|
||||
)
|
||||
|
||||
# Get average ratings from feedback
|
||||
feedback_docs = list(user_service.db.collection("beta_feedback").stream())
|
||||
total_beta_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
|
||||
)
|
||||
active_testers = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "active"))
|
||||
)
|
||||
pending_feedback = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "pending_feedback"))
|
||||
)
|
||||
completed_feedback = get_count(
|
||||
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "completed"))
|
||||
)
|
||||
feedback_docs = [doc.to_dict() for doc in user_service.db.collection("beta_feedback").stream()]
|
||||
|
||||
# Calculate average ratings from feedback
|
||||
avg_overall = 0
|
||||
avg_ease = 0
|
||||
avg_accuracy = 0
|
||||
@@ -987,8 +1480,7 @@ async def get_beta_stats(
|
||||
|
||||
if feedback_docs:
|
||||
total = len(feedback_docs)
|
||||
for doc in feedback_docs:
|
||||
data = doc.to_dict()
|
||||
for data in feedback_docs:
|
||||
avg_overall += data.get("overall_rating", 0)
|
||||
avg_ease += data.get("ease_of_use_rating", 0)
|
||||
avg_accuracy += data.get("lyrics_accuracy_rating", 0)
|
||||
|
||||
+42
-1
@@ -41,7 +41,15 @@ class Settings(BaseSettings):
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Lyrics Correction Settings
|
||||
# Skip all auto-correction (both agentic AI and heuristic handlers).
|
||||
# When true, raw transcription goes directly to human review without any automatic fixes.
|
||||
# This is the default because auto-correction currently creates more work for reviewers.
|
||||
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
|
||||
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# Only used when skip_correction=false
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
@@ -68,6 +76,12 @@ class Settings(BaseSettings):
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
@@ -95,13 +109,40 @@ class Settings(BaseSettings):
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
|
||||
# Rate Limiting Configuration
|
||||
# Enable/disable rate limiting system-wide (useful for development)
|
||||
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
|
||||
# Maximum jobs a user can create per day (0 = unlimited)
|
||||
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
|
||||
# Maximum YouTube uploads system-wide per day (0 = unlimited)
|
||||
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
|
||||
# Maximum beta enrollments from same IP per day (0 = unlimited)
|
||||
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
|
||||
|
||||
# E2E test bypass key for rate limiting (set via secret in production)
|
||||
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
|
||||
# Default CDG/TXT generation settings
|
||||
# When True, CDG and TXT packages are generated by default (when a theme is set)
|
||||
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
|
||||
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
|
||||
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Push Notifications Configuration
|
||||
# When enabled, users can subscribe to push notifications for job status updates
|
||||
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
|
||||
# Maximum number of push subscriptions per user (oldest removed when exceeded)
|
||||
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
|
||||
# VAPID subject (email or URL for push service to contact)
|
||||
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""
|
||||
Custom exceptions for the karaoke generation backend.
|
||||
|
||||
These exceptions are used for structured error handling across the application,
|
||||
particularly for rate limiting and validation errors.
|
||||
"""
|
||||
|
||||
|
||||
class RateLimitExceededError(Exception):
|
||||
"""
|
||||
Raised when a rate limit is exceeded.
|
||||
|
||||
Includes information about when the limit will reset for client retry logic.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
|
||||
remaining_seconds: Seconds until the limit resets (for Retry-After header)
|
||||
current_count: Current usage count
|
||||
limit_value: The limit that was exceeded
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
limit_type: str = "unknown",
|
||||
remaining_seconds: int = 0,
|
||||
current_count: int = 0,
|
||||
limit_value: int = 0
|
||||
):
|
||||
self.message = message
|
||||
self.limit_type = limit_type
|
||||
self.remaining_seconds = remaining_seconds
|
||||
self.current_count = current_count
|
||||
self.limit_value = limit_value
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class EmailValidationError(Exception):
|
||||
"""
|
||||
Raised when email validation fails.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, reason: str = "invalid"):
|
||||
self.message = message
|
||||
self.reason = reason
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class IPBlockedError(Exception):
|
||||
"""
|
||||
Raised when a request comes from a blocked IP address.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
ip_address: The blocked IP address (may be partially masked)
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, ip_address: str = ""):
|
||||
self.message = message
|
||||
self.ip_address = ip_address
|
||||
super().__init__(message)
|
||||
+74
-4
@@ -7,10 +7,15 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
from backend.services.nltk_preloader import preload_all_nltk_resources
|
||||
from backend.services.langfuse_preloader import preload_langfuse_handler
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware
|
||||
from backend.workers.registry import worker_registry
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
@@ -67,7 +72,28 @@ async def lifespan(app: FastAPI):
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
|
||||
# Preload NLP models and resources to avoid cold start delays
|
||||
# See docs/archive/2026-01-08-performance-investigation.md for background
|
||||
|
||||
# 1. SpaCy model (60+ second delay without preload)
|
||||
try:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
except Exception as e:
|
||||
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
||||
|
||||
# 2. NLTK cmudict (50-100+ second delay without preload)
|
||||
try:
|
||||
preload_all_nltk_resources()
|
||||
except Exception as e:
|
||||
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
||||
|
||||
# 3. Langfuse callback handler (200+ second delay without preload)
|
||||
try:
|
||||
preload_langfuse_handler()
|
||||
except Exception as e:
|
||||
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
@@ -75,8 +101,23 @@ async def lifespan(app: FastAPI):
|
||||
logger.error(f"Credential validation failed: {e}")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
|
||||
# Shutdown - wait for any active workers to complete before terminating
|
||||
# This prevents Cloud Run from killing workers mid-processing
|
||||
logger.info("Shutdown requested, checking for active workers...")
|
||||
if worker_registry.has_active_workers():
|
||||
active = worker_registry.get_active_workers()
|
||||
logger.info(f"Active workers found: {active}")
|
||||
logger.info("Waiting for workers to complete (timeout: 600s)...")
|
||||
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
|
||||
if not completed:
|
||||
logger.error(
|
||||
"Shutdown timeout - some workers may not have completed cleanly. "
|
||||
f"Remaining workers: {worker_registry.get_active_workers()}"
|
||||
)
|
||||
else:
|
||||
logger.info("No active workers, proceeding with shutdown")
|
||||
|
||||
logger.info("Shutting down karaoke generation backend")
|
||||
|
||||
|
||||
@@ -104,6 +145,9 @@ app.add_middleware(
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Add tenant detection middleware (extracts tenant from subdomain/headers)
|
||||
app.add_middleware(TenantMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
@@ -115,6 +159,32 @@ app.include_router(audio_search.router, prefix="/api") # Audio search (artist+t
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
||||
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
|
||||
|
||||
# Exception handler for rate limiting
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
|
||||
|
||||
@app.exception_handler(RateLimitExceededError)
|
||||
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
|
||||
"""Handle rate limit exceeded errors with 429 status."""
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"detail": exc.message,
|
||||
"limit_type": exc.limit_type,
|
||||
"current_count": exc.current_count,
|
||||
"limit_value": exc.limit_value,
|
||||
},
|
||||
headers={
|
||||
"Retry-After": str(exc.remaining_seconds),
|
||||
} if exc.remaining_seconds > 0 else None,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
@@ -1,5 +1,11 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
|
||||
|
||||
__all__ = ["AuditLoggingMiddleware"]
|
||||
__all__ = [
|
||||
"AuditLoggingMiddleware",
|
||||
"TenantMiddleware",
|
||||
"get_tenant_from_request",
|
||||
"get_tenant_config_from_request",
|
||||
]
|
||||
@@ -0,0 +1,192 @@
|
||||
"""
|
||||
Tenant detection middleware for white-label B2B portals.
|
||||
|
||||
This middleware extracts the tenant ID from incoming requests and attaches
|
||||
the tenant configuration to the request state for use by downstream handlers.
|
||||
|
||||
Tenant detection priority:
|
||||
1. X-Tenant-ID header (explicitly set by frontend)
|
||||
2. `tenant` query parameter (for development/testing)
|
||||
3. Host header subdomain detection (production flow)
|
||||
|
||||
The middleware is non-blocking - if no tenant is detected, the request
|
||||
proceeds as a default Nomad Karaoke request (tenant_id = None).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.models.tenant import TenantConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Only allow query param tenant override in non-production environments
|
||||
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
|
||||
os.environ.get("ENVIRONMENT", "").lower() == "production"
|
||||
|
||||
|
||||
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
|
||||
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
|
||||
|
||||
|
||||
class TenantMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that detects tenant from request and attaches config to state.
|
||||
|
||||
After this middleware runs, routes can access:
|
||||
- request.state.tenant_id: str or None
|
||||
- request.state.tenant_config: TenantConfig or None
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = getattr(request.state, "tenant_id", None)
|
||||
tenant_config = getattr(request.state, "tenant_config", None)
|
||||
```
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Extract tenant ID from request
|
||||
tenant_id = self._extract_tenant_id(request)
|
||||
|
||||
# Load tenant config if tenant detected
|
||||
tenant_config: Optional[TenantConfig] = None
|
||||
if tenant_id:
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(tenant_id)
|
||||
|
||||
if tenant_config and not tenant_config.is_active:
|
||||
# Tenant exists but is inactive - treat as default
|
||||
logger.warning(f"Inactive tenant requested: {tenant_id}")
|
||||
tenant_id = None
|
||||
tenant_config = None
|
||||
|
||||
# Attach to request state
|
||||
request.state.tenant_id = tenant_id
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log tenant detection for debugging
|
||||
if tenant_id:
|
||||
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
|
||||
|
||||
# Process request
|
||||
response = await call_next(request)
|
||||
|
||||
# Add tenant ID to response headers for debugging
|
||||
if tenant_id:
|
||||
response.headers["X-Tenant-ID"] = tenant_id
|
||||
|
||||
return response
|
||||
|
||||
def _extract_tenant_id(self, request: Request) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from request using priority-based detection.
|
||||
|
||||
Returns:
|
||||
Tenant ID if detected, None for default Nomad Karaoke
|
||||
"""
|
||||
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
|
||||
header_tenant = request.headers.get("X-Tenant-ID")
|
||||
if header_tenant:
|
||||
return header_tenant.lower().strip()
|
||||
|
||||
# Priority 2: Query parameter (for development/testing only)
|
||||
# Disabled in production to prevent tenant spoofing
|
||||
if not IS_PRODUCTION:
|
||||
query_tenant = request.query_params.get("tenant")
|
||||
if query_tenant:
|
||||
return query_tenant.lower().strip()
|
||||
|
||||
# Priority 3: Host header subdomain detection
|
||||
host = request.headers.get("Host", "")
|
||||
return self._extract_tenant_from_host(host)
|
||||
|
||||
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from Host header subdomain.
|
||||
|
||||
Patterns handled:
|
||||
- vocalstar.nomadkaraoke.com -> vocalstar
|
||||
- vocalstar.gen.nomadkaraoke.com -> vocalstar
|
||||
- localhost:3000 -> None (local dev)
|
||||
- gen.nomadkaraoke.com -> None (main app)
|
||||
- api.nomadkaraoke.com -> None (API)
|
||||
|
||||
Returns:
|
||||
Tenant ID if subdomain matches tenant, None otherwise
|
||||
"""
|
||||
if not host:
|
||||
return None
|
||||
|
||||
# Normalize
|
||||
host_lower = host.lower()
|
||||
|
||||
# Remove port if present
|
||||
if ":" in host_lower:
|
||||
host_lower = host_lower.split(":")[0]
|
||||
|
||||
# Check if this is a nomadkaraoke.com domain
|
||||
if "nomadkaraoke.com" not in host_lower:
|
||||
return None
|
||||
|
||||
# Split into parts
|
||||
parts = host_lower.split(".")
|
||||
|
||||
# Need at least 3 parts: subdomain.nomadkaraoke.com
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
|
||||
# First part is the potential tenant ID
|
||||
potential_tenant = parts[0]
|
||||
|
||||
# Skip known non-tenant subdomains
|
||||
if potential_tenant in NON_TENANT_SUBDOMAINS:
|
||||
return None
|
||||
|
||||
# Verify tenant exists in GCS
|
||||
tenant_service = get_tenant_service()
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
return potential_tenant
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_tenant_from_request(request: Request) -> Optional[str]:
|
||||
"""
|
||||
Helper function to get tenant ID from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_id", None)
|
||||
|
||||
|
||||
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Helper function to get tenant config from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
config = get_tenant_config_from_request(request)
|
||||
if config and not config.features.audio_search:
|
||||
raise HTTPException(403, "Audio search not available")
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_config", None)
|
||||
+44
-6
@@ -9,6 +9,8 @@ from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from karaoke_gen.utils import normalize_text
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
@@ -205,6 +207,9 @@ class Job(BaseModel):
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -274,7 +279,16 @@ class Job(BaseModel):
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Output deletion tracking (for admin cleanup without deleting job)
|
||||
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
|
||||
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
@@ -472,7 +486,12 @@ class JobCreate(BaseModel):
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -484,14 +503,33 @@ class JobCreate(BaseModel):
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
|
||||
# Tenant scoping for white-label portals
|
||||
tenant_id: Optional[str] = None # Tenant ID for job scoping
|
||||
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
"""Validate URL is not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_artist_title(cls, v):
|
||||
"""Normalize artist/title text to standardize Unicode characters.
|
||||
|
||||
This ensures consistent data storage by converting:
|
||||
- Curly quotes -> straight quotes
|
||||
- Various dashes -> hyphen
|
||||
- Unusual whitespace -> regular space
|
||||
"""
|
||||
if v is not None and isinstance(v, str):
|
||||
if not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
# normalize_text handles stripping and Unicode normalization
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
|
||||
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
"""
|
||||
Tenant data models for white-label B2B portals.
|
||||
|
||||
Tenants are B2B customers who get their own branded portal with custom
|
||||
configuration. Each tenant has:
|
||||
- Custom branding (logo, colors, site title)
|
||||
- Feature flags (enable/disable audio search, distribution, etc.)
|
||||
- Default settings (theme, distribution mode)
|
||||
- Auth configuration (allowed email domains, fixed tokens)
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class TenantBranding(BaseModel):
|
||||
"""Branding configuration for a tenant's portal."""
|
||||
|
||||
logo_url: Optional[str] = Field(
|
||||
None, description="URL to tenant's logo image (PNG with transparency preferred)"
|
||||
)
|
||||
logo_height: int = Field(40, description="Logo height in pixels for header display")
|
||||
primary_color: str = Field(
|
||||
"#ff5bb8", description="Primary brand color (hex format)"
|
||||
)
|
||||
secondary_color: str = Field(
|
||||
"#8b5cf6", description="Secondary brand color (hex format)"
|
||||
)
|
||||
accent_color: Optional[str] = Field(
|
||||
None, description="Accent color for highlights (hex format)"
|
||||
)
|
||||
background_color: Optional[str] = Field(
|
||||
None, description="Custom background color (defaults to dark theme)"
|
||||
)
|
||||
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
|
||||
site_title: str = Field(
|
||||
"Karaoke Generator", description="Browser tab title"
|
||||
)
|
||||
tagline: Optional[str] = Field(
|
||||
None, description="Optional tagline shown below logo"
|
||||
)
|
||||
|
||||
|
||||
class TenantFeatures(BaseModel):
|
||||
"""Feature flags controlling what's available in the tenant's portal."""
|
||||
|
||||
# Audio input methods
|
||||
audio_search: bool = Field(
|
||||
True, description="Enable audio search (flacfetch integration)"
|
||||
)
|
||||
file_upload: bool = Field(True, description="Enable direct file upload")
|
||||
youtube_url: bool = Field(True, description="Enable YouTube URL input")
|
||||
|
||||
# Distribution options
|
||||
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
|
||||
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
|
||||
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
|
||||
|
||||
# Customization options
|
||||
theme_selection: bool = Field(
|
||||
True, description="Allow user to select theme (false = use default)"
|
||||
)
|
||||
color_overrides: bool = Field(
|
||||
True, description="Allow user to customize colors"
|
||||
)
|
||||
|
||||
# Output formats
|
||||
enable_cdg: bool = Field(True, description="Generate CDG format output")
|
||||
enable_4k: bool = Field(True, description="Generate 4K video output")
|
||||
|
||||
# Advanced features
|
||||
admin_access: bool = Field(
|
||||
False, description="Allow access to admin dashboard"
|
||||
)
|
||||
|
||||
|
||||
class TenantDefaults(BaseModel):
|
||||
"""Default settings applied to all jobs for this tenant."""
|
||||
|
||||
theme_id: Optional[str] = Field(
|
||||
None, description="Default theme ID (required if theme_selection=false)"
|
||||
)
|
||||
locked_theme: Optional[str] = Field(
|
||||
None,
|
||||
description="If set, users cannot change theme - always uses this theme ID"
|
||||
)
|
||||
distribution_mode: str = Field(
|
||||
"all",
|
||||
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
|
||||
)
|
||||
brand_prefix: Optional[str] = Field(
|
||||
None, description="Prefix for output filenames (e.g., 'VSTAR')"
|
||||
)
|
||||
youtube_description_template: Optional[str] = Field(
|
||||
None, description="Default YouTube description template"
|
||||
)
|
||||
|
||||
|
||||
class TenantAuth(BaseModel):
|
||||
"""Authentication configuration for a tenant."""
|
||||
|
||||
allowed_email_domains: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
|
||||
)
|
||||
require_email_domain: bool = Field(
|
||||
True,
|
||||
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
|
||||
)
|
||||
fixed_token_ids: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
|
||||
)
|
||||
sender_email: Optional[str] = Field(
|
||||
None,
|
||||
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
|
||||
)
|
||||
|
||||
|
||||
class TenantConfig(BaseModel):
|
||||
"""Complete configuration for a white-label tenant."""
|
||||
|
||||
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
|
||||
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
|
||||
subdomain: str = Field(
|
||||
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
|
||||
)
|
||||
is_active: bool = Field(True, description="Whether tenant portal is active")
|
||||
|
||||
branding: TenantBranding = Field(default_factory=TenantBranding)
|
||||
features: TenantFeatures = Field(default_factory=TenantFeatures)
|
||||
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
|
||||
auth: TenantAuth = Field(default_factory=TenantAuth)
|
||||
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
def get_sender_email(self) -> str:
|
||||
"""Get the email sender address for this tenant."""
|
||||
if self.auth.sender_email:
|
||||
return self.auth.sender_email
|
||||
# Default pattern: {tenant_id}@nomadkaraoke.com
|
||||
return f"{self.id}@nomadkaraoke.com"
|
||||
|
||||
def is_email_allowed(self, email: str) -> bool:
|
||||
"""Check if an email address is allowed for this tenant."""
|
||||
if not self.auth.allowed_email_domains:
|
||||
# No domain restrictions
|
||||
return True
|
||||
|
||||
email_lower = email.lower()
|
||||
for domain in self.auth.allowed_email_domains:
|
||||
if email_lower.endswith(f"@{domain.lower()}"):
|
||||
return True
|
||||
|
||||
return not self.auth.require_email_domain
|
||||
|
||||
|
||||
class TenantPublicConfig(BaseModel):
|
||||
"""
|
||||
Public tenant configuration returned to frontend.
|
||||
|
||||
This excludes sensitive auth details like token IDs.
|
||||
"""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
subdomain: str
|
||||
is_active: bool
|
||||
branding: TenantBranding
|
||||
features: TenantFeatures
|
||||
defaults: TenantDefaults
|
||||
# Auth info limited to what frontend needs
|
||||
allowed_email_domains: List[str] = Field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
|
||||
"""Create public config from full config."""
|
||||
return cls(
|
||||
id=config.id,
|
||||
name=config.name,
|
||||
subdomain=config.subdomain,
|
||||
is_active=config.is_active,
|
||||
branding=config.branding,
|
||||
features=config.features,
|
||||
defaults=TenantDefaults(
|
||||
theme_id=config.defaults.theme_id,
|
||||
locked_theme=config.defaults.locked_theme,
|
||||
distribution_mode=config.defaults.distribution_mode,
|
||||
# Don't expose brand_prefix or youtube_description_template
|
||||
),
|
||||
allowed_email_domains=config.auth.allowed_email_domains,
|
||||
)
|
||||
|
||||
|
||||
class TenantConfigResponse(BaseModel):
|
||||
"""Response from GET /api/tenant/config endpoint."""
|
||||
|
||||
tenant: Optional[TenantPublicConfig] = Field(
|
||||
None, description="Tenant config if found, null for default Nomad Karaoke"
|
||||
)
|
||||
is_default: bool = Field(
|
||||
True, description="True if using default Nomad Karaoke config"
|
||||
)
|
||||
+38
-2
@@ -8,9 +8,9 @@ Supports:
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Optional, List
|
||||
from typing import Optional, List, Dict
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
@@ -39,17 +39,37 @@ class CreditTransaction(BaseModel):
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class PushSubscription(BaseModel):
|
||||
"""
|
||||
Web Push subscription for a user's device.
|
||||
|
||||
Stores the push subscription endpoint and encryption keys needed
|
||||
to send push notifications to the user's browser/device.
|
||||
"""
|
||||
endpoint: str # Push service endpoint URL
|
||||
keys: Dict[str, str] # p256dh and auth keys for encryption
|
||||
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
last_used_at: Optional[datetime] = None # Last time a notification was sent
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
|
||||
For multi-tenant white-label portals, users are scoped to a tenant.
|
||||
Users with tenant_id=None are default Nomad Karaoke users.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
@@ -80,6 +100,10 @@ class User(BaseModel):
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
# Push notification subscriptions (Web Push API)
|
||||
# Users can subscribe from multiple devices/browsers
|
||||
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
@@ -87,6 +111,9 @@ class MagicLinkToken(BaseModel):
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
|
||||
For multi-tenant portals, the tenant_id determines which portal
|
||||
the user will be redirected to after verification.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
@@ -97,6 +124,9 @@ class MagicLinkToken(BaseModel):
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
@@ -104,6 +134,8 @@ class Session(BaseModel):
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
|
||||
For multi-tenant portals, sessions are scoped to a tenant.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
@@ -114,6 +146,9 @@ class Session(BaseModel):
|
||||
user_agent: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
@@ -149,6 +184,7 @@ class UserPublic(BaseModel):
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
|
||||
@@ -204,7 +204,9 @@ class AudioSearchService:
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
source_id=item.get("info_hash"),
|
||||
# source_id: use explicit source_id field first (YouTube video ID, Spotify track ID),
|
||||
# fall back to info_hash for torrent sources (RED/OPS)
|
||||
source_id=item.get("source_id") or item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
|
||||
@@ -109,30 +109,42 @@ class DropboxService:
|
||||
|
||||
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
|
||||
"""
|
||||
Calculate next brand code from existing folders.
|
||||
Calculate next brand code from existing folders, filling gaps in the sequence.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the next sequential code.
|
||||
returns the first available code, filling any gaps left by deleted outputs.
|
||||
|
||||
Args:
|
||||
path: Dropbox path containing organized folders
|
||||
brand_prefix: Brand prefix (e.g., "NOMAD")
|
||||
|
||||
Returns:
|
||||
Next brand code (e.g., "NOMAD-1164")
|
||||
Next brand code (e.g., "NOMAD-0003" if 0003 is the first gap)
|
||||
"""
|
||||
folders = self.list_folders(path)
|
||||
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
|
||||
|
||||
max_num = 0
|
||||
existing_nums: set[int] = set()
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
num = int(match.group(1))
|
||||
max_num = max(max_num, num)
|
||||
existing_nums.add(int(match.group(1)))
|
||||
|
||||
next_code = f"{brand_prefix}-{max_num + 1:04d}"
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
|
||||
max_existing = max(existing_nums) if existing_nums else 0
|
||||
|
||||
# Only fill gaps for numbers >= 1001 (preserve legacy gaps below 1000)
|
||||
if max_existing >= 1000:
|
||||
next_num = 1001
|
||||
while next_num in existing_nums:
|
||||
next_num += 1
|
||||
else:
|
||||
next_num = max_existing + 1
|
||||
|
||||
next_code = f"{brand_prefix}-{next_num:04d}"
|
||||
if max_existing >= 1000 and next_num <= max_existing:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (filling gap, max existing: {max_existing})")
|
||||
else:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_existing})")
|
||||
return next_code
|
||||
|
||||
def upload_file(self, local_path: str, remote_path: str) -> None:
|
||||
|
||||
+804
-293
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,646 @@
|
||||
"""
|
||||
Email Validation Service.
|
||||
|
||||
Provides email validation, normalization, and blocklist checking
|
||||
to prevent abuse during beta enrollment and other flows.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Tuple, Set
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask an email address for privacy-safe logging."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 2:
|
||||
masked_local = "*" * len(local)
|
||||
else:
|
||||
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
|
||||
return f"{masked_local}@{domain}"
|
||||
|
||||
|
||||
# Firestore collection for blocklist data
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
BLOCKLIST_CONFIG_DOC = "config"
|
||||
|
||||
# Default disposable email domains (can be extended via admin UI)
|
||||
DEFAULT_DISPOSABLE_DOMAINS = {
|
||||
# Popular temporary email services
|
||||
"10minutemail.com",
|
||||
"10minutemail.net",
|
||||
"guerrillamail.com",
|
||||
"guerrillamail.org",
|
||||
"guerrillamail.net",
|
||||
"guerrillamail.biz",
|
||||
"guerrillamailblock.com",
|
||||
"tempmail.com",
|
||||
"tempmail.net",
|
||||
"temp-mail.org",
|
||||
"temp-mail.io",
|
||||
"throwawaymail.com",
|
||||
"mailinator.com",
|
||||
"mailinator.net",
|
||||
"mailinator.org",
|
||||
"mailinator.info",
|
||||
"maildrop.cc",
|
||||
"yopmail.com",
|
||||
"yopmail.fr",
|
||||
"yopmail.net",
|
||||
"fakeinbox.com",
|
||||
"fakemailgenerator.com",
|
||||
"dispostable.com",
|
||||
"getairmail.com",
|
||||
"getnada.com",
|
||||
"mohmal.com",
|
||||
"trashmail.com",
|
||||
"trashmail.net",
|
||||
"trashmail.org",
|
||||
"sharklasers.com",
|
||||
"grr.la",
|
||||
"guerrillamail.de",
|
||||
"pokemail.net",
|
||||
"spam4.me",
|
||||
"spamgourmet.com",
|
||||
"mytrashmail.com",
|
||||
"mailnesia.com",
|
||||
"mailcatch.com",
|
||||
"mintemail.com",
|
||||
"tempr.email",
|
||||
"tempail.com",
|
||||
"emailondeck.com",
|
||||
"incognitomail.com",
|
||||
"inboxalias.com",
|
||||
"33mail.com",
|
||||
"spamex.com",
|
||||
"spamfree24.org",
|
||||
"spamspot.com",
|
||||
"mailnull.com",
|
||||
"mailsac.com",
|
||||
"emailfake.com",
|
||||
"fakemail.fr",
|
||||
"tempinbox.com",
|
||||
"throwaway.email",
|
||||
"burnermail.io",
|
||||
"anonbox.net",
|
||||
"anonymbox.com",
|
||||
"discard.email",
|
||||
"discardmail.com",
|
||||
"discardmail.de",
|
||||
"mailexpire.com",
|
||||
"mailforspam.com",
|
||||
"meltmail.com",
|
||||
"mt2009.com",
|
||||
"mt2014.com",
|
||||
"nospam.ze.tc",
|
||||
"nospamfor.us",
|
||||
"nowmymail.com",
|
||||
"receiveee.com",
|
||||
"safe-mail.net",
|
||||
"spamavert.com",
|
||||
"spambob.com",
|
||||
"spambog.com",
|
||||
"spambox.us",
|
||||
"spamcannon.com",
|
||||
"spamcannon.net",
|
||||
"spamcon.org",
|
||||
"spamcorptastic.com",
|
||||
"spamday.com",
|
||||
"spamfree.eu",
|
||||
"spamherelots.com",
|
||||
"spamhereplease.com",
|
||||
"spamhole.com",
|
||||
"spamify.com",
|
||||
"spaminator.de",
|
||||
"spamkill.info",
|
||||
"spaml.com",
|
||||
"spaml.de",
|
||||
"spamoff.de",
|
||||
"spamobox.com",
|
||||
"spamslicer.com",
|
||||
"spamstack.net",
|
||||
"spamthis.co.uk",
|
||||
"spamthisplease.com",
|
||||
"supergreatmail.com",
|
||||
"suremail.info",
|
||||
"teleworm.us",
|
||||
"tempemail.co.za",
|
||||
"tempemail.net",
|
||||
"tempmailaddress.com",
|
||||
"tempmailo.com",
|
||||
"thankyou2010.com",
|
||||
"thisisnotmyrealemail.com",
|
||||
"tm.slsrs.ru",
|
||||
"tmpeml.info",
|
||||
"trash-mail.at",
|
||||
"trash-mail.de",
|
||||
"trash2009.com",
|
||||
"trashemail.de",
|
||||
"trashmail.at",
|
||||
"trashmailer.com",
|
||||
"wegwerfmail.de",
|
||||
"wegwerfmail.net",
|
||||
"wegwerfmail.org",
|
||||
"wh4f.org",
|
||||
"willhackforfood.biz",
|
||||
"willselfdestruct.com",
|
||||
"xmaily.com",
|
||||
"xyzfree.net",
|
||||
"yep.it",
|
||||
"yogamaven.com",
|
||||
"yuurok.com",
|
||||
"zehnminutenmail.de",
|
||||
"zippymail.info",
|
||||
# Additional common ones
|
||||
"mailnator.com",
|
||||
"bugmenot.com",
|
||||
"dodgeit.com",
|
||||
"dodgit.com",
|
||||
"e4ward.com",
|
||||
"emailsensei.com",
|
||||
"hushmail.com",
|
||||
"jetable.org",
|
||||
"kasmail.com",
|
||||
"mailblock.net",
|
||||
"mailcatch.com",
|
||||
"mymailoasis.com",
|
||||
"nervmich.net",
|
||||
"nervtmansen.de",
|
||||
"oneoffemail.com",
|
||||
"pookmail.com",
|
||||
"shortmail.net",
|
||||
"sneakemail.com",
|
||||
"sogetthis.com",
|
||||
"tempemailaddress.com",
|
||||
"tempomail.fr",
|
||||
"temporaryemail.net",
|
||||
"temporaryforwarding.com",
|
||||
"temporaryinbox.com",
|
||||
"thankyou2010.com",
|
||||
"tyldd.com",
|
||||
"uggsrock.com",
|
||||
"veryrealemail.com",
|
||||
"yourewronghereswhy.com",
|
||||
}
|
||||
|
||||
# Gmail-like domains that support alias normalization
|
||||
GMAIL_LIKE_DOMAINS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
}
|
||||
|
||||
|
||||
class EmailValidationService:
|
||||
"""
|
||||
Service for email validation and abuse prevention.
|
||||
|
||||
Features:
|
||||
- Email normalization (Gmail alias stripping)
|
||||
- Disposable domain detection
|
||||
- Blocklist checking (emails, IPs)
|
||||
- IP-based enrollment rate limiting
|
||||
"""
|
||||
|
||||
_instance: Optional["EmailValidationService"] = None
|
||||
_blocklist_cache: Optional[dict] = None
|
||||
_blocklist_cache_time: Optional[datetime] = None
|
||||
_cache_ttl_seconds: int = 300 # 5 minutes
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize the email validation service."""
|
||||
if db is None:
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
db = get_firestore_client()
|
||||
self.db = db
|
||||
|
||||
@classmethod
|
||||
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
|
||||
"""Get singleton instance of the service."""
|
||||
if cls._instance is None:
|
||||
cls._instance = cls(db)
|
||||
return cls._instance
|
||||
|
||||
def normalize_email(self, email: str) -> str:
|
||||
"""
|
||||
Normalize an email address.
|
||||
|
||||
For Gmail-like domains:
|
||||
- Removes dots from local part
|
||||
- Removes everything after + in local part
|
||||
- Converts to lowercase
|
||||
|
||||
For other domains:
|
||||
- Only converts to lowercase
|
||||
|
||||
Args:
|
||||
email: Email address to normalize
|
||||
|
||||
Returns:
|
||||
Normalized email address
|
||||
|
||||
Examples:
|
||||
j.o.h.n+spam@gmail.com -> john@gmail.com
|
||||
John.Doe@example.com -> john.doe@example.com
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return email.lower().strip() if email else ""
|
||||
|
||||
email = email.lower().strip()
|
||||
local, domain = email.rsplit("@", 1)
|
||||
|
||||
if domain in GMAIL_LIKE_DOMAINS:
|
||||
# Remove dots from local part
|
||||
local = local.replace(".", "")
|
||||
# Remove everything after +
|
||||
if "+" in local:
|
||||
local = local.split("+")[0]
|
||||
|
||||
return f"{local}@{domain}"
|
||||
|
||||
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
|
||||
"""
|
||||
Get blocklist configuration from Firestore.
|
||||
|
||||
Uses caching to reduce Firestore reads.
|
||||
|
||||
Returns:
|
||||
Dict with disposable_domains, blocked_emails, blocked_ips sets
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache validity
|
||||
if (
|
||||
not force_refresh
|
||||
and self._blocklist_cache is not None
|
||||
and self._blocklist_cache_time is not None
|
||||
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
|
||||
):
|
||||
return self._blocklist_cache
|
||||
|
||||
# Fetch from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
config = {
|
||||
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
|
||||
"blocked_emails": set(data.get("blocked_emails", [])),
|
||||
"blocked_ips": set(data.get("blocked_ips", [])),
|
||||
}
|
||||
else:
|
||||
# Use defaults if no config exists
|
||||
config = {
|
||||
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
|
||||
"blocked_emails": set(),
|
||||
"blocked_ips": set(),
|
||||
}
|
||||
|
||||
# Update cache
|
||||
EmailValidationService._blocklist_cache = config
|
||||
EmailValidationService._blocklist_cache_time = now
|
||||
|
||||
return config
|
||||
|
||||
def is_disposable_domain(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email uses a disposable domain.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the domain is known to be disposable
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False
|
||||
|
||||
domain = email.lower().split("@")[-1]
|
||||
config = self.get_blocklist_config()
|
||||
return domain in config["disposable_domains"]
|
||||
|
||||
def is_email_blocked(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is explicitly blocked.
|
||||
|
||||
Checks both the raw email and normalized version.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the email is blocked
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
email_lower = email.lower().strip()
|
||||
email_normalized = self.normalize_email(email)
|
||||
|
||||
return (
|
||||
email_lower in config["blocked_emails"]
|
||||
or email_normalized in config["blocked_emails"]
|
||||
)
|
||||
|
||||
def is_ip_blocked(self, ip_address: str) -> bool:
|
||||
"""
|
||||
Check if an IP address is blocked.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to check
|
||||
|
||||
Returns:
|
||||
True if the IP is blocked
|
||||
"""
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
return ip_address in config["blocked_ips"]
|
||||
|
||||
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
|
||||
"""
|
||||
Validate an email for beta enrollment.
|
||||
|
||||
Performs all validation checks:
|
||||
1. Basic format validation
|
||||
2. Disposable domain check
|
||||
3. Blocked email check
|
||||
|
||||
Args:
|
||||
email: Email address to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_message)
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False, "Invalid email format"
|
||||
|
||||
# Check disposable domain
|
||||
if self.is_disposable_domain(email):
|
||||
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
|
||||
return False, "Disposable email addresses are not allowed"
|
||||
|
||||
# Check blocked email
|
||||
if self.is_email_blocked(email):
|
||||
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
|
||||
return False, "This email address is not allowed"
|
||||
|
||||
return True, ""
|
||||
|
||||
def hash_ip(self, ip_address: str) -> str:
|
||||
"""
|
||||
Hash an IP address for privacy-preserving storage.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to hash
|
||||
|
||||
Returns:
|
||||
SHA-256 hash of the IP
|
||||
"""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management (Admin)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Add an email to the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
if not email:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
else:
|
||||
data = {}
|
||||
emails = set()
|
||||
|
||||
emails.add(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
|
||||
if email not in emails:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
emails.discard(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
else:
|
||||
data = {}
|
||||
ips = set()
|
||||
|
||||
ips.add(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
ips.discard(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def get_blocklist_stats(self) -> dict:
|
||||
"""Get statistics about current blocklists."""
|
||||
config = self.get_blocklist_config(force_refresh=True)
|
||||
return {
|
||||
"disposable_domains_count": len(config["disposable_domains"]),
|
||||
"blocked_emails_count": len(config["blocked_emails"]),
|
||||
"blocked_ips_count": len(config["blocked_ips"]),
|
||||
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
|
||||
}
|
||||
|
||||
|
||||
def get_email_validation_service(
|
||||
db: Optional[firestore.Client] = None,
|
||||
) -> EmailValidationService:
|
||||
"""Get the singleton EmailValidationService instance."""
|
||||
return EmailValidationService.get_instance(db)
|
||||
@@ -40,6 +40,9 @@ class EncodingInput:
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Instrumental selection (clean, with_backing, or custom)
|
||||
instrumental_selection: str = "clean"
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
@@ -197,6 +200,7 @@ class LocalEncodingBackend(EncodingBackend):
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
countdown_padding_seconds=input_config.options.get("countdown_padding_seconds"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
@@ -328,8 +332,15 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
"instrumental_selection": input_config.instrumental_selection,
|
||||
}
|
||||
|
||||
# Add countdown padding if present (for audio sync with countdown-padded vocals)
|
||||
countdown_padding = input_config.options.get("countdown_padding_seconds")
|
||||
if countdown_padding:
|
||||
encoding_config["countdown_padding_seconds"] = countdown_padding
|
||||
self.logger.info(f"GCE encoding with countdown_padding_seconds={countdown_padding}")
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
@@ -351,8 +362,8 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict if needed
|
||||
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
@@ -360,15 +371,18 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
if "4k_lossless" in filename:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "4k_lossy" in filename:
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename:
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
elif filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
@@ -25,6 +25,11 @@ from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts)
|
||||
MAX_RETRIES = 3
|
||||
INITIAL_BACKOFF_SECONDS = 2.0
|
||||
MAX_BACKOFF_SECONDS = 10.0
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
@@ -63,6 +68,86 @@ class EncodingService:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
headers: Dict[str, str],
|
||||
json_payload: Optional[Dict[str, Any]] = None,
|
||||
timeout: float = 30.0,
|
||||
job_id: str = "unknown",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make an HTTP request with retry logic for transient failures.
|
||||
|
||||
This handles connection errors that occur when the GCE worker is
|
||||
restarting (e.g., during deployments) by retrying with exponential backoff.
|
||||
|
||||
Args:
|
||||
method: HTTP method (GET, POST)
|
||||
url: Request URL
|
||||
headers: Request headers
|
||||
json_payload: JSON body for POST requests
|
||||
timeout: Request timeout in seconds
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- status (int): HTTP status code
|
||||
- json (Any): Parsed JSON response body (if status 200, else None)
|
||||
- text (str): Raw response text (if status != 200, else None)
|
||||
|
||||
Raises:
|
||||
aiohttp.ClientConnectorError: If all retries fail due to connection errors
|
||||
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
|
||||
asyncio.TimeoutError: If all retries fail due to timeout
|
||||
"""
|
||||
last_exception = None
|
||||
backoff = INITIAL_BACKOFF_SECONDS
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
if method.upper() == "POST":
|
||||
async with session.post(
|
||||
url, json=json_payload, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
# Return a copy of the response data since we exit the context
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
else: # GET
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
f"Retrying in {backoff:.1f}s..."
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
|
||||
)
|
||||
|
||||
raise last_exception
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -101,17 +186,23 @@ class EncodingService:
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
return await resp.json()
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -131,17 +222,22 @@ class EncodingService:
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
|
||||
resp = await self._request_with_retry(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=headers,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
return await resp.json()
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
@@ -242,6 +338,133 @@ class EncodingService:
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
@@ -116,6 +116,7 @@ class FirestoreService:
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
@@ -128,6 +129,7 @@ class FirestoreService:
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
@@ -150,6 +152,10 @@ class FirestoreService:
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Filter by tenant_id (white-label portal scoping)
|
||||
if tenant_id:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
@@ -510,3 +516,24 @@ class FirestoreService:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_firestore_client: Optional[firestore.Client] = None
|
||||
|
||||
|
||||
def get_firestore_client() -> firestore.Client:
|
||||
"""
|
||||
Get a shared Firestore client instance.
|
||||
|
||||
This returns a raw Firestore client (not the FirestoreService) for use
|
||||
in services that need direct Firestore access without the job-specific
|
||||
abstractions provided by FirestoreService.
|
||||
|
||||
Returns:
|
||||
Firestore client instance
|
||||
"""
|
||||
global _firestore_client
|
||||
if _firestore_client is None:
|
||||
_firestore_client = firestore.Client(project=settings.google_cloud_project)
|
||||
return _firestore_client
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# GCE Encoding Worker
|
||||
|
||||
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/encode` | POST | Submit a full encoding job |
|
||||
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
|
||||
| `/status/{job_id}` | GET | Get job status and progress |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
|
||||
|
||||
## Encoding Process
|
||||
|
||||
### Full Encoding (`/encode`)
|
||||
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
|
||||
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
|
||||
- Lossless 4K MP4
|
||||
- Lossy 4K MP4
|
||||
- Lossless MKV
|
||||
- 720p MP4
|
||||
3. Uploads results to GCS
|
||||
4. Reports progress via status endpoint
|
||||
|
||||
### Preview Encoding (`/encode-preview`)
|
||||
1. Downloads ASS subtitle and audio files from GCS
|
||||
2. Runs FFmpeg to create a quick 480x270 preview video
|
||||
3. Uploads result to GCS
|
||||
4. Used for real-time preview in the web UI
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **FFmpeg**: Static build from John Van Sickle
|
||||
- **Python 3.13**: Built from source (required for karaoke-gen)
|
||||
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
|
||||
|
||||
## Deployment
|
||||
|
||||
This service is deployed via the VM's startup script. The script:
|
||||
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
|
||||
2. Builds Python 3.13 from source
|
||||
3. Creates venv and installs dependencies
|
||||
4. Writes `main.py` to `/opt/encoding-worker/`
|
||||
5. Creates systemd service
|
||||
6. Starts the service on port 8080
|
||||
|
||||
## Local Development
|
||||
|
||||
This code is extracted from the infrastructure for maintainability. To test locally:
|
||||
|
||||
```bash
|
||||
cd backend/services/gce_encoding
|
||||
pip install -r requirements.txt
|
||||
pip install ../../../ # Install karaoke-gen wheel
|
||||
ENCODING_API_KEY=test python main.py
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
|
||||
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
|
||||
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
|
||||
- **GCS integration**: Direct download/upload via google-cloud-storage client
|
||||
@@ -0,0 +1,22 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This package contains the FastAPI application that runs on the GCE encoding worker VM.
|
||||
It provides HTTP endpoints for video encoding jobs.
|
||||
|
||||
The service:
|
||||
- Downloads input files from GCS
|
||||
- Runs FFmpeg encoding via LocalEncodingService
|
||||
- Uploads results back to GCS
|
||||
- Provides job status tracking
|
||||
|
||||
Endpoints:
|
||||
- POST /encode - Submit a full encoding job
|
||||
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
|
||||
- GET /status/{job_id} - Get job status
|
||||
- GET /health - Health check
|
||||
"""
|
||||
|
||||
from .main import app
|
||||
|
||||
__all__ = ["app"]
|
||||
@@ -0,0 +1,610 @@
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Encoding Worker", version="1.0.0")
|
||||
|
||||
# API key authentication
|
||||
API_KEY = os.environ.get("ENCODING_API_KEY", "")
|
||||
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(None)):
|
||||
# Verify API key for authentication
|
||||
if not API_KEY:
|
||||
logger.warning("No API key configured - authentication disabled")
|
||||
return True
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return True
|
||||
|
||||
# Job tracking
|
||||
jobs: dict[str, dict] = {}
|
||||
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
|
||||
|
||||
# GCS client
|
||||
storage_client = storage.Client()
|
||||
|
||||
class EncodeRequest(BaseModel):
|
||||
job_id: str
|
||||
input_gcs_path: str # gs://bucket/path/to/inputs/
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
|
||||
class EncodePreviewRequest(BaseModel):
|
||||
job_id: str
|
||||
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
|
||||
audio_gcs_path: str # gs://bucket/path/to/audio.flac
|
||||
output_gcs_path: str # gs://bucket/path/to/output.mp4
|
||||
background_color: str = "black"
|
||||
background_image_gcs_path: Optional[str] = None
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a file or folder from GCS
|
||||
# Parse gs://bucket/path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blobs = list(bucket.list_blobs(prefix=prefix))
|
||||
|
||||
for blob in blobs:
|
||||
# Get relative path from prefix
|
||||
rel_path = blob.name[len(prefix):].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
dest = local_path / rel_path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(dest))
|
||||
logger.info(f"Downloaded: {blob.name} -> {dest}")
|
||||
|
||||
|
||||
def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a file or folder to GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
|
||||
if local_path.is_file():
|
||||
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
|
||||
else:
|
||||
for file in local_path.rglob("*"):
|
||||
if file.is_file():
|
||||
rel_path = file.relative_to(local_path)
|
||||
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(file))
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a single file from GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(local_path))
|
||||
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
|
||||
|
||||
|
||||
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a single file to a specific GCS path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Download input files
|
||||
ass_path = work_dir / "subtitles.ass"
|
||||
audio_path = work_dir / "audio.flac"
|
||||
|
||||
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 30
|
||||
|
||||
# Download background image if provided
|
||||
bg_image_path = None
|
||||
if request.background_image_gcs_path:
|
||||
bg_image_path = work_dir / "background.png"
|
||||
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
|
||||
|
||||
# Download custom font if provided and register with fontconfig
|
||||
if request.font_gcs_path:
|
||||
# Use standard fontconfig location that's already in the search path
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = request.font_gcs_path.split("/")[-1]
|
||||
font_path = fonts_dir / font_filename
|
||||
download_single_file_from_gcs(request.font_gcs_path, font_path)
|
||||
logger.info(f"Downloaded custom font: {font_path}")
|
||||
# Update fontconfig cache so libass can find the font
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
|
||||
|
||||
# Build FFmpeg command
|
||||
output_path = work_dir / "preview.mp4"
|
||||
|
||||
# Escape special characters in path for FFmpeg filter syntax
|
||||
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
|
||||
def escape_ffmpeg_filter_path(path: str) -> str:
|
||||
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
|
||||
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
|
||||
|
||||
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
|
||||
|
||||
# Base command with frame rate
|
||||
cmd = ["ffmpeg", "-y", "-r", "24"]
|
||||
|
||||
# Video input: background image or solid color
|
||||
if bg_image_path and bg_image_path.exists():
|
||||
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
|
||||
# Scale and pad background to 480x270
|
||||
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
|
||||
else:
|
||||
# Solid color background
|
||||
color = request.background_color or "black"
|
||||
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
|
||||
vf = f"ass={escaped_ass_path}"
|
||||
|
||||
# Audio input
|
||||
cmd.extend(["-i", str(audio_path)])
|
||||
|
||||
# Video filter and encoding settings
|
||||
cmd.extend([
|
||||
"-vf", vf,
|
||||
"-c:a", "aac", "-b:a", "96k",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "superfast",
|
||||
"-crf", "28",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart",
|
||||
"-threads", "8",
|
||||
"-shortest",
|
||||
str(output_path)
|
||||
])
|
||||
|
||||
jobs[job_id]["progress"] = 40
|
||||
logger.info(f"Running preview encoding: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
jobs[job_id]["progress"] = 80
|
||||
logger.info(f"Preview encoded: {output_path}")
|
||||
|
||||
# Upload output to GCS
|
||||
upload_single_file_to_gcs(output_path, request.output_gcs_path)
|
||||
jobs[job_id]["progress"] = 95
|
||||
|
||||
jobs[job_id]["output_files"] = [request.output_gcs_path]
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
'''Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
Called at the start of each job to enable hot code updates without restart.
|
||||
In-progress jobs continue with their version, new jobs get latest code.
|
||||
'''
|
||||
try:
|
||||
logger.info("Checking for latest karaoke-gen wheel in GCS...")
|
||||
|
||||
# Download latest wheel
|
||||
result = subprocess.run(
|
||||
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
# Use 5-minute timeout - first install at job start may need to resolve dependencies
|
||||
# Subsequent installs are faster since dependencies are cached
|
||||
install_result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
|
||||
capture_output=True, text=True, timeout=300
|
||||
)
|
||||
|
||||
if install_result.returncode != 0:
|
||||
logger.warning(f"Wheel installation failed: {install_result.stderr}")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully installed wheel: {wheel_path}")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Wheel download/install timed out, using fallback")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure latest wheel: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
'''Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
|
||||
output files match local CLI exactly:
|
||||
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
- Concatenated title + karaoke + end screens
|
||||
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
|
||||
|
||||
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
|
||||
'''
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Import LocalEncodingService from installed wheel (required, no fallback)
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
logger.info("Using LocalEncodingService from installed wheel")
|
||||
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
karaoke_video = find_file(
|
||||
work_dir,
|
||||
"*With Vocals*.mov", "*With Vocals*.mkv",
|
||||
"*Vocals*.mov", "*Vocals*.mkv",
|
||||
"*.mkv", "*.mov"
|
||||
)
|
||||
# Exclude title/end/output videos
|
||||
if karaoke_video:
|
||||
name_lower = karaoke_video.name.lower()
|
||||
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
|
||||
# Search more specifically for karaoke video
|
||||
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
|
||||
|
||||
# Instrumental audio - respect user's selection from encoding config
|
||||
instrumental_selection = config.get("instrumental_selection", "clean")
|
||||
logger.info(f"Instrumental selection from config: {instrumental_selection}")
|
||||
|
||||
if instrumental_selection == "with_backing":
|
||||
# User selected instrumental with backing vocals
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
else:
|
||||
# Default to clean instrumental
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
logger.info(f"Found files:")
|
||||
logger.info(f" Title video: {title_video}")
|
||||
logger.info(f" Karaoke video: {karaoke_video}")
|
||||
logger.info(f" End video: {end_video}")
|
||||
logger.info(f" Instrumental ({instrumental_selection}): {instrumental}")
|
||||
|
||||
# Check for countdown padding - if vocals were padded, instrumental must match
|
||||
countdown_padding_seconds = config.get("countdown_padding_seconds")
|
||||
if countdown_padding_seconds:
|
||||
logger.info(f" Countdown padding: {countdown_padding_seconds}s - will be handled by LocalEncodingService")
|
||||
|
||||
# Validate required files
|
||||
if not title_video:
|
||||
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
|
||||
if not karaoke_video:
|
||||
raise ValueError(f"No karaoke video found in {work_dir}")
|
||||
if not instrumental:
|
||||
raise ValueError(f"No instrumental audio found in {work_dir}")
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
# Build encoding config with proper file names
|
||||
# Note: countdown_padding_seconds is passed to LocalEncodingService which handles padding
|
||||
encoding_config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
instrumental_audio=str(instrumental),
|
||||
end_video=str(end_video) if end_video else None,
|
||||
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
countdown_padding_seconds=countdown_padding_seconds,
|
||||
)
|
||||
|
||||
# Create service and run encoding
|
||||
service = LocalEncodingService(logger=logger)
|
||||
|
||||
jobs[job_id]["progress"] = 30
|
||||
logger.info("Starting LocalEncodingService.encode_all_formats()")
|
||||
|
||||
result = service.encode_all_formats(encoding_config)
|
||||
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Encoding failed: {result.error}")
|
||||
|
||||
jobs[job_id]["progress"] = 90
|
||||
|
||||
# Collect output files
|
||||
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
|
||||
logger.info(f"Encoding complete. Output files: {output_files}")
|
||||
return output_dir
|
||||
|
||||
except ImportError as e:
|
||||
# No fallback - wheel must be installed
|
||||
error_msg = (
|
||||
f"LocalEncodingService not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
# This means in-progress jobs continue with their version, new jobs get latest code
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Download input files
|
||||
jobs[job_id]["progress"] = 5
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
executor,
|
||||
run_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request.encoding_config
|
||||
)
|
||||
|
||||
# Upload outputs
|
||||
jobs[job_id]["progress"] = 95
|
||||
logger.info(f"Uploading to {request.output_gcs_path}")
|
||||
upload_to_gcs(output_dir, request.output_gcs_path)
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
local_output_files = jobs[job_id].get("output_files", [])
|
||||
blob_paths = []
|
||||
for local_path in local_output_files:
|
||||
filename = Path(local_path).name
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
blob_paths.append(blob_path)
|
||||
jobs[job_id]["output_files"] = blob_paths
|
||||
logger.info(f"Output files (blob paths): {blob_paths}")
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
# Process a preview encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run preview encoding in thread pool (CPU-bound)
|
||||
# Note: run_preview_encoding handles download/upload internally
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_preview_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Preview job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - preview already encoded
|
||||
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_preview_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.get("/status/{job_id}")
|
||||
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
|
||||
# Get the status of an encoding job
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
return JobStatus(**jobs[job_id])
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
# Health check endpoint
|
||||
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
|
||||
|
||||
# Get karaoke-gen wheel version if installed
|
||||
wheel_version = None
|
||||
try:
|
||||
from importlib.metadata import version as get_version
|
||||
wheel_version = get_version("karaoke-gen")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_jobs": active_jobs,
|
||||
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
|
||||
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
|
||||
"wheel_version": wheel_version,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
@@ -0,0 +1,16 @@
|
||||
# GCE Encoding Worker dependencies
|
||||
# These are installed in the VM's Python virtual environment
|
||||
|
||||
# Web framework
|
||||
fastapi>=0.109.0
|
||||
uvicorn>=0.27.0
|
||||
|
||||
# GCS client
|
||||
google-cloud-storage>=2.14.0
|
||||
|
||||
# Async utilities (optional, for future improvements)
|
||||
aiofiles>=23.2.0
|
||||
aiohttp>=3.9.0
|
||||
|
||||
# Note: The karaoke-gen wheel is installed separately from GCS.
|
||||
# It provides LocalEncodingService which does the actual encoding.
|
||||
@@ -15,6 +15,7 @@ import os
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -253,7 +254,10 @@ class GoogleDriveService:
|
||||
Returns:
|
||||
Dictionary mapping category to uploaded file ID
|
||||
"""
|
||||
filename_base = f"{brand_code} - {base_name}"
|
||||
# Sanitize base_name to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that could cause issues with Google Drive API queries and file naming
|
||||
safe_base_name = sanitize_filename(base_name) if base_name else base_name
|
||||
filename_base = f"{brand_code} - {safe_base_name}"
|
||||
uploaded_files = {}
|
||||
|
||||
logger.info(
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""
|
||||
Centralized job defaults service.
|
||||
|
||||
This module provides consistent handling of job creation defaults across all
|
||||
endpoints (file_upload, audio_search, made-for-you webhook, etc.).
|
||||
|
||||
Centralizing these defaults prevents divergence between code paths and ensures
|
||||
all jobs receive the same default configuration.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
@dataclass
|
||||
class EffectiveDistributionSettings:
|
||||
"""Distribution settings with defaults applied from environment variables."""
|
||||
dropbox_path: Optional[str]
|
||||
gdrive_folder_id: Optional[str]
|
||||
discord_webhook_url: Optional[str]
|
||||
brand_prefix: Optional[str]
|
||||
enable_youtube_upload: bool
|
||||
youtube_description: Optional[str]
|
||||
|
||||
|
||||
def get_effective_distribution_settings(
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
discord_webhook_url: Optional[str] = None,
|
||||
brand_prefix: Optional[str] = None,
|
||||
enable_youtube_upload: Optional[bool] = None,
|
||||
youtube_description: Optional[str] = None,
|
||||
) -> EffectiveDistributionSettings:
|
||||
"""
|
||||
Get distribution settings with defaults applied from environment variables.
|
||||
|
||||
This ensures consistent handling of defaults across all job creation endpoints.
|
||||
Each parameter, if not provided (None), falls back to the corresponding
|
||||
environment variable configured in settings.
|
||||
|
||||
Args:
|
||||
dropbox_path: Explicit Dropbox path or None for default
|
||||
gdrive_folder_id: Explicit Google Drive folder ID or None for default
|
||||
discord_webhook_url: Explicit Discord webhook URL or None for default
|
||||
brand_prefix: Explicit brand prefix or None for default
|
||||
enable_youtube_upload: Explicit flag or None for default
|
||||
youtube_description: Explicit description or None for default
|
||||
|
||||
Returns:
|
||||
EffectiveDistributionSettings with defaults applied
|
||||
"""
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=dropbox_path or settings.default_dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
|
||||
brand_prefix=brand_prefix or settings.default_brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload,
|
||||
youtube_description=youtube_description or settings.default_youtube_description,
|
||||
)
|
||||
|
||||
|
||||
def resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool] = None,
|
||||
enable_txt: Optional[bool] = None,
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
The resolution logic is:
|
||||
1. If explicit True/False is provided, use that value
|
||||
2. Otherwise, if a theme is set, use the server defaults (settings.default_enable_cdg/txt)
|
||||
3. If no theme is set, default to False (CDG/TXT require style configuration)
|
||||
|
||||
This ensures CDG/TXT are only enabled when:
|
||||
- A theme is configured (provides necessary style params), AND
|
||||
- The server defaults allow it (DEFAULT_ENABLE_CDG=true by default)
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
settings = get_settings()
|
||||
|
||||
# Default based on whether theme is set AND server defaults
|
||||
# Theme is required because CDG/TXT need style configuration
|
||||
theme_is_set = theme_id is not None
|
||||
default_cdg = theme_is_set and settings.default_enable_cdg
|
||||
default_txt = theme_is_set and settings.default_enable_txt
|
||||
|
||||
# Explicit values override defaults, None uses computed default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_cdg
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_txt
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
# Singleton instance (optional, for convenience)
|
||||
_service_instance = None
|
||||
|
||||
|
||||
def get_job_defaults_service():
|
||||
"""Get the job defaults service (module-level functions work fine, this is for consistency)."""
|
||||
return {
|
||||
'get_effective_distribution_settings': get_effective_distribution_settings,
|
||||
'resolve_cdg_txt_defaults': resolve_cdg_txt_defaults,
|
||||
}
|
||||
+347
-17
@@ -14,6 +14,7 @@ from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
@@ -23,6 +24,16 @@ from backend.services.storage_service import StorageService
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""Manager for job lifecycle and state."""
|
||||
|
||||
@@ -31,13 +42,52 @@ class JobManager:
|
||||
self.firestore = FirestoreService()
|
||||
self.storage = StorageService()
|
||||
|
||||
def create_job(self, job_create: JobCreate) -> Job:
|
||||
def create_job(self, job_create: JobCreate, is_admin: bool = False) -> Job:
|
||||
"""
|
||||
Create a new job with initial state PENDING.
|
||||
|
||||
|
||||
Jobs start in PENDING state and transition to DOWNLOADING
|
||||
when a worker picks them up.
|
||||
|
||||
Args:
|
||||
job_create: Job creation parameters
|
||||
is_admin: Whether the requesting user is an admin (bypasses rate limits)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme_id is not provided (all jobs require a theme)
|
||||
RateLimitExceededError: If user has exceeded their daily job limit
|
||||
"""
|
||||
# Check rate limit FIRST (before any other validation)
|
||||
# This prevents wasted work if user is rate limited
|
||||
if job_create.user_email:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
allowed, remaining, message = rate_limit_service.check_user_job_limit(
|
||||
user_email=job_create.user_email,
|
||||
is_admin=is_admin
|
||||
)
|
||||
if not allowed:
|
||||
from backend.services.rate_limit_service import _seconds_until_midnight_utc
|
||||
|
||||
# Get actual current count - remaining is clamped to 0 which loses info
|
||||
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
|
||||
raise RateLimitExceededError(
|
||||
message=message,
|
||||
limit_type="jobs_per_day",
|
||||
remaining_seconds=_seconds_until_midnight_utc(),
|
||||
current_count=current_count,
|
||||
limit_value=settings.rate_limit_jobs_per_day
|
||||
)
|
||||
|
||||
# Enforce theme requirement - all jobs must have a theme
|
||||
# This prevents unstyled videos from ever being generated
|
||||
if not job_create.theme_id:
|
||||
raise ValueError(
|
||||
"theme_id is required for all jobs. "
|
||||
"Use get_theme_service().get_default_theme_id() to get the default theme."
|
||||
)
|
||||
|
||||
job_id = str(uuid.uuid4())[:8]
|
||||
|
||||
now = datetime.utcnow()
|
||||
@@ -74,11 +124,27 @@ class JobManager:
|
||||
keep_brand_code=job_create.keep_brand_code,
|
||||
# Request metadata (for tracking and filtering)
|
||||
request_metadata=job_create.request_metadata,
|
||||
# Tenant scoping
|
||||
tenant_id=job_create.tenant_id,
|
||||
# Made-for-you order fields
|
||||
made_for_you=job_create.made_for_you,
|
||||
customer_email=job_create.customer_email,
|
||||
customer_notes=job_create.customer_notes,
|
||||
)
|
||||
|
||||
self.firestore.create_job(job)
|
||||
logger.info(f"Created new job {job_id} with status PENDING")
|
||||
|
||||
|
||||
# Record job creation for rate limiting (after successful persistence)
|
||||
if job_create.user_email:
|
||||
try:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
rate_limit_service.record_job_creation(job_create.user_email, job_id)
|
||||
except Exception as e:
|
||||
# Don't fail job creation if rate limit recording fails
|
||||
logger.warning(f"Failed to record job creation for rate limiting: {e}")
|
||||
|
||||
return job
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
@@ -114,6 +180,7 @@ class JobManager:
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
@@ -126,6 +193,7 @@ class JobManager:
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (job owner)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
@@ -138,6 +206,7 @@ class JobManager:
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
user_email=user_email,
|
||||
tenant_id=tenant_id,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
@@ -305,22 +374,22 @@ class JobManager:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Generate review token when entering AWAITING_REVIEW state
|
||||
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
|
||||
if new_status == JobStatus.AWAITING_REVIEW:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
from backend.api.dependencies import generate_review_token
|
||||
review_token = generate_review_token()
|
||||
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['review_token'] = review_token
|
||||
updates['review_token_expires_at'] = review_token_expires
|
||||
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
|
||||
|
||||
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
|
||||
logger.info(f"Generated review token for job {job_id} (no expiry)")
|
||||
|
||||
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
|
||||
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
|
||||
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
|
||||
from backend.api.dependencies import generate_review_token, get_review_token_expiry
|
||||
from backend.api.dependencies import generate_review_token
|
||||
instrumental_token = generate_review_token() # Reuse same token generator
|
||||
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
|
||||
updates['instrumental_token'] = instrumental_token
|
||||
updates['instrumental_token_expires_at'] = instrumental_token_expires
|
||||
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
|
||||
updates['instrumental_token_expires_at'] = None # No expiry - token is job-scoped
|
||||
logger.info(f"Generated instrumental token for job {job_id} (no expiry)")
|
||||
|
||||
# If we have state_data_updates, merge them with existing state_data
|
||||
merged_state_data = None
|
||||
@@ -359,26 +428,287 @@ class JobManager:
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
|
||||
# Trigger notifications asynchronously (fire-and-forget)
|
||||
self._trigger_state_notifications(job_id, new_status)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Trigger email and push notifications based on state transitions.
|
||||
|
||||
This is fire-and-forget - notification failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: New job status
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
# Get the job to access user info
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.user_email:
|
||||
logger.debug(f"No user email for job {job_id}, skipping notifications")
|
||||
return
|
||||
|
||||
# Job completion notification
|
||||
if new_status == JobStatus.COMPLETE:
|
||||
self._schedule_completion_email(job)
|
||||
self._send_push_notification(job, "complete")
|
||||
|
||||
# Idle reminder scheduling for blocking states
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
|
||||
self._schedule_idle_reminder(job, new_status)
|
||||
# Send push notification for blocking states
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
self._send_push_notification(job, action_type)
|
||||
|
||||
except Exception as e:
|
||||
# Never let notification failures affect job processing
|
||||
logger.error(f"Error triggering notifications for job {job_id}: {e}")
|
||||
|
||||
def _schedule_completion_email(self, job: Job) -> None:
|
||||
"""
|
||||
Schedule sending a job completion email.
|
||||
|
||||
For made-for-you orders, also transfers ownership from admin to customer.
|
||||
Uses asyncio to fire-and-forget the email sending.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# For made-for-you orders, send to customer and transfer ownership
|
||||
recipient_email = job.user_email
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
# Transfer ownership from admin to customer (non-blocking - email still goes out if this fails)
|
||||
try:
|
||||
self.update_job(job.job_id, {'user_email': job.customer_email})
|
||||
logger.info(f"Transferred ownership of made-for-you job {job.job_id} to {_mask_email(job.customer_email)}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to transfer ownership for job {job.job_id}: {e}")
|
||||
|
||||
# Create async task (fire-and-forget)
|
||||
async def send_email():
|
||||
await notification_service.send_job_completion_email(
|
||||
job_id=job.job_id,
|
||||
user_email=recipient_email,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
brand_code=brand_code,
|
||||
)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
# If we're in an async context, create a task
|
||||
loop.create_task(send_email())
|
||||
except RuntimeError:
|
||||
# No event loop - we're likely in a sync context
|
||||
# Use daemon thread to avoid blocking job completion
|
||||
def run_in_thread():
|
||||
asyncio.run(send_email())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled completion email for job {job.job_id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
|
||||
|
||||
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Schedule an idle reminder for a blocking state.
|
||||
|
||||
Records the timestamp when the blocking state was entered and
|
||||
schedules a Cloud Tasks task for 5 minutes later.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
# Record when we entered blocking state (for idle detection)
|
||||
blocking_entered_at = datetime.utcnow().isoformat()
|
||||
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
|
||||
# Update state_data with blocking state info (handle None state_data)
|
||||
existing_state_data = job.state_data or {}
|
||||
state_data_update = {
|
||||
'blocking_state_entered_at': blocking_entered_at,
|
||||
'blocking_action_type': action_type,
|
||||
'reminder_sent': False, # Will be set to True after reminder is sent
|
||||
}
|
||||
|
||||
self.firestore.update_job(job.job_id, {
|
||||
'state_data': {**existing_state_data, **state_data_update}
|
||||
})
|
||||
|
||||
# Schedule the idle reminder check via worker service (5 min delay)
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
async def schedule_reminder():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.schedule_idle_reminder(job.job_id)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(schedule_reminder())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
# Use daemon thread to avoid blocking job processing
|
||||
def run_in_thread():
|
||||
asyncio.run(schedule_reminder())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
|
||||
|
||||
def _send_push_notification(self, job: Job, action_type: str) -> None:
|
||||
"""
|
||||
Send a push notification for job state changes.
|
||||
|
||||
Fire-and-forget - failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job: Job object
|
||||
action_type: Type of notification ("lyrics", "instrumental", or "complete")
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Skip if push notifications not enabled
|
||||
if not push_service.is_enabled():
|
||||
logger.debug("Push notifications not enabled, skipping")
|
||||
return
|
||||
|
||||
# Build job dict for notification service
|
||||
job_dict = {
|
||||
"job_id": job.job_id,
|
||||
"user_email": job.user_email,
|
||||
"artist": job.artist,
|
||||
"title": job.title,
|
||||
}
|
||||
|
||||
async def send_notification():
|
||||
if action_type == "complete":
|
||||
await push_service.send_completion_notification(job_dict)
|
||||
else:
|
||||
await push_service.send_blocking_notification(job_dict, action_type)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(send_notification())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
def run_in_thread():
|
||||
asyncio.run(send_notification())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.debug(f"Scheduled push notification for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to send push notification for job {job.job_id}: {e}")
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
Update a specific key in the job's state_data field.
|
||||
|
||||
|
||||
This is used by workers to store stage-specific metadata.
|
||||
"""
|
||||
job = self.get_job(job_id)
|
||||
if not job:
|
||||
logger.error(f"Job {job_id} not found")
|
||||
return
|
||||
|
||||
|
||||
state_data = job.state_data.copy()
|
||||
state_data[key] = value
|
||||
|
||||
|
||||
self.update_job(job_id, {'state_data': state_data})
|
||||
logger.debug(f"Job {job_id} state_data updated: {key} = {value}")
|
||||
|
||||
def delete_state_data_key(self, job_id: str, key: str) -> bool:
|
||||
"""
|
||||
Delete a specific key from the job's state_data field.
|
||||
|
||||
Uses Firestore DELETE_FIELD to atomically remove the key.
|
||||
This is used to clear worker progress markers, allowing workers
|
||||
to re-run without skipping due to idempotency checks.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
key: The state_data key to delete (e.g., "render_progress")
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD
|
||||
|
||||
try:
|
||||
job_ref = self.firestore.db.collection("jobs").document(job_id)
|
||||
job_ref.update({f"state_data.{key}": DELETE_FIELD})
|
||||
logger.debug(f"Job {job_id}: Deleted state_data key '{key}'")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to delete state_data key '{key}': {e}")
|
||||
return False
|
||||
|
||||
def delete_state_data_keys(self, job_id: str, keys: List[str]) -> List[str]:
|
||||
"""
|
||||
Delete multiple keys from the job's state_data field in a single operation.
|
||||
|
||||
Uses Firestore DELETE_FIELD to atomically remove the keys.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
keys: List of state_data keys to delete
|
||||
|
||||
Returns:
|
||||
List of keys that were successfully deleted
|
||||
"""
|
||||
from google.cloud.firestore_v1 import DELETE_FIELD
|
||||
|
||||
if not keys:
|
||||
return []
|
||||
|
||||
try:
|
||||
job_ref = self.firestore.db.collection("jobs").document(job_id)
|
||||
update_payload = {f"state_data.{key}": DELETE_FIELD for key in keys}
|
||||
job_ref.update(update_payload)
|
||||
logger.info(f"Job {job_id}: Deleted state_data keys: {keys}")
|
||||
return keys
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to delete state_data keys {keys}: {e}")
|
||||
return []
|
||||
|
||||
def fail_job(self, job_id: str, error_message: str, error_details: Optional[Dict[str, Any]] = None) -> bool:
|
||||
"""
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
"""
|
||||
Job notification service for sending emails on job state changes.
|
||||
|
||||
Handles:
|
||||
- Job completion emails (when job enters COMPLETE status)
|
||||
- Action reminder emails (when user is idle at blocking states)
|
||||
|
||||
This service orchestrates between the template service and email service,
|
||||
building the complete notification flow for jobs.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.template_service import get_template_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Environment variable to enable/disable auto emails
|
||||
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
|
||||
|
||||
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
|
||||
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobNotificationService:
|
||||
"""
|
||||
Service for sending job-related email notifications.
|
||||
|
||||
Coordinates between template service (for message rendering) and
|
||||
email service (for sending).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize notification service."""
|
||||
self.email_service = get_email_service()
|
||||
self.template_service = get_template_service()
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
|
||||
"""Build the lyrics review URL for a job."""
|
||||
# Use hash-based routing for static hosting compatibility
|
||||
return f"{self.frontend_url}/app/jobs#/{job_id}/review"
|
||||
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
|
||||
"""Build the instrumental selection URL for a job."""
|
||||
# Use hash-based routing for static hosting compatibility
|
||||
return f"{self.frontend_url}/app/jobs#/{job_id}/instrumental"
|
||||
|
||||
async def send_job_completion_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
user_name: User's display name (optional)
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
brand_code: Release ID (e.g., "NOMAD-1178")
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping completion email")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the completion message using template service
|
||||
message_content = self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
# Send the email with CC to admin
|
||||
success = self.email_service.send_job_completion(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
artist=artist,
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending completion email for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_action_reminder_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
action_type: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
audio_hash: Optional[str] = None,
|
||||
review_token: Optional[str] = None,
|
||||
instrumental_token: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send action-needed reminder email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
audio_hash: Audio hash for review URL
|
||||
review_token: Review token for URL
|
||||
instrumental_token: Instrumental token for URL
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping reminder")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the appropriate template
|
||||
if action_type == "lyrics":
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token)
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=review_url,
|
||||
)
|
||||
elif action_type == "instrumental":
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
|
||||
message_content = self.template_service.render_action_needed_instrumental(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
instrumental_url=instrumental_url,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Unknown action type: {action_type}")
|
||||
return False
|
||||
|
||||
# Send the reminder email (no CC)
|
||||
success = self.email_service.send_action_reminder(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
action_type=action_type,
|
||||
artist=artist,
|
||||
title=title,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
def get_completion_message(
|
||||
self,
|
||||
job_id: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Get the rendered completion message for a job (for admin copy functionality).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
|
||||
Returns:
|
||||
Rendered message content as plain text
|
||||
"""
|
||||
return self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
# Global instance
|
||||
_job_notification_service: Optional[JobNotificationService] = None
|
||||
|
||||
|
||||
def get_job_notification_service() -> JobNotificationService:
|
||||
"""Get the global job notification service instance."""
|
||||
global _job_notification_service
|
||||
if _job_notification_service is None:
|
||||
_job_notification_service = JobNotificationService()
|
||||
return _job_notification_service
|
||||
@@ -0,0 +1,130 @@
|
||||
"""Langfuse callback handler preloader for container startup.
|
||||
|
||||
Initializes the Langfuse callback handler at container startup to avoid
|
||||
slow initialization during request processing. The CallbackHandler()
|
||||
constructor makes blocking network calls to the Langfuse API, which can
|
||||
take 3+ minutes on Cloud Run cold starts.
|
||||
|
||||
IMPORTANT: Langfuse v3 is built on OpenTelemetry. If not configured carefully,
|
||||
it will install itself as the GLOBAL OpenTelemetry tracer provider, capturing
|
||||
ALL spans from every library (FastAPI, HTTP clients, etc.) - not just LLM calls.
|
||||
This caused us to exceed our Langfuse hobby tier monthly event limit with
|
||||
thousands of "(empty)" HTTP request traces.
|
||||
|
||||
The fix: Initialize the Langfuse client with an ISOLATED TracerProvider before
|
||||
creating the CallbackHandler. This ensures only LangChain/LLM traces are sent
|
||||
to Langfuse, while other OTEL instrumentation remains separate.
|
||||
|
||||
See:
|
||||
- docs/archive/2026-01-08-performance-investigation.md for background
|
||||
- https://github.com/orgs/langfuse/discussions/9136 for the OTEL issue
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded handler
|
||||
_preloaded_handler: Optional[Any] = None
|
||||
_initialization_attempted: bool = False
|
||||
|
||||
|
||||
def preload_langfuse_handler() -> Optional[Any]:
|
||||
"""Preload Langfuse callback handler at startup.
|
||||
|
||||
Only initializes if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY
|
||||
environment variables are set.
|
||||
|
||||
CRITICAL: We initialize the Langfuse client with an isolated TracerProvider
|
||||
BEFORE creating the CallbackHandler. This prevents Langfuse from hijacking
|
||||
the global OpenTelemetry provider and capturing all HTTP request spans.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not configured
|
||||
"""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
|
||||
if _initialization_attempted:
|
||||
logger.debug("Langfuse initialization already attempted")
|
||||
return _preloaded_handler
|
||||
|
||||
_initialization_attempted = True
|
||||
|
||||
# Check if Langfuse is configured
|
||||
public_key = os.getenv("LANGFUSE_PUBLIC_KEY")
|
||||
secret_key = os.getenv("LANGFUSE_SECRET_KEY")
|
||||
host = os.getenv("LANGFUSE_HOST", "https://us.cloud.langfuse.com")
|
||||
|
||||
if not (public_key and secret_key):
|
||||
logger.info("Langfuse not configured (missing keys), skipping preload")
|
||||
return None
|
||||
|
||||
logger.info("Preloading Langfuse callback handler with isolated tracer...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
# Import OpenTelemetry TracerProvider for isolation
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from langfuse import Langfuse
|
||||
from langfuse.langchain import CallbackHandler
|
||||
|
||||
# CRITICAL: Initialize Langfuse client with an ISOLATED TracerProvider
|
||||
# This prevents Langfuse from setting itself as the global OTEL provider,
|
||||
# which would cause ALL HTTP requests to be traced (not just LLM calls).
|
||||
# See: https://github.com/orgs/langfuse/discussions/9136
|
||||
Langfuse(
|
||||
public_key=public_key,
|
||||
secret_key=secret_key,
|
||||
host=host,
|
||||
tracer_provider=TracerProvider(), # Isolated - won't touch global OTEL
|
||||
)
|
||||
logger.info("Langfuse client initialized with isolated TracerProvider")
|
||||
|
||||
# Now create the CallbackHandler - it will reuse the existing client
|
||||
# instead of creating a new one that hijacks global OTEL
|
||||
_preloaded_handler = CallbackHandler()
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"Langfuse handler preloaded in {elapsed:.2f}s (host: {host})")
|
||||
|
||||
return _preloaded_handler
|
||||
|
||||
except ImportError as e:
|
||||
logger.warning(f"langfuse or opentelemetry package not installed, skipping preload: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
elapsed = time.time() - start_time
|
||||
logger.error(f"Failed to preload Langfuse handler after {elapsed:.2f}s: {e}")
|
||||
# Don't raise - Langfuse is optional
|
||||
return None
|
||||
|
||||
|
||||
def get_preloaded_langfuse_handler() -> Optional[Any]:
|
||||
"""Get the preloaded Langfuse callback handler if available.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not preloaded/configured
|
||||
"""
|
||||
return _preloaded_handler
|
||||
|
||||
|
||||
def is_langfuse_preloaded() -> bool:
|
||||
"""Check if Langfuse handler has been preloaded."""
|
||||
return _preloaded_handler is not None
|
||||
|
||||
|
||||
def is_langfuse_configured() -> bool:
|
||||
"""Check if Langfuse environment variables are configured."""
|
||||
return bool(
|
||||
os.getenv("LANGFUSE_PUBLIC_KEY") and os.getenv("LANGFUSE_SECRET_KEY")
|
||||
)
|
||||
|
||||
|
||||
def clear_preloaded_handler() -> None:
|
||||
"""Clear preloaded handler. Useful for testing."""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
_preloaded_handler = None
|
||||
_initialization_attempted = False
|
||||
@@ -38,6 +38,9 @@ class EncodingConfig:
|
||||
output_lossless_mkv: Optional[str] = None
|
||||
output_720p_mp4: Optional[str] = None
|
||||
|
||||
# Audio synchronization
|
||||
countdown_padding_seconds: Optional[float] = None # Pad instrumental to match countdown-padded vocals
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingResult:
|
||||
@@ -260,6 +263,38 @@ class LocalEncodingService:
|
||||
|
||||
return self._execute_command(cpu_command, description)
|
||||
|
||||
def _pad_audio_file(self, input_audio: str, output_audio: str, padding_seconds: float) -> str:
|
||||
"""Pad an audio file by prepending silence at the beginning.
|
||||
|
||||
Uses FFmpeg adelay filter - same approach as KaraokeFinalise._pad_audio_file().
|
||||
This ensures the instrumental audio is synchronized with countdown-padded vocals.
|
||||
|
||||
Args:
|
||||
input_audio: Path to input audio file
|
||||
output_audio: Path for the padded output file
|
||||
padding_seconds: Amount of silence to prepend (in seconds)
|
||||
|
||||
Returns:
|
||||
Path to the output audio file
|
||||
"""
|
||||
delay_ms = int(padding_seconds * 1000)
|
||||
cmd = [
|
||||
"ffmpeg", "-y", "-hide_banner", "-loglevel", "error",
|
||||
"-i", input_audio,
|
||||
"-af", f"adelay={delay_ms}|{delay_ms}",
|
||||
output_audio
|
||||
]
|
||||
self.logger.info(f"Padding audio with {padding_seconds}s silence: {os.path.basename(input_audio)}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would pad audio file")
|
||||
return output_audio
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"FFmpeg padding failed: {result.stderr}")
|
||||
return output_audio
|
||||
|
||||
def remux_with_instrumental(
|
||||
self,
|
||||
input_video: str,
|
||||
@@ -465,12 +500,27 @@ class LocalEncodingService:
|
||||
output_files = {}
|
||||
|
||||
try:
|
||||
# Pad instrumental if countdown padding was applied to vocals
|
||||
actual_instrumental = config.instrumental_audio
|
||||
if config.countdown_padding_seconds and config.countdown_padding_seconds > 0:
|
||||
if "(Padded)" not in config.instrumental_audio:
|
||||
base, ext = os.path.splitext(config.instrumental_audio)
|
||||
padded_path = f"{base} (Padded){ext}"
|
||||
if not os.path.exists(padded_path) or self.dry_run:
|
||||
self._pad_audio_file(config.instrumental_audio, padded_path, config.countdown_padding_seconds)
|
||||
self.logger.info(f"Created padded instrumental: {padded_path}")
|
||||
else:
|
||||
self.logger.info(f"Using existing padded instrumental: {padded_path}")
|
||||
actual_instrumental = padded_path
|
||||
else:
|
||||
self.logger.info(f"Instrumental already padded: {config.instrumental_audio}")
|
||||
|
||||
# Step 1: Remux with instrumental audio
|
||||
if config.output_karaoke_mp4:
|
||||
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
|
||||
if not self.remux_with_instrumental(
|
||||
config.karaoke_video,
|
||||
config.instrumental_audio,
|
||||
actual_instrumental,
|
||||
config.output_karaoke_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
|
||||
@@ -0,0 +1,407 @@
|
||||
"""
|
||||
Local Preview Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based preview video encoding functionality, extracted from
|
||||
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
|
||||
is used across local CLI, Cloud Run, and GCE worker environments.
|
||||
|
||||
This service handles:
|
||||
- Generating preview videos with ASS subtitle overlay
|
||||
- Hardware acceleration detection (NVENC) with fallback to software encoding
|
||||
- Background image/color support
|
||||
- Custom font support
|
||||
- Optimized settings for fast preview generation
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingConfig:
|
||||
"""Configuration for preview video encoding."""
|
||||
ass_path: str # Path to ASS subtitles file
|
||||
audio_path: str # Path to audio file
|
||||
output_path: str # Path for output video file
|
||||
|
||||
# Optional background settings
|
||||
background_image_path: Optional[str] = None # Path to background image
|
||||
background_color: str = "black" # Fallback background color
|
||||
|
||||
# Optional font settings
|
||||
font_path: Optional[str] = None # Path to custom font file
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingResult:
|
||||
"""Result of preview encoding operation."""
|
||||
success: bool
|
||||
output_path: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalPreviewEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based preview video encoding.
|
||||
|
||||
This is the single source of truth for preview encoding logic, used by:
|
||||
- Local CLI (via VideoGenerator which delegates here)
|
||||
- Cloud Run (when GCE is unavailable)
|
||||
- GCE worker (via installed wheel)
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# Preview video settings - these are the canonical values
|
||||
PREVIEW_WIDTH = 480
|
||||
PREVIEW_HEIGHT = 270
|
||||
PREVIEW_FPS = 24
|
||||
PREVIEW_AUDIO_BITRATE = "96k"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local preview encoding service.
|
||||
|
||||
Args:
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._nvenc_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._hwaccel_flags: Optional[List[str]] = None
|
||||
|
||||
def _detect_nvenc_support(self) -> bool:
|
||||
"""
|
||||
Detect if NVENC hardware encoding is available.
|
||||
|
||||
Returns:
|
||||
True if NVENC is available, False otherwise
|
||||
"""
|
||||
try:
|
||||
self.logger.info("Detecting NVENC hardware acceleration...")
|
||||
|
||||
# Test h264_nvenc encoder directly
|
||||
test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
|
||||
"-c:v", "h264_nvenc", "-f", "null", "-"
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
# Try alternative test with different source
|
||||
alt_test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
|
||||
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
|
||||
]
|
||||
|
||||
alt_result = subprocess.run(
|
||||
alt_test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if alt_result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
self.logger.info("NVENC not available, using software encoding")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.debug("NVENC detection timed out")
|
||||
return False
|
||||
except Exception as e:
|
||||
self.logger.debug(f"NVENC detection failed: {e}")
|
||||
return False
|
||||
|
||||
def _configure_hardware_acceleration(self) -> None:
|
||||
"""Configure hardware acceleration settings based on detected capabilities."""
|
||||
if self._nvenc_available is None:
|
||||
self._nvenc_available = self._detect_nvenc_support()
|
||||
|
||||
if self._nvenc_available:
|
||||
self._video_encoder = "h264_nvenc"
|
||||
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
self.logger.info("Using NVENC hardware acceleration for preview encoding")
|
||||
else:
|
||||
self._video_encoder = "libx264"
|
||||
self._hwaccel_flags = []
|
||||
self.logger.info("Using software encoding (libx264) for preview")
|
||||
|
||||
@property
|
||||
def nvenc_available(self) -> bool:
|
||||
"""Check if NVENC hardware acceleration is available."""
|
||||
if self._nvenc_available is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._nvenc_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def hwaccel_flags(self) -> List[str]:
|
||||
"""Get hardware acceleration flags."""
|
||||
if self._hwaccel_flags is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._hwaccel_flags
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""
|
||||
Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\\ -> \\\\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
|
||||
- Spaces: escape with backslash ( -> \\ )
|
||||
- Special characters: :,[];
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
# Escape FFmpeg filter special characters
|
||||
escaped = escaped.replace(":", "\\:")
|
||||
escaped = escaped.replace(",", "\\,")
|
||||
escaped = escaped.replace("[", "\\[")
|
||||
escaped = escaped.replace("]", "\\]")
|
||||
escaped = escaped.replace(";", "\\;")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
|
||||
"""
|
||||
Build ASS filter with optional font directory support.
|
||||
|
||||
Args:
|
||||
ass_path: Path to ASS subtitles file
|
||||
font_path: Optional path to custom font file
|
||||
|
||||
Returns:
|
||||
FFmpeg ASS filter string
|
||||
"""
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.debug(f"Using font directory: {font_dir}")
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
|
||||
"""
|
||||
Build FFmpeg command for preview video generation.
|
||||
|
||||
This is the canonical preview encoding command, ensuring consistency
|
||||
across all environments (local CLI, Cloud Run, GCE worker).
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
FFmpeg command as a list of arguments
|
||||
"""
|
||||
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-r", str(self.PREVIEW_FPS),
|
||||
]
|
||||
|
||||
# Add hardware acceleration flags if available
|
||||
cmd.extend(self.hwaccel_flags)
|
||||
|
||||
# Input source (background image or solid color)
|
||||
if config.background_image_path and os.path.isfile(config.background_image_path):
|
||||
self.logger.debug(f"Using background image: {config.background_image_path}")
|
||||
cmd.extend([
|
||||
"-loop", "1",
|
||||
"-i", config.background_image_path,
|
||||
])
|
||||
# Build video filter with scaling and ASS subtitles
|
||||
video_filter = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"Using solid {config.background_color} background")
|
||||
cmd.extend([
|
||||
"-f", "lavfi",
|
||||
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
|
||||
])
|
||||
# Just ASS subtitles, no scaling needed
|
||||
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
|
||||
|
||||
cmd.extend([
|
||||
"-i", config.audio_path,
|
||||
"-vf", video_filter,
|
||||
"-c:a", "aac",
|
||||
"-b:a", self.PREVIEW_AUDIO_BITRATE,
|
||||
"-c:v", self.video_encoder,
|
||||
])
|
||||
|
||||
# Add encoder-specific settings optimized for speed
|
||||
if self.nvenc_available:
|
||||
cmd.extend([
|
||||
"-preset", "p1", # Fastest NVENC preset
|
||||
"-tune", "ll", # Low latency
|
||||
"-rc", "cbr", # Constant bitrate for speed
|
||||
"-b:v", "800k", # Lower bitrate for speed
|
||||
"-profile:v", "baseline", # Most compatible profile
|
||||
"-level", "3.1", # Lower level for speed
|
||||
])
|
||||
self.logger.debug("Using NVENC with maximum speed settings")
|
||||
else:
|
||||
cmd.extend([
|
||||
"-profile:v", "baseline",
|
||||
"-level", "3.0",
|
||||
"-preset", "superfast",
|
||||
"-tune", "fastdecode",
|
||||
"-b:v", "600k",
|
||||
"-maxrate", "800k",
|
||||
"-bufsize", "1200k",
|
||||
"-crf", "28",
|
||||
])
|
||||
self.logger.debug("Using software encoding with maximum speed settings")
|
||||
|
||||
cmd.extend([
|
||||
"-pix_fmt", "yuv420p", # Required for browser compatibility
|
||||
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
|
||||
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
|
||||
"-keyint_min", "48",
|
||||
"-sc_threshold", "0", # Disable scene change detection for speed
|
||||
"-threads", "0", # Use all available CPU threads
|
||||
"-shortest",
|
||||
"-y",
|
||||
config.output_path,
|
||||
])
|
||||
|
||||
return cmd
|
||||
|
||||
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
|
||||
"""
|
||||
Encode a preview video.
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
PreviewEncodingResult with success status and output path
|
||||
"""
|
||||
self.logger.info(f"Encoding preview video: {config.output_path}")
|
||||
|
||||
# Validate input files
|
||||
if not os.path.isfile(config.ass_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"ASS subtitles file not found: {config.ass_path}"
|
||||
)
|
||||
|
||||
if not os.path.isfile(config.audio_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"Audio file not found: {config.audio_path}"
|
||||
)
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir = os.path.dirname(config.output_path)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
try:
|
||||
# Build and execute FFmpeg command
|
||||
cmd = self._build_preview_ffmpeg_command(config)
|
||||
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=300 # 5 minute timeout for preview encoding
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
|
||||
self.logger.error(f"FFmpeg failed: {error_msg}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"FFmpeg preview encoding failed: {error_msg}"
|
||||
)
|
||||
|
||||
self.logger.info(f"Preview encoded successfully: {config.output_path}")
|
||||
return PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path=config.output_path
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error("Preview encoding timed out")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Preview encoding timed out after 5 minutes"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Preview encoding failed: {e}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function
|
||||
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
|
||||
|
||||
|
||||
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
|
||||
"""
|
||||
Get a local preview encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalPreviewEncodingService
|
||||
|
||||
Returns:
|
||||
LocalPreviewEncodingService instance
|
||||
"""
|
||||
global _local_preview_encoding_service
|
||||
|
||||
if _local_preview_encoding_service is None:
|
||||
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
|
||||
|
||||
return _local_preview_encoding_service
|
||||
@@ -0,0 +1,122 @@
|
||||
"""NLTK resource preloader for container startup.
|
||||
|
||||
Loads NLTK data at container startup to avoid slow downloads during request processing.
|
||||
Cloud Run's ephemeral filesystem means NLTK data must be re-downloaded on each cold start,
|
||||
which can take 30-100+ seconds for cmudict.
|
||||
|
||||
See docs/archive/2026-01-08-performance-investigation.md for background.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded resources
|
||||
_preloaded_resources: Dict[str, Any] = {}
|
||||
|
||||
|
||||
def preload_nltk_cmudict() -> None:
|
||||
"""Preload NLTK's CMU Pronouncing Dictionary at startup.
|
||||
|
||||
The cmudict is used by SyllablesMatchHandler for syllable counting.
|
||||
Without preloading, each SyllablesMatchHandler init downloads ~30MB,
|
||||
which took 50-100+ seconds in Cloud Run.
|
||||
"""
|
||||
global _preloaded_resources
|
||||
|
||||
if "cmudict" in _preloaded_resources:
|
||||
logger.info("NLTK cmudict already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK cmudict...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
# Ensure the data is downloaded
|
||||
try:
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Try to access it - will raise LookupError if not downloaded
|
||||
_ = cmudict.dict()
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK cmudict data...")
|
||||
nltk.download("cmudict", quiet=True)
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Load into memory
|
||||
cmu_dict = cmudict.dict()
|
||||
_preloaded_resources["cmudict"] = cmu_dict
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK cmudict preloaded in {elapsed:.2f}s ({len(cmu_dict)} entries)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload NLTK cmudict: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_cmudict() -> Optional[Dict]:
|
||||
"""Get the preloaded CMU dictionary if available.
|
||||
|
||||
Returns:
|
||||
The preloaded cmudict dictionary, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_resources.get("cmudict")
|
||||
|
||||
|
||||
def is_cmudict_preloaded() -> bool:
|
||||
"""Check if cmudict has been preloaded."""
|
||||
return "cmudict" in _preloaded_resources
|
||||
|
||||
|
||||
def preload_nltk_punkt() -> None:
|
||||
"""Preload NLTK's punkt tokenizer (optional, used for sentence tokenization)."""
|
||||
global _preloaded_resources
|
||||
|
||||
if "punkt" in _preloaded_resources:
|
||||
logger.info("NLTK punkt already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK punkt tokenizer...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
try:
|
||||
from nltk.tokenize import word_tokenize
|
||||
|
||||
# Test it works
|
||||
_ = word_tokenize("test")
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK punkt data...")
|
||||
nltk.download("punkt", quiet=True)
|
||||
nltk.download("punkt_tab", quiet=True)
|
||||
|
||||
_preloaded_resources["punkt"] = True
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK punkt preloaded in {elapsed:.2f}s")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to preload NLTK punkt (non-critical): {e}")
|
||||
|
||||
|
||||
def preload_all_nltk_resources() -> None:
|
||||
"""Preload all NLTK resources used by the application."""
|
||||
preload_nltk_cmudict()
|
||||
# punkt is optional and less critical
|
||||
try:
|
||||
preload_nltk_punkt()
|
||||
except Exception:
|
||||
pass # Non-critical
|
||||
|
||||
|
||||
def clear_preloaded_resources() -> None:
|
||||
"""Clear all preloaded resources. Useful for testing."""
|
||||
global _preloaded_resources
|
||||
_preloaded_resources.clear()
|
||||
@@ -120,7 +120,7 @@ class PackagingService:
|
||||
)
|
||||
|
||||
self.logger.info("Generating CDG and MP3 files")
|
||||
from lyrics_transcriber.output.cdg import CDGGenerator
|
||||
from karaoke_gen.lyrics_transcriber.output.cdg import CDGGenerator
|
||||
|
||||
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
|
||||
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
|
||||
|
||||
@@ -0,0 +1,409 @@
|
||||
"""
|
||||
Push Notification Service for Web Push Notifications.
|
||||
|
||||
Handles sending push notifications to users' browsers/devices using the Web Push protocol.
|
||||
Uses pywebpush library for VAPID authentication and push message encryption.
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, List
|
||||
|
||||
from pywebpush import webpush, WebPushException
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import User, PushSubscription
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SubscriptionGoneError(Exception):
|
||||
"""Raised when a push subscription is no longer valid (410 Gone)."""
|
||||
pass
|
||||
|
||||
|
||||
class PushNotificationService:
|
||||
"""
|
||||
Service for sending Web Push notifications.
|
||||
|
||||
Requires VAPID keys to be configured in Secret Manager:
|
||||
- vapid-public-key: Base64-encoded public key
|
||||
- vapid-private-key: Base64-encoded private key
|
||||
"""
|
||||
|
||||
def __init__(self, db=None):
|
||||
"""
|
||||
Initialize the push notification service.
|
||||
|
||||
Args:
|
||||
db: Firestore client (optional, will use singleton if not provided)
|
||||
"""
|
||||
self.settings = get_settings()
|
||||
self._vapid_private_key: Optional[str] = None
|
||||
self._vapid_public_key: Optional[str] = None
|
||||
self._db = db
|
||||
|
||||
@property
|
||||
def db(self):
|
||||
"""Get Firestore client (lazy initialization)."""
|
||||
if self._db is None:
|
||||
from google.cloud import firestore
|
||||
self._db = firestore.Client()
|
||||
return self._db
|
||||
|
||||
@property
|
||||
def vapid_private_key(self) -> Optional[str]:
|
||||
"""Get VAPID private key from Secret Manager (cached)."""
|
||||
if self._vapid_private_key is None:
|
||||
self._vapid_private_key = self.settings.get_secret("vapid-private-key")
|
||||
return self._vapid_private_key
|
||||
|
||||
@property
|
||||
def vapid_public_key(self) -> Optional[str]:
|
||||
"""Get VAPID public key from Secret Manager (cached)."""
|
||||
if self._vapid_public_key is None:
|
||||
self._vapid_public_key = self.settings.get_secret("vapid-public-key")
|
||||
return self._vapid_public_key
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if push notifications are enabled and properly configured."""
|
||||
return (
|
||||
self.settings.enable_push_notifications and
|
||||
self.vapid_private_key is not None and
|
||||
self.vapid_public_key is not None
|
||||
)
|
||||
|
||||
def get_public_key(self) -> Optional[str]:
|
||||
"""Get the VAPID public key for client-side subscription."""
|
||||
if not self.settings.enable_push_notifications:
|
||||
return None
|
||||
return self.vapid_public_key
|
||||
|
||||
async def send_push(
|
||||
self,
|
||||
user_email: str,
|
||||
title: str,
|
||||
body: str,
|
||||
url: str = "/app/",
|
||||
tag: Optional[str] = None
|
||||
) -> int:
|
||||
"""
|
||||
Send a push notification to all of a user's subscribed devices.
|
||||
|
||||
Args:
|
||||
user_email: The user's email address
|
||||
title: Notification title
|
||||
body: Notification body text
|
||||
url: URL to open when notification is clicked
|
||||
tag: Optional tag for notification grouping (replaces notifications with same tag)
|
||||
|
||||
Returns:
|
||||
Number of notifications successfully sent
|
||||
"""
|
||||
if not self.is_enabled():
|
||||
logger.debug("Push notifications not enabled, skipping")
|
||||
return 0
|
||||
|
||||
# Get user from Firestore
|
||||
user_doc = self.db.collection("gen_users").document(user_email).get()
|
||||
if not user_doc.exists:
|
||||
logger.warning(f"User {user_email} not found for push notification")
|
||||
return 0
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
if not subscriptions:
|
||||
logger.debug(f"User {user_email} has no push subscriptions")
|
||||
return 0
|
||||
|
||||
# Build notification payload
|
||||
import json
|
||||
payload = json.dumps({
|
||||
"title": title,
|
||||
"body": body,
|
||||
"url": url,
|
||||
"tag": tag or "default",
|
||||
"icon": "/nomad-logo.png"
|
||||
})
|
||||
|
||||
# Send to each subscription
|
||||
success_count = 0
|
||||
invalid_subscriptions = []
|
||||
|
||||
for sub in subscriptions:
|
||||
try:
|
||||
webpush(
|
||||
subscription_info={
|
||||
"endpoint": sub["endpoint"],
|
||||
"keys": sub["keys"]
|
||||
},
|
||||
data=payload,
|
||||
vapid_private_key=self.vapid_private_key,
|
||||
vapid_claims={
|
||||
"sub": self.settings.vapid_subject
|
||||
}
|
||||
)
|
||||
success_count += 1
|
||||
logger.debug(f"Push sent to {sub.get('device_name', 'unknown device')}")
|
||||
|
||||
except WebPushException as e:
|
||||
logger.warning(f"Push failed: {e}")
|
||||
# Check if subscription is gone (410) or unauthorized (401/403)
|
||||
if e.response and e.response.status_code in (404, 410):
|
||||
logger.info(f"Subscription gone, will remove: {sub['endpoint'][:50]}...")
|
||||
invalid_subscriptions.append(sub["endpoint"])
|
||||
elif e.response and e.response.status_code in (401, 403):
|
||||
logger.warning(f"Push auth failed for {sub['endpoint'][:50]}...")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error sending push: {e}")
|
||||
|
||||
# Remove invalid subscriptions
|
||||
if invalid_subscriptions:
|
||||
await self._remove_invalid_subscriptions(user_email, invalid_subscriptions)
|
||||
|
||||
logger.info(f"Push notifications sent to {success_count}/{len(subscriptions)} devices for {user_email}")
|
||||
return success_count
|
||||
|
||||
async def _remove_invalid_subscriptions(self, user_email: str, endpoints: List[str]) -> None:
|
||||
"""Remove invalid subscriptions from user's list."""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
if not user_doc.exists:
|
||||
return
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
current_subs = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Filter out invalid subscriptions
|
||||
valid_subs = [s for s in current_subs if s["endpoint"] not in endpoints]
|
||||
|
||||
if len(valid_subs) < len(current_subs):
|
||||
user_ref.update({"push_subscriptions": valid_subs})
|
||||
logger.info(f"Removed {len(current_subs) - len(valid_subs)} invalid subscriptions for {user_email}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to remove invalid subscriptions: {e}")
|
||||
|
||||
async def send_blocking_notification(
|
||||
self,
|
||||
job: dict,
|
||||
action_type: str # "lyrics" or "instrumental"
|
||||
) -> int:
|
||||
"""
|
||||
Send notification when a job enters a blocking state requiring user action.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
|
||||
Returns:
|
||||
Number of notifications sent
|
||||
"""
|
||||
user_email = job.get("user_email")
|
||||
if not user_email:
|
||||
return 0
|
||||
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
|
||||
if action_type == "lyrics":
|
||||
notif_title = "Review Lyrics"
|
||||
notif_body = f'"{title}" by {artist} needs lyrics review'
|
||||
url = f"/review/{job_id}"
|
||||
tag = f"lyrics-{job_id}"
|
||||
else: # instrumental
|
||||
notif_title = "Select Instrumental"
|
||||
notif_body = f'"{title}" by {artist} needs instrumental selection'
|
||||
url = f"/instrumental/{job_id}"
|
||||
tag = f"instrumental-{job_id}"
|
||||
|
||||
return await self.send_push(
|
||||
user_email=user_email,
|
||||
title=notif_title,
|
||||
body=notif_body,
|
||||
url=url,
|
||||
tag=tag
|
||||
)
|
||||
|
||||
async def send_completion_notification(self, job: dict) -> int:
|
||||
"""
|
||||
Send notification when a job completes successfully.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
|
||||
Returns:
|
||||
Number of notifications sent
|
||||
"""
|
||||
user_email = job.get("user_email")
|
||||
if not user_email:
|
||||
return 0
|
||||
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
|
||||
return await self.send_push(
|
||||
user_email=user_email,
|
||||
title="Video Ready!",
|
||||
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
|
||||
url=f"/app/?job={job_id}",
|
||||
tag=f"complete-{job_id}"
|
||||
)
|
||||
|
||||
async def add_subscription(
|
||||
self,
|
||||
user_email: str,
|
||||
endpoint: str,
|
||||
keys: dict,
|
||||
device_name: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Add a push subscription for a user.
|
||||
|
||||
Enforces max subscriptions per user - oldest removed if limit exceeded.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
endpoint: Push service endpoint URL
|
||||
keys: Encryption keys (p256dh, auth)
|
||||
device_name: Optional device identifier
|
||||
|
||||
Returns:
|
||||
True if subscription was added successfully
|
||||
"""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
|
||||
if not user_doc.exists:
|
||||
logger.warning(f"User {user_email} not found")
|
||||
return False
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Check if this endpoint already exists (update it)
|
||||
existing_idx = None
|
||||
for i, sub in enumerate(subscriptions):
|
||||
if sub["endpoint"] == endpoint:
|
||||
existing_idx = i
|
||||
break
|
||||
|
||||
new_sub = {
|
||||
"endpoint": endpoint,
|
||||
"keys": keys,
|
||||
"device_name": device_name,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"last_used_at": None
|
||||
}
|
||||
|
||||
if existing_idx is not None:
|
||||
# Update existing subscription
|
||||
subscriptions[existing_idx] = new_sub
|
||||
logger.info(f"Updated existing push subscription for {user_email}")
|
||||
else:
|
||||
# Add new subscription
|
||||
subscriptions.append(new_sub)
|
||||
logger.info(f"Added new push subscription for {user_email}")
|
||||
|
||||
# Enforce max subscriptions (remove oldest)
|
||||
max_subs = self.settings.max_push_subscriptions_per_user
|
||||
if len(subscriptions) > max_subs:
|
||||
# Sort by created_at and keep only the newest
|
||||
subscriptions.sort(key=lambda s: s.get("created_at", ""), reverse=True)
|
||||
removed = subscriptions[max_subs:]
|
||||
subscriptions = subscriptions[:max_subs]
|
||||
logger.info(f"Removed {len(removed)} old subscriptions for {user_email} (max {max_subs})")
|
||||
|
||||
user_ref.update({"push_subscriptions": subscriptions})
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to add subscription for {user_email}: {e}")
|
||||
return False
|
||||
|
||||
async def remove_subscription(self, user_email: str, endpoint: str) -> bool:
|
||||
"""
|
||||
Remove a push subscription for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
endpoint: Push service endpoint URL to remove
|
||||
|
||||
Returns:
|
||||
True if subscription was removed
|
||||
"""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
|
||||
if not user_doc.exists:
|
||||
return False
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Filter out the subscription
|
||||
new_subs = [s for s in subscriptions if s["endpoint"] != endpoint]
|
||||
|
||||
if len(new_subs) < len(subscriptions):
|
||||
user_ref.update({"push_subscriptions": new_subs})
|
||||
logger.info(f"Removed push subscription for {user_email}")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to remove subscription for {user_email}: {e}")
|
||||
return False
|
||||
|
||||
async def list_subscriptions(self, user_email: str) -> List[dict]:
|
||||
"""
|
||||
List all push subscriptions for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
|
||||
Returns:
|
||||
List of subscription info (endpoint, device_name, created_at, last_used_at)
|
||||
"""
|
||||
try:
|
||||
user_doc = self.db.collection("gen_users").document(user_email).get()
|
||||
|
||||
if not user_doc.exists:
|
||||
return []
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Return safe subset of subscription data
|
||||
return [
|
||||
{
|
||||
"endpoint": s["endpoint"],
|
||||
"device_name": s.get("device_name"),
|
||||
"created_at": s.get("created_at"),
|
||||
"last_used_at": s.get("last_used_at")
|
||||
}
|
||||
for s in subscriptions
|
||||
]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list subscriptions for {user_email}: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_push_service: Optional[PushNotificationService] = None
|
||||
|
||||
|
||||
def get_push_notification_service(db=None) -> PushNotificationService:
|
||||
"""Get the push notification service singleton."""
|
||||
global _push_service
|
||||
if _push_service is None:
|
||||
_push_service = PushNotificationService(db)
|
||||
return _push_service
|
||||
@@ -0,0 +1,641 @@
|
||||
"""
|
||||
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
|
||||
|
||||
Uses Firestore for distributed rate limit tracking with date-based document IDs
|
||||
that automatically reset at UTC midnight.
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Tuple, Optional, List, Dict, Any
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Firestore collection names
|
||||
RATE_LIMITS_COLLECTION = "rate_limits"
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
OVERRIDES_COLLECTION = "overrides"
|
||||
|
||||
|
||||
def _get_today_date_str() -> str:
|
||||
"""Get today's date in UTC as YYYY-MM-DD string."""
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def _seconds_until_midnight_utc() -> int:
|
||||
"""Calculate seconds until UTC midnight (when rate limits reset)."""
|
||||
now = datetime.now(timezone.utc)
|
||||
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
# Add 1 day to get next midnight
|
||||
from datetime import timedelta
|
||||
next_midnight = midnight + timedelta(days=1)
|
||||
return int((next_midnight - now).total_seconds())
|
||||
|
||||
|
||||
def _hash_ip(ip_address: str) -> str:
|
||||
"""Hash an IP address for storage (privacy-preserving)."""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
class RateLimitService:
|
||||
"""
|
||||
Service for managing rate limits across the platform.
|
||||
|
||||
Rate limits use Firestore for distributed tracking with automatic
|
||||
reset at UTC midnight via date-based document IDs.
|
||||
"""
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize rate limit service with Firestore client."""
|
||||
if db is None:
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
else:
|
||||
self.db = db
|
||||
self._blocklist_cache: Optional[Dict[str, Any]] = None
|
||||
self._blocklist_cache_time: Optional[datetime] = None
|
||||
self._cache_ttl_seconds = 300 # 5 minute cache
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Job Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if a user can create a new job.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
is_admin: Whether the user is an admin (admins bypass limits)
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
- allowed: True if user can create job
|
||||
- remaining: Number of jobs remaining today
|
||||
- message: Human-readable status message
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
if is_admin:
|
||||
return True, -1, "Admin users bypass rate limits"
|
||||
|
||||
limit = settings.rate_limit_jobs_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No job limit configured"
|
||||
|
||||
# Check for user override
|
||||
override = self.get_user_override(user_email)
|
||||
if override and override.get("bypass_job_limit"):
|
||||
return True, -1, "User has rate limit bypass"
|
||||
if override and override.get("custom_daily_job_limit"):
|
||||
limit = override["custom_daily_job_limit"]
|
||||
|
||||
# Get current count
|
||||
current_count = self.get_user_job_count_today(user_email)
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} jobs remaining today"
|
||||
|
||||
def get_user_job_count_today(self, user_email: str) -> int:
|
||||
"""Get the number of jobs created by a user today."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_job_creation(self, user_email: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a job creation for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
job_ids = data.get("job_ids", [])
|
||||
job_ids.append(job_id)
|
||||
else:
|
||||
count = 1
|
||||
job_ids = [job_id]
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"user_email": user_email,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"job_ids": job_ids,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# YouTube Upload Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if the system can perform a YouTube upload.
|
||||
|
||||
YouTube uploads are limited system-wide due to API quota constraints.
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_youtube_uploads_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No YouTube upload limit configured"
|
||||
|
||||
current_count = self.get_youtube_uploads_today()
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} YouTube uploads remaining today"
|
||||
|
||||
def get_youtube_uploads_today(self) -> int:
|
||||
"""Get the number of YouTube uploads performed today (system-wide)."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
|
||||
"""
|
||||
Record a YouTube upload for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
uploads = data.get("uploads", [])
|
||||
else:
|
||||
count = 1
|
||||
uploads = []
|
||||
|
||||
uploads.append({
|
||||
"job_id": job_id,
|
||||
"user_email": user_email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"uploads": uploads,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded YouTube upload for job {job_id}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Beta Enrollment IP Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if an IP address can enroll in the beta program.
|
||||
|
||||
Args:
|
||||
ip_address: Client IP address
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_beta_ip_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No beta IP limit configured"
|
||||
|
||||
# Check today's enrollment count for this IP
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
current_count = 0
|
||||
if doc.exists:
|
||||
current_count = doc.to_dict().get("count", 0)
|
||||
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Too many beta enrollments from this network today. Please try again tomorrow."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
|
||||
|
||||
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
|
||||
"""
|
||||
Record a beta enrollment for IP rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
enrollments = data.get("enrollments", [])
|
||||
else:
|
||||
count = 1
|
||||
enrollments = []
|
||||
|
||||
enrollments.append({
|
||||
"email": email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"ip_hash": ip_hash,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"enrollments": enrollments,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# User Overrides (Whitelist)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get rate limit override settings for a user.
|
||||
|
||||
Returns:
|
||||
Override settings dict or None if no override exists
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
|
||||
def set_user_override(
|
||||
self,
|
||||
user_email: str,
|
||||
bypass_job_limit: bool = False,
|
||||
custom_daily_job_limit: Optional[int] = None,
|
||||
reason: str = "",
|
||||
admin_email: str = ""
|
||||
) -> None:
|
||||
"""
|
||||
Set rate limit override for a user.
|
||||
|
||||
Args:
|
||||
user_email: User to override
|
||||
bypass_job_limit: If True, user bypasses all job limits
|
||||
custom_daily_job_limit: Custom limit (None = use default)
|
||||
reason: Reason for override
|
||||
admin_email: Admin who set the override
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc_ref.set({
|
||||
"email": user_email.lower(),
|
||||
"bypass_job_limit": bypass_job_limit,
|
||||
"custom_daily_job_limit": custom_daily_job_limit,
|
||||
"reason": reason,
|
||||
"created_by": admin_email,
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
})
|
||||
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
|
||||
|
||||
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
|
||||
"""Remove rate limit override for a user."""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return False
|
||||
|
||||
doc_ref.delete()
|
||||
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def list_user_overrides(self) -> List[Dict[str, Any]]:
|
||||
"""List all user rate limit overrides."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get all user rate limit overrides as a dict keyed by email."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return {doc.id: doc.to_dict() for doc in docs}
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Load blocklist from Firestore with caching.
|
||||
|
||||
Returns:
|
||||
Dict with keys: disposable_domains, blocked_emails, blocked_ips
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache
|
||||
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
|
||||
cache_age = (now - self._blocklist_cache_time).total_seconds()
|
||||
if cache_age < self._cache_ttl_seconds:
|
||||
return self._blocklist_cache
|
||||
|
||||
# Load from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
# Initialize with empty blocklist
|
||||
self._blocklist_cache = {
|
||||
"disposable_domains": [],
|
||||
"blocked_emails": [],
|
||||
"blocked_ips": [],
|
||||
}
|
||||
else:
|
||||
self._blocklist_cache = doc.to_dict()
|
||||
|
||||
self._blocklist_cache_time = now
|
||||
return self._blocklist_cache
|
||||
|
||||
def is_disposable_domain(self, domain: str) -> bool:
|
||||
"""Check if a domain is in the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist()
|
||||
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
|
||||
|
||||
def is_blocked_email(self, email: str) -> bool:
|
||||
"""Check if an email is explicitly blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
|
||||
|
||||
def is_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Check if an IP address is blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return ip_address in blocklist.get("blocked_ips", [])
|
||||
|
||||
def get_blocklist(self) -> Dict[str, Any]:
|
||||
"""Get all blocklist data."""
|
||||
return self._load_blocklist(force_refresh=True)
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
domains.add(domain.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None # Invalidate cache
|
||||
logger.info(f"Added disposable domain {domain} by {admin_email}")
|
||||
|
||||
def remove_disposable_domain(self, domain: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
|
||||
if domain.lower() not in [d.lower() for d in domains]:
|
||||
return False
|
||||
|
||||
domains = {d for d in domains if d.lower() != domain.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed disposable domain {domain}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> None:
|
||||
"""Add an email to the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
emails.add(email.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked email {email} by {admin_email}")
|
||||
|
||||
def remove_blocked_email(self, email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
|
||||
if email.lower() not in [e.lower() for e in emails]:
|
||||
return False
|
||||
|
||||
emails = {e for e in emails if e.lower() != email.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked email {email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
ips.add(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return False
|
||||
|
||||
ips.discard(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked IP {ip_address}")
|
||||
return True
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Stats
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_rate_limit_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns:
|
||||
Dict with current usage stats
|
||||
"""
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
# Get YouTube upload count
|
||||
youtube_count = self.get_youtube_uploads_today()
|
||||
|
||||
# Count unique users with jobs today
|
||||
users_with_jobs = set()
|
||||
total_jobs_today = 0
|
||||
|
||||
# Query all user job documents for today
|
||||
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
|
||||
"date", "==", date_str
|
||||
).stream()
|
||||
|
||||
for doc in docs:
|
||||
data = doc.to_dict()
|
||||
if data.get("user_email"):
|
||||
users_with_jobs.add(data["user_email"])
|
||||
total_jobs_today += data.get("count", 0)
|
||||
|
||||
return {
|
||||
"date": date_str,
|
||||
"youtube_uploads_today": youtube_count,
|
||||
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
|
||||
"total_jobs_today": total_jobs_today,
|
||||
"users_with_jobs_today": len(users_with_jobs),
|
||||
"job_limit_per_user": settings.rate_limit_jobs_per_day,
|
||||
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
|
||||
"rate_limiting_enabled": settings.enable_rate_limiting,
|
||||
"seconds_until_reset": _seconds_until_midnight_utc(),
|
||||
}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_rate_limit_service: Optional[RateLimitService] = None
|
||||
|
||||
|
||||
def get_rate_limit_service() -> RateLimitService:
|
||||
"""Get the singleton RateLimitService instance."""
|
||||
global _rate_limit_service
|
||||
if _rate_limit_service is None:
|
||||
_rate_limit_service = RateLimitService()
|
||||
return _rate_limit_service
|
||||
@@ -0,0 +1,65 @@
|
||||
"""SpaCy model preloader for container startup.
|
||||
|
||||
Loads SpaCy models at container startup to avoid slow loading during request processing.
|
||||
Cloud Run filesystem I/O can cause 60+ second delays when loading SpaCy models lazily.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded models
|
||||
_preloaded_models: dict = {}
|
||||
|
||||
|
||||
def preload_spacy_model(model_name: str = "en_core_web_sm") -> None:
|
||||
"""Preload a SpaCy model at startup.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model to load (default: en_core_web_sm)
|
||||
"""
|
||||
global _preloaded_models
|
||||
|
||||
if model_name in _preloaded_models:
|
||||
logger.info(f"SpaCy model '{model_name}' already preloaded")
|
||||
return
|
||||
|
||||
logger.info(f"Preloading SpaCy model '{model_name}'...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load(model_name)
|
||||
_preloaded_models[model_name] = nlp
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"SpaCy model '{model_name}' preloaded in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload SpaCy model '{model_name}': {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_model(model_name: str = "en_core_web_sm") -> Optional[object]:
|
||||
"""Get a preloaded SpaCy model if available.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model name
|
||||
|
||||
Returns:
|
||||
The preloaded SpaCy Language object, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_models.get(model_name)
|
||||
|
||||
|
||||
def is_model_preloaded(model_name: str = "en_core_web_sm") -> bool:
|
||||
"""Check if a SpaCy model has been preloaded."""
|
||||
return model_name in _preloaded_models
|
||||
|
||||
|
||||
def clear_preloaded_models() -> None:
|
||||
"""Clear all preloaded models. Useful for testing."""
|
||||
global _preloaded_models
|
||||
_preloaded_models.clear()
|
||||
@@ -46,6 +46,14 @@ CREDIT_PACKAGES = {
|
||||
},
|
||||
}
|
||||
|
||||
# Made-for-you service package (not a credit package - creates a job directly)
|
||||
MADE_FOR_YOU_PACKAGE = {
|
||||
"price_cents": 1500, # $15.00
|
||||
"name": "Made For You Karaoke Video",
|
||||
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
|
||||
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
|
||||
}
|
||||
|
||||
|
||||
class StripeService:
|
||||
"""Service for Stripe payment processing."""
|
||||
@@ -58,6 +66,10 @@ class StripeService:
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
# After consolidation, buy URL is the same as frontend URL
|
||||
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
|
||||
# Payment method configuration ID from Stripe Dashboard
|
||||
# Enables Google Pay, Apple Pay, Link, and other wallet methods
|
||||
# Get this from: Dashboard > Settings > Payment methods > [Your Config] > Copy ID
|
||||
self.payment_method_config = os.getenv("STRIPE_PAYMENT_METHOD_CONFIG")
|
||||
|
||||
if self.secret_key:
|
||||
stripe.api_key = self.secret_key
|
||||
@@ -65,6 +77,11 @@ class StripeService:
|
||||
else:
|
||||
logger.warning("STRIPE_SECRET_KEY not set - payments disabled")
|
||||
|
||||
if self.payment_method_config:
|
||||
logger.info(f"Using payment method config: {self.payment_method_config}")
|
||||
else:
|
||||
logger.info("No STRIPE_PAYMENT_METHOD_CONFIG set - using Stripe defaults")
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Stripe is properly configured."""
|
||||
return bool(self.secret_key)
|
||||
@@ -106,32 +123,40 @@ class StripeService:
|
||||
if not cancel_url:
|
||||
cancel_url = f"{self.buy_url}?cancelled=true"
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(
|
||||
payment_method_types=['card'],
|
||||
line_items=[{
|
||||
# Build checkout session params
|
||||
session_params = {
|
||||
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
|
||||
'line_items': [{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': package['name'],
|
||||
'description': package['description'],
|
||||
'images': ['https://gen.nomadkaraoke.com/nomad-logo.png'],
|
||||
},
|
||||
'unit_amount': package['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
mode='payment',
|
||||
success_url=success_url,
|
||||
cancel_url=cancel_url,
|
||||
customer_email=user_email,
|
||||
metadata={
|
||||
'mode': 'payment',
|
||||
'success_url': success_url,
|
||||
'cancel_url': cancel_url,
|
||||
'customer_email': user_email,
|
||||
'metadata': {
|
||||
'package_id': package_id,
|
||||
'credits': str(package['credits']),
|
||||
'user_email': user_email,
|
||||
},
|
||||
# Allow promotion codes
|
||||
allow_promotion_codes=True,
|
||||
)
|
||||
'allow_promotion_codes': True,
|
||||
}
|
||||
|
||||
# Add payment method configuration if set (enables Google Pay, Link, etc.)
|
||||
if self.payment_method_config:
|
||||
session_params['payment_method_configuration'] = self.payment_method_config
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(**session_params)
|
||||
|
||||
logger.info(f"Created checkout session {session.id} for {user_email}, package {package_id}")
|
||||
return True, session.url, "Checkout session created"
|
||||
@@ -143,6 +168,103 @@ class StripeService:
|
||||
logger.error(f"Error creating checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def create_made_for_you_checkout_session(
|
||||
self,
|
||||
customer_email: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
source_type: str = "search",
|
||||
youtube_url: Optional[str] = None,
|
||||
notes: Optional[str] = None,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for a made-for-you order.
|
||||
|
||||
This is for the full-service karaoke video creation where Nomad Karaoke
|
||||
handles all the work (lyrics review, instrumental selection, etc.).
|
||||
|
||||
Args:
|
||||
customer_email: Customer's email for delivery
|
||||
artist: Song artist
|
||||
title: Song title
|
||||
source_type: Audio source type (search, youtube, upload)
|
||||
youtube_url: YouTube URL if source_type is youtube
|
||||
notes: Any special requests from customer
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
try:
|
||||
# Default URLs - redirect to gen app success page
|
||||
if not success_url:
|
||||
success_url = f"{self.frontend_url}/order/success?session_id={{CHECKOUT_SESSION_ID}}"
|
||||
if not cancel_url:
|
||||
cancel_url = "https://nomadkaraoke.com/#do-it-for-me"
|
||||
|
||||
# Build metadata for job creation after payment
|
||||
metadata = {
|
||||
'order_type': 'made_for_you',
|
||||
'customer_email': customer_email,
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'source_type': source_type,
|
||||
}
|
||||
if youtube_url:
|
||||
metadata['youtube_url'] = youtube_url
|
||||
if notes:
|
||||
# Truncate notes to fit Stripe's 500 char limit per metadata value
|
||||
metadata['notes'] = notes[:500] if len(notes) > 500 else notes
|
||||
|
||||
# Build checkout session params
|
||||
session_params = {
|
||||
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
|
||||
'line_items': [{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': f"Karaoke Video: {artist} - {title}",
|
||||
'description': MADE_FOR_YOU_PACKAGE['description'],
|
||||
'images': MADE_FOR_YOU_PACKAGE['images'],
|
||||
},
|
||||
'unit_amount': MADE_FOR_YOU_PACKAGE['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
'mode': 'payment',
|
||||
'success_url': success_url,
|
||||
'cancel_url': cancel_url,
|
||||
'customer_email': customer_email,
|
||||
'metadata': metadata,
|
||||
'allow_promotion_codes': True,
|
||||
}
|
||||
|
||||
# Add payment method configuration if set (enables Google Pay, Link, etc.)
|
||||
if self.payment_method_config:
|
||||
session_params['payment_method_configuration'] = self.payment_method_config
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(**session_params)
|
||||
|
||||
logger.info(
|
||||
f"Created made-for-you checkout session {session.id} for {customer_email}, "
|
||||
f"song: {artist} - {title}"
|
||||
)
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating made-for-you checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating made-for-you checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def verify_webhook_signature(self, payload: bytes, signature: str) -> Tuple[bool, Optional[Dict], str]:
|
||||
"""
|
||||
Verify a Stripe webhook signature.
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Template service for managing email templates stored in GCS.
|
||||
|
||||
Templates are stored in GCS bucket and rendered with job-specific variables.
|
||||
This allows updating email content without code deployment.
|
||||
|
||||
Template locations:
|
||||
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
|
||||
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
|
||||
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Default templates (fallback if GCS fetch fails)
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
|
||||
|
||||
Our system has transcribed and synchronized the lyrics, but they may need some corrections. Please review and make any corrections necessary so we can finish generating your video.
|
||||
|
||||
Review your lyrics here:
|
||||
{review_url}
|
||||
|
||||
This usually takes just a few minutes.
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is almost done!
|
||||
|
||||
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
|
||||
|
||||
Select your instrumental here:
|
||||
{instrumental_url}
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
|
||||
class TemplateService:
|
||||
"""
|
||||
Service for fetching and rendering email templates from GCS.
|
||||
|
||||
Templates support the following variables:
|
||||
- {name} - User's display name or "there" if unknown
|
||||
- {youtube_url} - YouTube video URL
|
||||
- {dropbox_url} - Dropbox folder URL
|
||||
- {artist} - Artist name
|
||||
- {title} - Song title
|
||||
- {job_id} - Job ID
|
||||
- {review_url} - Lyrics review URL
|
||||
- {instrumental_url} - Instrumental selection URL
|
||||
- {feedback_url} - Feedback form URL
|
||||
"""
|
||||
|
||||
TEMPLATE_PREFIX = "templates/"
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize template service."""
|
||||
self.settings = get_settings()
|
||||
self._storage_client = None
|
||||
self._bucket = None
|
||||
|
||||
@property
|
||||
def storage_client(self):
|
||||
"""Lazy-initialize storage client."""
|
||||
if self._storage_client is None:
|
||||
from google.cloud import storage
|
||||
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
|
||||
return self._storage_client
|
||||
|
||||
@property
|
||||
def bucket(self):
|
||||
"""Get the GCS bucket for templates."""
|
||||
if self._bucket is None:
|
||||
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
|
||||
return self._bucket
|
||||
|
||||
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
|
||||
"""
|
||||
Fetch a template from GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
|
||||
Returns:
|
||||
Template content as string, or None if not found
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
if blob.exists():
|
||||
content = blob.download_as_text()
|
||||
logger.debug(f"Fetched template from GCS: {blob_path}")
|
||||
return content
|
||||
else:
|
||||
logger.warning(f"Template not found in GCS: {blob_path}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch template {blob_path}: {e}")
|
||||
return None
|
||||
|
||||
def get_job_completion_template(self) -> str:
|
||||
"""Get the job completion email template."""
|
||||
template = self._fetch_template_from_gcs("job-completion.txt")
|
||||
if template is None:
|
||||
logger.info("Using default job completion template")
|
||||
return DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_lyrics_template(self) -> str:
|
||||
"""Get the lyrics review reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
|
||||
if template is None:
|
||||
logger.info("Using default lyrics reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_instrumental_template(self) -> str:
|
||||
"""Get the instrumental selection reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
|
||||
if template is None:
|
||||
logger.info("Using default instrumental reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
return template
|
||||
|
||||
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Render a template with the given variables.
|
||||
|
||||
Missing variables are replaced with empty strings.
|
||||
Handles conditional sections like feedback URL.
|
||||
|
||||
Args:
|
||||
template: Template string with {variable} placeholders
|
||||
variables: Dictionary of variable values
|
||||
|
||||
Returns:
|
||||
Rendered template string
|
||||
"""
|
||||
result = template
|
||||
|
||||
# Handle feedback URL section - remove if not provided
|
||||
if not variables.get("feedback_url"):
|
||||
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
|
||||
result = re.sub(
|
||||
r'\n*If you have a moment.*?\{feedback_url\}\n*',
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["feedback_url"] = ""
|
||||
|
||||
# Replace all variables
|
||||
for key, value in variables.items():
|
||||
placeholder = "{" + key + "}"
|
||||
result = result.replace(placeholder, str(value) if value else "")
|
||||
|
||||
# Clean up any remaining unreplaced placeholders
|
||||
result = re.sub(r'\{[a-z_]+\}', '', result)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def render_job_completion(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
job_id: Optional[str] = None,
|
||||
feedback_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Render the job completion email template.
|
||||
|
||||
Args:
|
||||
name: User's display name (defaults to "there")
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
job_id: Job ID
|
||||
feedback_url: Feedback form URL (optional)
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_job_completion_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"job_id": job_id or "",
|
||||
"feedback_url": feedback_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_lyrics(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
review_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the lyrics review reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
review_url: Lyrics review URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_lyrics_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"review_url": review_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_instrumental(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
instrumental_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the instrumental selection reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
instrumental_url: Instrumental selection URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_instrumental_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"instrumental_url": instrumental_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def upload_template(self, template_name: str, content: str) -> bool:
|
||||
"""
|
||||
Upload a template to GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
content: Template content
|
||||
|
||||
Returns:
|
||||
True if upload successful
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.upload_from_string(content, content_type="text/plain")
|
||||
logger.info(f"Uploaded template to GCS: {blob_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload template {blob_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_template_service: Optional[TemplateService] = None
|
||||
|
||||
|
||||
def get_template_service() -> TemplateService:
|
||||
"""Get the global template service instance."""
|
||||
global _template_service
|
||||
if _template_service is None:
|
||||
_template_service = TemplateService()
|
||||
return _template_service
|
||||
@@ -0,0 +1,285 @@
|
||||
"""
|
||||
Tenant service for managing white-label B2B portal configurations.
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json.
|
||||
The service provides:
|
||||
- Tenant config loading with caching
|
||||
- Subdomain to tenant ID resolution
|
||||
- Tenant validation
|
||||
|
||||
Similar to ThemeService, configs are cached for 5 minutes to reduce GCS reads.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from typing import Dict, Optional
|
||||
|
||||
from backend.models.tenant import TenantConfig, TenantPublicConfig
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for tenant configs
|
||||
TENANTS_PREFIX = "tenants"
|
||||
|
||||
# Default sender email (shared with EmailService for consistency)
|
||||
DEFAULT_SENDER_EMAIL = "gen@nomadkaraoke.com"
|
||||
|
||||
|
||||
class TenantService:
|
||||
"""Service for managing tenant configurations from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the tenant service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._config_cache: Dict[str, TenantConfig] = {}
|
||||
self._cache_times: Dict[str, datetime] = {}
|
||||
self._subdomain_map: Dict[str, str] = {} # subdomain -> tenant_id
|
||||
self._subdomain_map_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache
|
||||
|
||||
def _get_config_path(self, tenant_id: str) -> str:
|
||||
"""Get the GCS path for a tenant's config file."""
|
||||
return f"{TENANTS_PREFIX}/{tenant_id}/config.json"
|
||||
|
||||
def _is_cache_valid(self, tenant_id: str) -> bool:
|
||||
"""Check if the config cache for a tenant is still valid."""
|
||||
if tenant_id not in self._config_cache or tenant_id not in self._cache_times:
|
||||
return False
|
||||
age = datetime.now() - self._cache_times[tenant_id]
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _is_subdomain_map_valid(self) -> bool:
|
||||
"""Check if the subdomain map cache is still valid."""
|
||||
if not self._subdomain_map or self._subdomain_map_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._subdomain_map_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def get_tenant_config(
|
||||
self, tenant_id: str, force_refresh: bool = False
|
||||
) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Load tenant configuration from GCS with caching.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
TenantConfig if found, None otherwise
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid(tenant_id):
|
||||
return self._config_cache[tenant_id]
|
||||
|
||||
try:
|
||||
config_path = self._get_config_path(tenant_id)
|
||||
if not self.storage.file_exists(config_path):
|
||||
logger.debug(f"Tenant config not found: {tenant_id}")
|
||||
return None
|
||||
|
||||
data = self.storage.download_json(config_path)
|
||||
config = TenantConfig(**data)
|
||||
|
||||
# Update cache
|
||||
self._config_cache[tenant_id] = config
|
||||
self._cache_times[tenant_id] = datetime.now()
|
||||
|
||||
# Update subdomain map and its timestamp
|
||||
self._subdomain_map[config.subdomain.lower()] = tenant_id
|
||||
self._subdomain_map_time = datetime.now()
|
||||
|
||||
logger.info(f"Loaded tenant config: {tenant_id}")
|
||||
return config
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load tenant config {tenant_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_tenant_by_subdomain(self, subdomain: str) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Get tenant config by subdomain.
|
||||
|
||||
Args:
|
||||
subdomain: Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')
|
||||
|
||||
Returns:
|
||||
TenantConfig if found, None otherwise
|
||||
"""
|
||||
subdomain_lower = subdomain.lower()
|
||||
|
||||
# Check cache first
|
||||
if subdomain_lower in self._subdomain_map:
|
||||
tenant_id = self._subdomain_map[subdomain_lower]
|
||||
return self.get_tenant_config(tenant_id)
|
||||
|
||||
# Try to find tenant by listing all configs
|
||||
# This is a fallback for when we don't have the subdomain cached
|
||||
tenant_id = self._resolve_subdomain_to_tenant_id(subdomain_lower)
|
||||
if tenant_id:
|
||||
return self.get_tenant_config(tenant_id)
|
||||
|
||||
return None
|
||||
|
||||
def _resolve_subdomain_to_tenant_id(self, subdomain: str) -> Optional[str]:
|
||||
"""
|
||||
Resolve a subdomain to tenant ID by checking GCS.
|
||||
|
||||
This extracts the tenant ID from the subdomain pattern:
|
||||
- 'vocalstar.nomadkaraoke.com' -> 'vocalstar'
|
||||
- 'vocalstar.gen.nomadkaraoke.com' -> 'vocalstar'
|
||||
"""
|
||||
# Extract potential tenant ID from subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
parts = subdomain.split(".")
|
||||
|
||||
if len(parts) >= 3:
|
||||
# First part is likely the tenant ID
|
||||
potential_tenant_id = parts[0]
|
||||
|
||||
# Check if config exists
|
||||
config_path = self._get_config_path(potential_tenant_id)
|
||||
if self.storage.file_exists(config_path):
|
||||
return potential_tenant_id
|
||||
|
||||
return None
|
||||
|
||||
def get_public_config(self, tenant_id: str) -> Optional[TenantPublicConfig]:
|
||||
"""
|
||||
Get the public (frontend-safe) tenant configuration.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
TenantPublicConfig if found, None otherwise
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if config:
|
||||
return TenantPublicConfig.from_config(config)
|
||||
return None
|
||||
|
||||
def get_public_config_by_subdomain(
|
||||
self, subdomain: str
|
||||
) -> Optional[TenantPublicConfig]:
|
||||
"""
|
||||
Get the public tenant configuration by subdomain.
|
||||
|
||||
Args:
|
||||
subdomain: Full subdomain
|
||||
|
||||
Returns:
|
||||
TenantPublicConfig if found, None otherwise
|
||||
"""
|
||||
config = self.get_tenant_by_subdomain(subdomain)
|
||||
if config:
|
||||
return TenantPublicConfig.from_config(config)
|
||||
return None
|
||||
|
||||
def tenant_exists(self, tenant_id: str) -> bool:
|
||||
"""
|
||||
Check if a tenant exists.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
True if tenant config exists, False otherwise
|
||||
"""
|
||||
config_path = self._get_config_path(tenant_id)
|
||||
return self.storage.file_exists(config_path)
|
||||
|
||||
def is_email_allowed_for_tenant(self, tenant_id: str, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is allowed for a specific tenant.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if email is allowed, False otherwise
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if not config:
|
||||
return False
|
||||
return config.is_email_allowed(email)
|
||||
|
||||
def get_tenant_sender_email(self, tenant_id: str) -> str:
|
||||
"""
|
||||
Get the email sender address for a tenant.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
Sender email address
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if config:
|
||||
return config.get_sender_email()
|
||||
# Default fallback (consistent with EmailService)
|
||||
return DEFAULT_SENDER_EMAIL
|
||||
|
||||
def invalidate_cache(self, tenant_id: Optional[str] = None) -> None:
|
||||
"""
|
||||
Invalidate tenant config cache.
|
||||
|
||||
Args:
|
||||
tenant_id: Specific tenant to invalidate, or None for all
|
||||
"""
|
||||
if tenant_id:
|
||||
self._config_cache.pop(tenant_id, None)
|
||||
self._cache_times.pop(tenant_id, None)
|
||||
# Also remove any subdomain map entries pointing to this tenant
|
||||
subdomains_to_remove = [
|
||||
subdomain
|
||||
for subdomain, tid in self._subdomain_map.items()
|
||||
if tid == tenant_id
|
||||
]
|
||||
for subdomain in subdomains_to_remove:
|
||||
self._subdomain_map.pop(subdomain, None)
|
||||
logger.info(f"Tenant config cache invalidated: {tenant_id}")
|
||||
else:
|
||||
self._config_cache.clear()
|
||||
self._cache_times.clear()
|
||||
self._subdomain_map.clear()
|
||||
self._subdomain_map_time = None
|
||||
logger.info("All tenant config caches invalidated")
|
||||
|
||||
def get_asset_url(self, tenant_id: str, asset_name: str) -> Optional[str]:
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
asset_name: Asset filename (e.g., 'logo.png')
|
||||
|
||||
Returns:
|
||||
Signed URL if asset exists, None otherwise
|
||||
"""
|
||||
asset_path = f"{TENANTS_PREFIX}/{tenant_id}/{asset_name}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
return self.storage.generate_signed_url(asset_path, expiration_minutes=60)
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_tenant_service: Optional[TenantService] = None
|
||||
_tenant_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_tenant_service() -> TenantService:
|
||||
"""Get or create the singleton TenantService instance (thread-safe)."""
|
||||
global _tenant_service
|
||||
if _tenant_service is None:
|
||||
with _tenant_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _tenant_service is None:
|
||||
_tenant_service = TenantService()
|
||||
return _tenant_service
|
||||
@@ -73,16 +73,26 @@ class UserService:
|
||||
logger.exception(f"Error getting user {email}")
|
||||
return None
|
||||
|
||||
def get_or_create_user(self, email: str) -> User:
|
||||
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
|
||||
"""
|
||||
Get existing user or create a new one.
|
||||
|
||||
New users receive a welcome credit to try the service.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
|
||||
Note: If user exists but has a different tenant_id, the existing user is returned.
|
||||
Users are uniquely identified by email, not email+tenant.
|
||||
"""
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
|
||||
if user:
|
||||
# If user exists but tenant_id differs, we still return the user
|
||||
# This allows users to access multiple tenants with one account
|
||||
# (though features may be restricted per-tenant)
|
||||
return user
|
||||
|
||||
# Create new user with welcome credit
|
||||
@@ -97,9 +107,10 @@ class UserService:
|
||||
email=email,
|
||||
credits=welcome_credit,
|
||||
credit_transactions=[welcome_transaction],
|
||||
tenant_id=tenant_id, # Associate with tenant on creation
|
||||
)
|
||||
self._save_user(user)
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s)")
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
|
||||
return user
|
||||
|
||||
def _save_user(self, user: User) -> None:
|
||||
@@ -153,6 +164,7 @@ class UserService:
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
tenant_id=user.tenant_id,
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
@@ -163,18 +175,25 @@ class UserService:
|
||||
self,
|
||||
email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create a magic link token for email authentication.
|
||||
|
||||
Returns the token object. The actual sending of the email
|
||||
should be handled by the caller (or an email service).
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
"""
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists
|
||||
self.get_or_create_user(email)
|
||||
# Ensure user exists (with tenant association)
|
||||
self.get_or_create_user(email, tenant_id=tenant_id)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
@@ -185,6 +204,7 @@ class UserService:
|
||||
expires_at=datetime.utcnow() + timedelta(minutes=MAGIC_LINK_EXPIRY_MINUTES),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
# Save to Firestore
|
||||
@@ -194,6 +214,54 @@ class UserService:
|
||||
logger.info(f"Created magic link for {email}")
|
||||
return magic_link
|
||||
|
||||
def create_admin_login_token(
|
||||
self,
|
||||
email: str,
|
||||
expiry_hours: int = 24,
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create an admin login token for email-embedded authentication links.
|
||||
|
||||
Similar to magic links but with configurable expiry (default 24 hours).
|
||||
Used for made-for-you order notification emails to allow admin one-click login.
|
||||
|
||||
Args:
|
||||
email: Admin's email address to authenticate as
|
||||
expiry_hours: Hours until token expires (default: 24, max: 168)
|
||||
|
||||
Returns:
|
||||
MagicLinkToken object containing the token
|
||||
|
||||
Raises:
|
||||
ValueError: If expiry_hours is out of valid range (1-168)
|
||||
"""
|
||||
# Validate expiry_hours (1 hour to 7 days)
|
||||
if not 1 <= expiry_hours <= 168:
|
||||
raise ValueError(f"expiry_hours must be between 1 and 168, got {expiry_hours}")
|
||||
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists
|
||||
self.get_or_create_user(email)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
|
||||
admin_login = MagicLinkToken(
|
||||
token=token,
|
||||
email=email,
|
||||
expires_at=datetime.utcnow() + timedelta(hours=expiry_hours),
|
||||
)
|
||||
|
||||
# Save to Firestore (same collection as magic links for unified verification)
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
doc_ref.set(admin_login.model_dump(mode='json'))
|
||||
|
||||
# Log with redacted email (show only domain) for PII protection
|
||||
domain = email.split('@')[-1] if '@' in email else 'unknown'
|
||||
logger.info(f"Created admin login token for ***@{domain} (expires in {expiry_hours}h)")
|
||||
return admin_login
|
||||
|
||||
def verify_magic_link(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Verify a magic link token using a Firestore transaction to prevent race conditions.
|
||||
@@ -262,9 +330,18 @@ class UserService:
|
||||
self,
|
||||
user_email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> Session:
|
||||
"""Create a new session for an authenticated user."""
|
||||
"""
|
||||
Create a new session for an authenticated user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
"""
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_prefix = token[:12]
|
||||
|
||||
@@ -274,6 +351,7 @@ class UserService:
|
||||
expires_at=datetime.utcnow() + timedelta(days=SESSION_ABSOLUTE_EXPIRY_DAYS),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
# Serialize and write to Firestore
|
||||
|
||||
@@ -37,6 +37,7 @@ WORKER_QUEUES = {
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
@@ -49,8 +50,12 @@ WORKER_DISPATCH_DEADLINES = {
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
"idle-reminder": 60, # 1 min - Quick check and potential email send
|
||||
}
|
||||
|
||||
# Default delay for idle reminders (seconds)
|
||||
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
@@ -430,6 +435,99 @@ class WorkerService:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
async def schedule_idle_reminder(
|
||||
self,
|
||||
job_id: str,
|
||||
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
|
||||
) -> bool:
|
||||
"""
|
||||
Schedule an idle reminder check for a job.
|
||||
|
||||
The reminder task will be delivered after the specified delay.
|
||||
When the task executes, it checks if the job is still in a blocking
|
||||
state and sends a reminder email if the user hasn't taken action.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
delay_seconds: Delay before executing the check (default: 5 minutes)
|
||||
|
||||
Returns:
|
||||
True if task was scheduled successfully, False otherwise
|
||||
"""
|
||||
if not self._use_cloud_tasks:
|
||||
# In development mode, log and skip (no delayed execution support)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
|
||||
f"Would have fired in {delay_seconds}s."
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import time
|
||||
|
||||
queue_name = WORKER_QUEUES.get("idle-reminder")
|
||||
if not queue_name:
|
||||
logger.error("Idle reminder queue not configured")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
|
||||
# Inject trace context
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_time = timestamp_pb2.Timestamp()
|
||||
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
|
||||
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
"schedule_time": schedule_time,
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
@@ -0,0 +1,349 @@
|
||||
"""
|
||||
YouTube download service for cloud backend.
|
||||
|
||||
This service provides a single entry point for all YouTube downloads in the cloud.
|
||||
It automatically routes downloads through the remote flacfetch service when configured,
|
||||
avoiding YouTube bot detection issues on Cloud Run IPs.
|
||||
|
||||
The flow is:
|
||||
1. Check if remote flacfetch is configured (FLACFETCH_API_URL)
|
||||
2. If yes: Use FlacfetchClient.download_by_id() - downloads on VM, uploads to GCS
|
||||
3. If no: Use local yt_dlp (with warning about bot detection risk)
|
||||
|
||||
All entry points (audio search selection, direct URL submission) should use this
|
||||
service for YouTube downloads to ensure consistent behavior.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import mimetypes
|
||||
import os
|
||||
import re
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
from .storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeDownloadError(Exception):
|
||||
"""Error downloading from YouTube."""
|
||||
pass
|
||||
|
||||
|
||||
class YouTubeDownloadService:
|
||||
"""
|
||||
Single point of entry for all YouTube downloads in the cloud backend.
|
||||
|
||||
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
|
||||
on the flacfetch VM which has YouTube cookies and avoids bot detection.
|
||||
|
||||
When remote is not configured, falls back to local yt_dlp with a warning
|
||||
that downloads may be blocked on Cloud Run IPs.
|
||||
|
||||
Usage:
|
||||
service = get_youtube_download_service()
|
||||
gcs_path = await service.download(
|
||||
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
||||
job_id="job123",
|
||||
artist="Rick Astley",
|
||||
title="Never Gonna Give You Up",
|
||||
)
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._flacfetch_client = get_flacfetch_client()
|
||||
self._storage_service = StorageService()
|
||||
|
||||
if self._flacfetch_client:
|
||||
logger.info("YouTubeDownloadService using REMOTE flacfetch (recommended)")
|
||||
else:
|
||||
logger.warning(
|
||||
"YouTubeDownloadService using LOCAL yt_dlp - "
|
||||
"downloads may fail due to YouTube bot detection on Cloud Run IPs"
|
||||
)
|
||||
|
||||
def is_remote_enabled(self) -> bool:
|
||||
"""Check if remote flacfetch is configured."""
|
||||
return self._flacfetch_client is not None
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio and upload to GCS.
|
||||
|
||||
Args:
|
||||
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (not gs:// prefix, just the path portion)
|
||||
Example: "uploads/job123/audio/Artist - Title.flac"
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
video_id = self._extract_video_id(url)
|
||||
if not video_id:
|
||||
raise YouTubeDownloadError(f"Could not extract video ID from URL: {url}")
|
||||
|
||||
logger.info(f"YouTube download: video_id={video_id}, job_id={job_id}")
|
||||
|
||||
if self._flacfetch_client:
|
||||
return await self._download_remote(video_id, job_id, artist, title)
|
||||
else:
|
||||
logger.warning(
|
||||
"Remote flacfetch not configured. Attempting local download, "
|
||||
"but this may fail due to YouTube bot detection on Cloud Run IPs."
|
||||
)
|
||||
return await self._download_local(url, job_id, artist, title)
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio by video ID.
|
||||
|
||||
Same as download() but takes a video ID directly instead of URL.
|
||||
|
||||
Args:
|
||||
video_id: YouTube video ID (e.g., "dQw4w9WgXcQ")
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (path portion only, no gs:// prefix)
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
return await self.download(url, job_id, artist, title)
|
||||
|
||||
def _extract_video_id(self, url: str) -> Optional[str]:
|
||||
"""
|
||||
Extract YouTube video ID from various URL formats.
|
||||
|
||||
Supports:
|
||||
- https://www.youtube.com/watch?v=VIDEO_ID
|
||||
- https://youtu.be/VIDEO_ID
|
||||
- https://www.youtube.com/shorts/VIDEO_ID
|
||||
- https://www.youtube.com/embed/VIDEO_ID
|
||||
- https://youtube.com/v/VIDEO_ID
|
||||
|
||||
Returns:
|
||||
Video ID string, or None if extraction fails
|
||||
"""
|
||||
patterns = [
|
||||
# Standard watch URL
|
||||
r'(?:youtube\.com/watch\?v=|youtube\.com/watch\?.*&v=)([a-zA-Z0-9_-]{11})',
|
||||
# Short URL
|
||||
r'youtu\.be/([a-zA-Z0-9_-]{11})',
|
||||
# Shorts URL
|
||||
r'youtube\.com/shorts/([a-zA-Z0-9_-]{11})',
|
||||
# Embed URL
|
||||
r'youtube\.com/embed/([a-zA-Z0-9_-]{11})',
|
||||
# Old-style URL
|
||||
r'youtube\.com/v/([a-zA-Z0-9_-]{11})',
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
match = re.search(pattern, url)
|
||||
if match:
|
||||
return match.group(1)
|
||||
|
||||
return None
|
||||
|
||||
async def _download_remote(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Download using remote flacfetch service."""
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
# Build output filename if artist/title provided
|
||||
output_filename = None
|
||||
if artist and title:
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
safe_artist = sanitize_filename(artist)
|
||||
safe_title = sanitize_filename(title)
|
||||
output_filename = f"{safe_artist} - {safe_title}"
|
||||
|
||||
logger.info(
|
||||
f"Remote YouTube download: video_id={video_id}, "
|
||||
f"gcs_path={gcs_destination}, filename={output_filename}"
|
||||
)
|
||||
|
||||
try:
|
||||
# Start download
|
||||
download_id = await self._flacfetch_client.download_by_id(
|
||||
source_name="YouTube",
|
||||
source_id=video_id,
|
||||
output_filename=output_filename,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
|
||||
logger.info(f"Remote download started: {download_id}")
|
||||
|
||||
# Wait for completion
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = await self._flacfetch_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=300, # 5 minute timeout for YouTube downloads
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
|
||||
# Extract GCS path from response
|
||||
gcs_path = final_status.get("gcs_path")
|
||||
if not gcs_path:
|
||||
raise YouTubeDownloadError(
|
||||
"Remote download completed but no GCS path returned"
|
||||
)
|
||||
|
||||
# Convert gs:// URL to path portion
|
||||
if gcs_path.startswith("gs://"):
|
||||
parts = gcs_path.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
gcs_path = parts[1]
|
||||
|
||||
logger.info(f"Remote YouTube download complete: {gcs_path}")
|
||||
return gcs_path
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise YouTubeDownloadError(f"Remote download failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote YouTube download error: {e}", exc_info=True)
|
||||
raise YouTubeDownloadError(f"Remote download failed: {e}") from e
|
||||
|
||||
async def _download_local(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download using local yt_dlp (fallback when remote not configured).
|
||||
|
||||
Warning: This may fail on Cloud Run due to YouTube bot detection.
|
||||
"""
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"youtube_{job_id}_")
|
||||
|
||||
try:
|
||||
from karaoke_gen.file_handler import FileHandler
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
# Create FileHandler
|
||||
file_handler = FileHandler(
|
||||
logger=logger,
|
||||
ffmpeg_base_command="ffmpeg -hide_banner -loglevel error -nostats -y",
|
||||
create_track_subfolders=False,
|
||||
dry_run=False
|
||||
)
|
||||
|
||||
# Build output filename
|
||||
safe_artist = sanitize_filename(artist) if artist else "Unknown"
|
||||
safe_title = sanitize_filename(title) if title else "Unknown"
|
||||
output_filename_no_extension = os.path.join(
|
||||
temp_dir, f"{safe_artist} - {safe_title}"
|
||||
)
|
||||
|
||||
# Get cookies from environment
|
||||
cookies_str = os.environ.get("YOUTUBE_COOKIES")
|
||||
if cookies_str:
|
||||
logger.info("Using YouTube cookies for local download")
|
||||
|
||||
# Download
|
||||
logger.info(f"Local YouTube download: {url}")
|
||||
downloaded_file = file_handler.download_video(
|
||||
url=url,
|
||||
output_filename_no_extension=output_filename_no_extension,
|
||||
cookies_str=cookies_str
|
||||
)
|
||||
|
||||
if not downloaded_file or not os.path.exists(downloaded_file):
|
||||
raise YouTubeDownloadError(
|
||||
f"Local download failed - no file returned. "
|
||||
"This is likely due to YouTube bot detection on Cloud Run IPs. "
|
||||
"Configure FLACFETCH_API_URL for remote downloads."
|
||||
)
|
||||
|
||||
logger.info(f"Downloaded video: {downloaded_file}")
|
||||
|
||||
# Convert to WAV for processing
|
||||
wav_file = file_handler.convert_to_wav(
|
||||
input_filename=downloaded_file,
|
||||
output_filename_no_extension=output_filename_no_extension
|
||||
)
|
||||
|
||||
if not wav_file or not os.path.exists(wav_file):
|
||||
raise YouTubeDownloadError("WAV conversion failed")
|
||||
|
||||
logger.info(f"Converted to WAV: {wav_file}")
|
||||
|
||||
# Upload to GCS
|
||||
filename = os.path.basename(wav_file)
|
||||
gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
content_type, _ = mimetypes.guess_type(wav_file)
|
||||
with open(wav_file, 'rb') as f:
|
||||
self._storage_service.upload_fileobj(
|
||||
f,
|
||||
gcs_path,
|
||||
content_type=content_type or 'audio/wav'
|
||||
)
|
||||
|
||||
logger.info(f"Uploaded to GCS: {gcs_path}")
|
||||
return gcs_path
|
||||
|
||||
except YouTubeDownloadError:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Local YouTube download error: {e}", exc_info=True)
|
||||
raise YouTubeDownloadError(f"Local download failed: {e}") from e
|
||||
finally:
|
||||
# Cleanup temp directory
|
||||
try:
|
||||
import shutil
|
||||
if os.path.exists(temp_dir):
|
||||
shutil.rmtree(temp_dir)
|
||||
except Exception as cleanup_error:
|
||||
logger.warning(f"Failed to cleanup temp dir: {cleanup_error}")
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_youtube_download_service: Optional[YouTubeDownloadService] = None
|
||||
|
||||
|
||||
def get_youtube_download_service() -> YouTubeDownloadService:
|
||||
"""Get the singleton YouTubeDownloadService instance."""
|
||||
global _youtube_download_service
|
||||
if _youtube_download_service is None:
|
||||
_youtube_download_service = YouTubeDownloadService()
|
||||
return _youtube_download_service
|
||||
|
||||
|
||||
def reset_youtube_download_service():
|
||||
"""Reset the singleton (for testing)."""
|
||||
global _youtube_download_service
|
||||
_youtube_download_service = None
|
||||
@@ -91,7 +91,14 @@ def mock_auth_dependency(request):
|
||||
if 'emulator' in test_path or 'integration' in test_path:
|
||||
yield
|
||||
return
|
||||
|
||||
|
||||
# Skip for service-only unit tests that don't need the FastAPI app
|
||||
# Also skip tests that manage their own auth mocks (e.g., push tests use AuthResult objects)
|
||||
service_only_tests = ['test_rate_limit_service', 'test_email_validation_service', 'test_rate_limits_api', 'test_gce_encoding_worker', 'test_push_routes', 'test_push_notification_service']
|
||||
if any(test_name in test_path for test_name in service_only_tests):
|
||||
yield
|
||||
return
|
||||
|
||||
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
|
||||
import os
|
||||
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
"""E2E tests for backend functionality."""
|
||||
@@ -0,0 +1,360 @@
|
||||
"""
|
||||
E2E tests that verify actual audio synchronization in generated videos.
|
||||
|
||||
These tests generate real (short) videos and verify the audio offset.
|
||||
Uses ffprobe to analyze video/audio streams.
|
||||
|
||||
These tests are SLOW and should be run on-demand, not on every PR.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import subprocess
|
||||
import tempfile
|
||||
import os
|
||||
import json
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
def ffprobe_available():
|
||||
"""Check if ffprobe is available."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['ffprobe', '-version'],
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
return result.returncode == 0
|
||||
except FileNotFoundError:
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests if ffprobe not available
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not ffprobe_available(),
|
||||
reason="ffprobe not available. Install ffmpeg to run these tests."
|
||||
)
|
||||
|
||||
|
||||
class TestAudioFileAnalysis:
|
||||
"""Helper tests for audio file analysis functions."""
|
||||
|
||||
def test_ffprobe_can_analyze_audio(self, tmp_path):
|
||||
"""Verify ffprobe can analyze audio files."""
|
||||
# Create a silent audio file for testing
|
||||
test_audio = tmp_path / "test.wav"
|
||||
|
||||
# Generate 5 seconds of silence using ffmpeg
|
||||
result = subprocess.run([
|
||||
'ffmpeg', '-y', '-f', 'lavfi',
|
||||
'-i', 'anullsrc=r=44100:cl=stereo',
|
||||
'-t', '5',
|
||||
str(test_audio)
|
||||
], capture_output=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
pytest.skip("Could not create test audio file")
|
||||
|
||||
# Analyze with ffprobe
|
||||
probe_result = subprocess.run([
|
||||
'ffprobe', '-v', 'quiet',
|
||||
'-print_format', 'json',
|
||||
'-show_format',
|
||||
str(test_audio)
|
||||
], capture_output=True, text=True)
|
||||
|
||||
assert probe_result.returncode == 0
|
||||
data = json.loads(probe_result.stdout)
|
||||
assert 'format' in data
|
||||
duration = float(data['format']['duration'])
|
||||
assert 4.9 < duration < 5.1 # ~5 seconds
|
||||
|
||||
|
||||
class TestCountdownAudioPadding:
|
||||
"""Tests for audio padding functionality."""
|
||||
|
||||
@pytest.fixture
|
||||
def source_audio(self, tmp_path):
|
||||
"""Create a short source audio file."""
|
||||
audio_path = tmp_path / "source.flac"
|
||||
|
||||
# Create a 5-second audio file with a tone at the start
|
||||
# This simulates audio content that starts immediately
|
||||
result = subprocess.run([
|
||||
'ffmpeg', '-y', '-f', 'lavfi',
|
||||
'-i', 'sine=frequency=440:duration=5',
|
||||
'-c:a', 'flac',
|
||||
str(audio_path)
|
||||
], capture_output=True)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"FFmpeg failed to create source audio: {result.stderr.decode()}")
|
||||
|
||||
return audio_path
|
||||
|
||||
def test_padding_adds_correct_duration(self, source_audio, tmp_path):
|
||||
"""Verify that padding adds exactly the specified duration."""
|
||||
padded_audio = tmp_path / "padded.flac"
|
||||
padding_seconds = 3.0
|
||||
|
||||
# Apply padding using the same method as CountdownProcessor
|
||||
# adelay filter adds delay in milliseconds
|
||||
delay_ms = int(padding_seconds * 1000)
|
||||
|
||||
result = subprocess.run([
|
||||
'ffmpeg', '-y',
|
||||
'-i', str(source_audio),
|
||||
'-af', f'adelay={delay_ms}|{delay_ms}',
|
||||
'-c:a', 'flac',
|
||||
str(padded_audio)
|
||||
], capture_output=True)
|
||||
|
||||
assert result.returncode == 0, f"FFmpeg failed: {result.stderr}"
|
||||
|
||||
# Analyze both files
|
||||
def get_duration(path):
|
||||
probe = subprocess.run([
|
||||
'ffprobe', '-v', 'quiet',
|
||||
'-print_format', 'json',
|
||||
'-show_format',
|
||||
str(path)
|
||||
], capture_output=True, text=True)
|
||||
data = json.loads(probe.stdout)
|
||||
return float(data['format']['duration'])
|
||||
|
||||
source_duration = get_duration(source_audio)
|
||||
padded_duration = get_duration(padded_audio)
|
||||
|
||||
# Padded should be source + padding
|
||||
expected_duration = source_duration + padding_seconds
|
||||
assert abs(padded_duration - expected_duration) < 0.1, (
|
||||
f"Expected duration ~{expected_duration}s, got {padded_duration}s"
|
||||
)
|
||||
|
||||
def test_karaoke_finalise_pad_audio_file(self, source_audio, tmp_path):
|
||||
"""Test KaraokeFinalise._pad_audio_file creates correctly padded audio."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
padded_audio = tmp_path / "padded_via_finalise.flac"
|
||||
padding_seconds = 3.0
|
||||
|
||||
finalise = KaraokeFinalise(non_interactive=True)
|
||||
finalise._pad_audio_file(str(source_audio), str(padded_audio), padding_seconds)
|
||||
|
||||
assert padded_audio.exists(), "Padded audio file was not created"
|
||||
|
||||
# Verify duration
|
||||
probe = subprocess.run([
|
||||
'ffprobe', '-v', 'quiet',
|
||||
'-print_format', 'json',
|
||||
'-show_format',
|
||||
str(padded_audio)
|
||||
], capture_output=True, text=True)
|
||||
|
||||
data = json.loads(probe.stdout)
|
||||
padded_duration = float(data['format']['duration'])
|
||||
|
||||
# Get source duration
|
||||
source_probe = subprocess.run([
|
||||
'ffprobe', '-v', 'quiet',
|
||||
'-print_format', 'json',
|
||||
'-show_format',
|
||||
str(source_audio)
|
||||
], capture_output=True, text=True)
|
||||
|
||||
source_data = json.loads(source_probe.stdout)
|
||||
source_duration = float(source_data['format']['duration'])
|
||||
|
||||
expected = source_duration + padding_seconds
|
||||
assert abs(padded_duration - expected) < 0.1, (
|
||||
f"Expected ~{expected}s, got {padded_duration}s"
|
||||
)
|
||||
|
||||
|
||||
class TestAudioSyncVerification:
|
||||
"""
|
||||
Tests that verify the final audio sync between vocals and instrumental.
|
||||
|
||||
These tests create test audio files and verify that when countdown is added:
|
||||
1. Vocals are padded by countdown duration
|
||||
2. Instrumental is padded by same duration
|
||||
3. Relative timing is preserved
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def vocals_audio(self, tmp_path):
|
||||
"""Create vocals audio with a distinctive marker at 0.5s."""
|
||||
audio_path = tmp_path / "vocals.flac"
|
||||
|
||||
# Create audio: 0.5s silence, then 1s tone, then silence
|
||||
# Total 5 seconds
|
||||
filter_complex = (
|
||||
'aevalsrc=0:d=0.5[silence1];'
|
||||
'sine=frequency=880:duration=1[tone];'
|
||||
'aevalsrc=0:d=3.5[silence2];'
|
||||
'[silence1][tone][silence2]concat=n=3:v=0:a=1'
|
||||
)
|
||||
|
||||
subprocess.run([
|
||||
'ffmpeg', '-y', '-f', 'lavfi',
|
||||
'-i', filter_complex,
|
||||
'-c:a', 'flac',
|
||||
str(audio_path)
|
||||
], capture_output=True)
|
||||
|
||||
return audio_path
|
||||
|
||||
@pytest.fixture
|
||||
def instrumental_audio(self, tmp_path):
|
||||
"""Create instrumental audio with marker at 0.5s."""
|
||||
audio_path = tmp_path / "instrumental.flac"
|
||||
|
||||
filter_complex = (
|
||||
'aevalsrc=0:d=0.5[silence1];'
|
||||
'sine=frequency=440:duration=1[tone];'
|
||||
'aevalsrc=0:d=3.5[silence2];'
|
||||
'[silence1][tone][silence2]concat=n=3:v=0:a=1'
|
||||
)
|
||||
|
||||
subprocess.run([
|
||||
'ffmpeg', '-y', '-f', 'lavfi',
|
||||
'-i', filter_complex,
|
||||
'-c:a', 'flac',
|
||||
str(audio_path)
|
||||
], capture_output=True)
|
||||
|
||||
return audio_path
|
||||
|
||||
def test_both_tracks_padded_equally(self, vocals_audio, instrumental_audio, tmp_path):
|
||||
"""
|
||||
When countdown is applied, both vocals and instrumental must be
|
||||
padded by the same amount to maintain sync.
|
||||
"""
|
||||
padding_seconds = 3.0
|
||||
delay_ms = int(padding_seconds * 1000)
|
||||
|
||||
padded_vocals = tmp_path / "padded_vocals.flac"
|
||||
padded_instrumental = tmp_path / "padded_instrumental.flac"
|
||||
|
||||
# Pad both
|
||||
for src, dst in [(vocals_audio, padded_vocals), (instrumental_audio, padded_instrumental)]:
|
||||
subprocess.run([
|
||||
'ffmpeg', '-y',
|
||||
'-i', str(src),
|
||||
'-af', f'adelay={delay_ms}|{delay_ms}',
|
||||
'-c:a', 'flac',
|
||||
str(dst)
|
||||
], capture_output=True)
|
||||
|
||||
# Get durations
|
||||
def get_duration(path):
|
||||
probe = subprocess.run([
|
||||
'ffprobe', '-v', 'quiet',
|
||||
'-print_format', 'json',
|
||||
'-show_format',
|
||||
str(path)
|
||||
], capture_output=True, text=True)
|
||||
data = json.loads(probe.stdout)
|
||||
return float(data['format']['duration'])
|
||||
|
||||
vocals_dur = get_duration(padded_vocals)
|
||||
inst_dur = get_duration(padded_instrumental)
|
||||
|
||||
# Both should have same duration (sync maintained)
|
||||
assert abs(vocals_dur - inst_dur) < 0.1, (
|
||||
f"Padded vocals ({vocals_dur}s) and instrumental ({inst_dur}s) "
|
||||
"have different durations - sync will be off"
|
||||
)
|
||||
|
||||
|
||||
class TestCountdownTimingInvariant:
|
||||
"""
|
||||
Tests for the critical audio timing invariant.
|
||||
|
||||
INVARIANT: If vocals start at time T (after countdown),
|
||||
instrumental must also start at time T (also padded).
|
||||
|
||||
Violation of this invariant causes audio desync.
|
||||
"""
|
||||
|
||||
def test_relative_timing_preserved_after_padding(self):
|
||||
"""
|
||||
The relative timing between vocals and instrumental must be preserved.
|
||||
|
||||
If vocals originally start at 0.5s with content at 0.5s,
|
||||
after 3s padding, vocals content is at 3.5s.
|
||||
Instrumental must also be padded so its content is at 3.5s.
|
||||
"""
|
||||
original_content_start = 0.5 # seconds
|
||||
padding_seconds = 3.0
|
||||
|
||||
# After padding
|
||||
vocals_content_after_padding = original_content_start + padding_seconds
|
||||
instrumental_start_after_padding = 0 + padding_seconds # instrumental starts at 0
|
||||
|
||||
# The relative offset should be preserved
|
||||
original_relative_offset = original_content_start - 0 # vocals relative to instrumental start
|
||||
new_relative_offset = vocals_content_after_padding - instrumental_start_after_padding
|
||||
|
||||
assert original_relative_offset == new_relative_offset, (
|
||||
"INVARIANT VIOLATION: Padding changed the relative timing "
|
||||
"between vocals and instrumental"
|
||||
)
|
||||
|
||||
def test_sync_check_formula(self):
|
||||
"""
|
||||
Formula for checking if audio is in sync:
|
||||
If has_countdown_padding=True:
|
||||
expected_instrumental_start = countdown_padding_seconds
|
||||
actual_instrumental_start should equal expected
|
||||
"""
|
||||
countdown_padding_seconds = 3.0
|
||||
has_countdown_padding = True
|
||||
|
||||
if has_countdown_padding:
|
||||
expected_instrumental_start = countdown_padding_seconds
|
||||
else:
|
||||
expected_instrumental_start = 0
|
||||
|
||||
# If instrumental is correctly padded
|
||||
actual_instrumental_start = 3.0 # (padded)
|
||||
|
||||
assert actual_instrumental_start == expected_instrumental_start, (
|
||||
"Audio sync check failed"
|
||||
)
|
||||
|
||||
|
||||
class TestAudioSyncDriftAcceptance:
|
||||
"""
|
||||
Tests for acceptable audio drift tolerance.
|
||||
|
||||
Small amounts of drift (<100ms) are acceptable in karaoke videos.
|
||||
Larger drift (>100ms) is noticeable and indicates a bug.
|
||||
"""
|
||||
|
||||
def test_acceptable_drift_threshold(self):
|
||||
"""Define the acceptable drift threshold."""
|
||||
acceptable_drift_ms = 100 # 100ms
|
||||
acceptable_drift_s = acceptable_drift_ms / 1000
|
||||
|
||||
# A 50ms drift is acceptable
|
||||
actual_drift_s = 0.050
|
||||
assert actual_drift_s <= acceptable_drift_s, "50ms drift should be acceptable"
|
||||
|
||||
# A 200ms drift is NOT acceptable
|
||||
actual_drift_s = 0.200
|
||||
assert actual_drift_s > acceptable_drift_s, "200ms drift is too much"
|
||||
|
||||
def test_countdown_sync_within_tolerance(self):
|
||||
"""
|
||||
After countdown processing, drift should be within tolerance.
|
||||
"""
|
||||
# Ideal case: no drift
|
||||
expected_sync = 3.0 # seconds
|
||||
actual_sync = 3.0 # seconds
|
||||
|
||||
drift = abs(actual_sync - expected_sync)
|
||||
acceptable_drift = 0.1 # 100ms
|
||||
|
||||
assert drift <= acceptable_drift, (
|
||||
f"Drift of {drift*1000}ms exceeds tolerance of {acceptable_drift*1000}ms"
|
||||
)
|
||||
@@ -39,6 +39,24 @@ os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import app if emulators are running
|
||||
if emulators_running():
|
||||
from unittest.mock import Mock
|
||||
|
||||
# Mock theme service BEFORE importing app to ensure all jobs get theme_id="nomad"
|
||||
_mock_theme_service = Mock()
|
||||
_mock_theme_service.get_default_theme_id.return_value = "nomad"
|
||||
_mock_theme_service.get_theme.return_value = None
|
||||
|
||||
# Apply patch at module load time (before app imports the service)
|
||||
_theme_patches = [
|
||||
patch("backend.services.theme_service.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.audio_search.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.file_upload.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.users.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.jobs.get_theme_service", return_value=_mock_theme_service),
|
||||
]
|
||||
for p in _theme_patches:
|
||||
p.start()
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
else:
|
||||
@@ -76,7 +94,10 @@ def mock_worker_service():
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
"""Create FastAPI test client with mocked workers.
|
||||
|
||||
Theme service is mocked at module load time (see above).
|
||||
"""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@@ -0,0 +1,361 @@
|
||||
"""
|
||||
Integration tests for countdown padding across the full worker pipeline.
|
||||
|
||||
These tests use Firestore emulator to verify real state updates flow correctly
|
||||
between workers. They mock only external services (Modal, AudioShake, etc.)
|
||||
but use real Firestore for state persistence.
|
||||
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
import tempfile
|
||||
|
||||
# Import base fixtures from conftest
|
||||
from backend.tests.emulator.conftest import emulators_running
|
||||
|
||||
# Skip all tests if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Only import if emulators are running
|
||||
if emulators_running():
|
||||
from backend.models.job import Job, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
|
||||
class TestCountdownStateFlowThroughPipeline:
|
||||
"""
|
||||
Integration tests verifying countdown state flows correctly through
|
||||
the full worker pipeline: lyrics -> render_video -> video.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
"""Create a real JobManager using emulator."""
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
"""Create a real FirestoreService using emulator for direct job operations."""
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.fixture
|
||||
def storage_service(self):
|
||||
"""Create a real StorageService using emulator."""
|
||||
return StorageService()
|
||||
|
||||
@pytest.fixture
|
||||
def job_with_early_start_song(self, job_manager, firestore_service):
|
||||
"""
|
||||
Create a job where lyrics start within 3 seconds (needs countdown).
|
||||
|
||||
This simulates a song where the first word appears at 0.5 seconds,
|
||||
which triggers countdown processing.
|
||||
"""
|
||||
job_id = f"countdown-test-early-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Early Start Song",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
'first_word_start_time': 0.5, # Within 3s threshold
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
# Create job in Firestore directly
|
||||
firestore_service.create_job(job)
|
||||
|
||||
yield job
|
||||
|
||||
# Cleanup
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.fixture
|
||||
def job_with_late_start_song(self, job_manager, firestore_service):
|
||||
"""
|
||||
Create a job where lyrics start after 3 seconds (no countdown needed).
|
||||
|
||||
This simulates a song where the first word appears at 5.0 seconds,
|
||||
which does NOT trigger countdown processing.
|
||||
"""
|
||||
job_id = f"countdown-test-late-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test Artist",
|
||||
title="Late Start Song",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
'first_word_start_time': 5.0, # After 3s threshold
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
firestore_service.create_job(job)
|
||||
|
||||
yield job
|
||||
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_state_update_persists_to_firestore(self, job_with_early_start_song, job_manager):
|
||||
"""
|
||||
Verify that state updates via update_state_data() persist to Firestore.
|
||||
|
||||
This is the foundation of the countdown state flow - if updates don't
|
||||
persist, video_worker can't read the countdown state.
|
||||
"""
|
||||
job = job_with_early_start_song
|
||||
|
||||
# Simulate render_video_worker updating countdown state
|
||||
new_lyrics_metadata = {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
'first_word_start_time': 0.5,
|
||||
}
|
||||
|
||||
job_manager.update_state_data(job.job_id, 'lyrics_metadata', new_lyrics_metadata)
|
||||
|
||||
# Read job back from Firestore
|
||||
refreshed_job = job_manager.get_job(job.job_id)
|
||||
|
||||
assert refreshed_job is not None
|
||||
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_video_worker_reads_updated_state(self, job_with_early_start_song, job_manager):
|
||||
"""
|
||||
Verify that video_worker can read countdown state after render_video_worker sets it.
|
||||
|
||||
This simulates the full flow:
|
||||
1. render_video_worker sets has_countdown_padding=True
|
||||
2. video_worker reads the state and extracts countdown_padding_seconds
|
||||
"""
|
||||
job = job_with_early_start_song
|
||||
|
||||
# Step 1: render_video_worker updates state
|
||||
new_lyrics_metadata = {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
job_manager.update_state_data(job.job_id, 'lyrics_metadata', new_lyrics_metadata)
|
||||
|
||||
# Step 2: video_worker reads state (simulated)
|
||||
refreshed_job = job_manager.get_job(job.job_id)
|
||||
|
||||
# video_worker extraction logic
|
||||
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_no_countdown_state_when_not_needed(self, job_with_late_start_song, job_manager):
|
||||
"""
|
||||
Verify that when song doesn't need countdown, state remains False.
|
||||
"""
|
||||
job = job_with_late_start_song
|
||||
|
||||
# Read initial state
|
||||
current_job = job_manager.get_job(job.job_id)
|
||||
lyrics_metadata = current_job.state_data.get('lyrics_metadata', {})
|
||||
|
||||
assert lyrics_metadata.get('has_countdown_padding') is False
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 0.0
|
||||
|
||||
# video_worker extraction logic
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
|
||||
class TestCountdownStateTransitions:
|
||||
"""Tests for state transitions during countdown processing."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_state_transitions_preserve_countdown_info(self, job_manager, firestore_service):
|
||||
"""
|
||||
Countdown state should survive job state transitions.
|
||||
|
||||
When job moves from RENDERING_VIDEO to ENCODING,
|
||||
the lyrics_metadata should remain intact.
|
||||
"""
|
||||
job_id = f"transition-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job in RENDERING_VIDEO state
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# Transition to ENCODING
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.ENCODING,
|
||||
progress=75,
|
||||
message="Starting encoding"
|
||||
)
|
||||
|
||||
# Read job and verify countdown state preserved
|
||||
refreshed_job = job_manager.get_job(job_id)
|
||||
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
|
||||
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TestWorkerStateDataIsolation:
|
||||
"""Tests that different workers don't overwrite each other's state data."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lyrics_metadata_not_overwritten_by_other_updates(self, job_manager, firestore_service):
|
||||
"""
|
||||
Updating video_worker state should NOT overwrite lyrics_metadata.
|
||||
|
||||
Each worker stores its state in separate keys within state_data.
|
||||
"""
|
||||
job_id = f"isolation-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job with countdown state
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# Update video_worker state (different key)
|
||||
job_manager.update_state_data(job_id, 'video_worker', {
|
||||
'instrumental_source': 'clean',
|
||||
'encoding_started': True,
|
||||
})
|
||||
|
||||
# Verify lyrics_metadata preserved
|
||||
refreshed_job = job_manager.get_job(job_id)
|
||||
lyrics_metadata = refreshed_job.state_data.get('lyrics_metadata', {})
|
||||
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
# Verify video_worker state also present
|
||||
video_worker_state = refreshed_job.state_data.get('video_worker', {})
|
||||
assert video_worker_state.get('instrumental_source') == 'clean'
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TestCountdownStateUpdateCodeExists:
|
||||
"""
|
||||
Verify the render_video_worker code contains the state update logic.
|
||||
|
||||
This is a code inspection test rather than an execution test, since running
|
||||
the actual worker requires too much complex setup (audio files, corrections, etc.).
|
||||
The actual state update behavior is verified by the unit tests.
|
||||
"""
|
||||
|
||||
def test_render_video_worker_has_state_update_code(self):
|
||||
"""
|
||||
Verify render_video_worker.py contains the code to update lyrics_metadata
|
||||
when countdown padding is added.
|
||||
|
||||
This ensures the fix hasn't been accidentally removed.
|
||||
"""
|
||||
import inspect
|
||||
from backend.workers import render_video_worker
|
||||
|
||||
source = inspect.getsource(render_video_worker)
|
||||
|
||||
# The fix should include updating lyrics_metadata when padding_added is True
|
||||
assert "update_state_data" in source, (
|
||||
"render_video_worker must call update_state_data"
|
||||
)
|
||||
assert "has_countdown_padding" in source, (
|
||||
"render_video_worker must set has_countdown_padding"
|
||||
)
|
||||
assert "countdown_padding_seconds" in source, (
|
||||
"render_video_worker must set countdown_padding_seconds"
|
||||
)
|
||||
assert "lyrics_metadata" in source, (
|
||||
"render_video_worker must update lyrics_metadata"
|
||||
)
|
||||
@@ -0,0 +1,380 @@
|
||||
"""
|
||||
Tests that countdown state survives worker boundaries and job reloads.
|
||||
|
||||
These tests verify that countdown state persists correctly in Firestore
|
||||
and can be read across different worker invocations.
|
||||
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import os
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
import time
|
||||
|
||||
# Import base fixtures from conftest
|
||||
from backend.tests.emulator.conftest import emulators_running
|
||||
|
||||
# Skip all tests if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Only import if emulators are running
|
||||
if emulators_running():
|
||||
from backend.models.job import Job, JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
|
||||
|
||||
class TestCountdownStatePersistence:
|
||||
"""Tests that countdown state persists between worker boundaries."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
"""Create a real JobManager using emulator."""
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
"""Create a real FirestoreService using emulator for direct job operations."""
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_state_persists_between_render_and_video_workers(self, job_manager, firestore_service):
|
||||
"""
|
||||
State set by render_video_worker is readable by video_worker.
|
||||
|
||||
Simulates:
|
||||
1. render_video_worker completes, sets has_countdown_padding=True
|
||||
2. Some time passes (worker boundary)
|
||||
3. video_worker starts, reads has_countdown_padding
|
||||
"""
|
||||
job_id = f"persist-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create initial job (as lyrics_worker would leave it)
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# === render_video_worker completes ===
|
||||
job_manager.update_state_data(job_id, 'lyrics_metadata', {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
})
|
||||
|
||||
# Simulate worker boundary (job is re-fetched from Firestore)
|
||||
# Create a NEW JobManager instance to simulate video_worker starting
|
||||
video_job_manager = JobManager()
|
||||
|
||||
# === video_worker starts ===
|
||||
job_from_video_worker = video_job_manager.get_job(job_id)
|
||||
|
||||
# video_worker reads countdown state
|
||||
lyrics_metadata = job_from_video_worker.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0, (
|
||||
"Countdown state did not persist between workers"
|
||||
)
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_state_survives_job_reload(self, job_manager, firestore_service):
|
||||
"""
|
||||
State persists when job is reloaded from Firestore multiple times.
|
||||
"""
|
||||
job_id = f"reload-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job with countdown state
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# Reload job multiple times
|
||||
for i in range(3):
|
||||
reloaded_job = job_manager.get_job(job_id)
|
||||
|
||||
lyrics_metadata = reloaded_job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True, (
|
||||
f"Countdown state lost after reload #{i+1}"
|
||||
)
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0, (
|
||||
f"Countdown duration changed after reload #{i+1}"
|
||||
)
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_partial_state_update_preserves_other_fields(self, job_manager, firestore_service):
|
||||
"""
|
||||
Updating one field in state_data doesn't affect other fields.
|
||||
"""
|
||||
job_id = f"partial-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job with multiple state fields
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
'original_field': 'should_be_preserved',
|
||||
},
|
||||
'audio_complete': True,
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# Update video_worker state (different key)
|
||||
job_manager.update_state_data(job_id, 'video_worker', {
|
||||
'encoding_started': True,
|
||||
})
|
||||
|
||||
# Verify lyrics_metadata is completely preserved
|
||||
reloaded_job = job_manager.get_job(job_id)
|
||||
lyrics_metadata = reloaded_job.state_data.get('lyrics_metadata', {})
|
||||
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
assert lyrics_metadata.get('original_field') == 'should_be_preserved'
|
||||
assert reloaded_job.state_data.get('audio_complete') is True
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TestCountdownStateConsistency:
|
||||
"""Tests for state consistency under concurrent access patterns."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_concurrent_reads_see_same_state(self, job_manager, firestore_service):
|
||||
"""
|
||||
Multiple concurrent reads should see the same countdown state.
|
||||
"""
|
||||
job_id = f"concurrent-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job with countdown
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# Simulate concurrent reads
|
||||
reads = []
|
||||
for _ in range(5):
|
||||
reloaded = job_manager.get_job(job_id)
|
||||
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
|
||||
reads.append({
|
||||
'has_countdown_padding': lyrics_metadata.get('has_countdown_padding'),
|
||||
'countdown_padding_seconds': lyrics_metadata.get('countdown_padding_seconds'),
|
||||
})
|
||||
|
||||
# All reads should be consistent
|
||||
for read in reads:
|
||||
assert read['has_countdown_padding'] is True
|
||||
assert read['countdown_padding_seconds'] == 3.0
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
class TestCountdownStateMigration:
|
||||
"""Tests for handling jobs with missing or old countdown state format."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_manager(self):
|
||||
return JobManager()
|
||||
|
||||
@pytest.fixture
|
||||
def firestore_service(self):
|
||||
return FirestoreService()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handles_missing_lyrics_metadata(self, job_manager, firestore_service):
|
||||
"""
|
||||
Jobs without lyrics_metadata should be handled gracefully.
|
||||
|
||||
This can happen for jobs created before the countdown feature.
|
||||
"""
|
||||
job_id = f"migration-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job WITHOUT lyrics_metadata (old format)
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={} # No lyrics_metadata
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
# video_worker logic should handle this
|
||||
reloaded = job_manager.get_job(job_id)
|
||||
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# Should return None, not crash
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handles_missing_countdown_fields(self, job_manager, firestore_service):
|
||||
"""
|
||||
Jobs with lyrics_metadata but missing countdown fields handled gracefully.
|
||||
"""
|
||||
job_id = f"fields-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
# Create job with lyrics_metadata but no countdown fields
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'some_other_field': 'value',
|
||||
# No has_countdown_padding
|
||||
# No countdown_padding_seconds
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
reloaded = job_manager.get_job(job_id)
|
||||
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# Should return None when has_countdown_padding is missing/falsy
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_handles_countdown_true_without_duration(self, job_manager, firestore_service):
|
||||
"""
|
||||
If has_countdown_padding=True but duration missing, default to 3.0.
|
||||
"""
|
||||
job_id = f"default-duration-test-{datetime.now(UTC).timestamp()}"
|
||||
|
||||
try:
|
||||
job = Job(
|
||||
job_id=job_id,
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
# countdown_padding_seconds intentionally missing
|
||||
}
|
||||
}
|
||||
)
|
||||
firestore_service.create_job(job)
|
||||
|
||||
reloaded = job_manager.get_job(job_id)
|
||||
lyrics_metadata = reloaded.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# Should default to 3.0
|
||||
assert countdown_padding_seconds == 3.0
|
||||
|
||||
finally:
|
||||
try:
|
||||
firestore_service.delete_job(job_id)
|
||||
except Exception:
|
||||
pass
|
||||
@@ -0,0 +1,167 @@
|
||||
"""
|
||||
Emulator integration tests for made-for-you order features.
|
||||
|
||||
Tests with real Firestore emulator via API endpoints to verify:
|
||||
1. Admin login token creation and verification
|
||||
2. Made-for-you job fields are properly stored
|
||||
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
|
||||
from .conftest import emulators_running
|
||||
|
||||
# Skip all tests if emulators not running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
|
||||
class TestAdminLoginTokenIntegration:
|
||||
"""
|
||||
Integration tests for admin login token feature via API.
|
||||
|
||||
These tests verify that admin login tokens can be verified
|
||||
through the actual API endpoint.
|
||||
"""
|
||||
|
||||
def test_verify_invalid_token_returns_401(self, client):
|
||||
"""Invalid token returns 401 unauthorized."""
|
||||
response = client.get("/api/users/auth/verify?token=invalid-token-xyz")
|
||||
assert response.status_code == 401
|
||||
|
||||
def test_verify_missing_token_returns_422(self, client):
|
||||
"""Missing token parameter returns 422 validation error."""
|
||||
response = client.get("/api/users/auth/verify")
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_verify_empty_token_returns_401(self, client):
|
||||
"""Empty token returns 401."""
|
||||
response = client.get("/api/users/auth/verify?token=")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestMadeForYouJobModel:
|
||||
"""
|
||||
Tests for made-for-you job model fields.
|
||||
|
||||
Verifies that the Job model properly supports made_for_you fields.
|
||||
"""
|
||||
|
||||
def test_job_model_has_made_for_you_fields(self):
|
||||
"""Job model includes made_for_you and customer_email fields."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# Create a made-for-you job directly
|
||||
job = Job(
|
||||
job_id="test-mfy-model",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Please make it perfect!",
|
||||
)
|
||||
|
||||
assert job.made_for_you is True
|
||||
assert job.customer_email == "customer@example.com"
|
||||
assert job.customer_notes == "Please make it perfect!"
|
||||
|
||||
def test_job_model_defaults_made_for_you_false(self):
|
||||
"""Regular jobs default made_for_you to False."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime, timezone
|
||||
|
||||
job = Job(
|
||||
job_id="test-regular-model",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Default should be False or None
|
||||
assert job.made_for_you in [False, None]
|
||||
assert job.customer_email is None
|
||||
|
||||
|
||||
class TestOwnershipTransferLogic:
|
||||
"""
|
||||
Tests for ownership transfer logic (unit-style, runs in emulator context).
|
||||
|
||||
These tests verify the conditional logic for made-for-you ownership transfer
|
||||
by importing the job_manager and testing its behavior directly.
|
||||
"""
|
||||
|
||||
def test_ownership_transfer_condition_made_for_you_true(self):
|
||||
"""
|
||||
Verify the condition: if made_for_you=True and customer_email exists,
|
||||
ownership should transfer.
|
||||
"""
|
||||
# Replicate the exact logic from _schedule_completion_email
|
||||
class MockJob:
|
||||
made_for_you = True
|
||||
customer_email = "customer@example.com"
|
||||
user_email = "admin@nomadkaraoke.com"
|
||||
job_id = "test-123"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
# The logic being tested
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is True
|
||||
assert recipient_email == "customer@example.com"
|
||||
|
||||
def test_ownership_transfer_condition_made_for_you_false(self):
|
||||
"""
|
||||
Verify: if made_for_you=False, no ownership transfer.
|
||||
"""
|
||||
class MockJob:
|
||||
made_for_you = False
|
||||
customer_email = "customer@example.com"
|
||||
user_email = "user@example.com"
|
||||
job_id = "test-456"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is False
|
||||
assert recipient_email == "user@example.com"
|
||||
|
||||
def test_ownership_transfer_condition_missing_customer_email(self):
|
||||
"""
|
||||
Verify: if made_for_you=True but no customer_email, no transfer.
|
||||
"""
|
||||
class MockJob:
|
||||
made_for_you = True
|
||||
customer_email = None
|
||||
user_email = "admin@nomadkaraoke.com"
|
||||
job_id = "test-789"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is False
|
||||
assert recipient_email == "admin@nomadkaraoke.com"
|
||||
@@ -0,0 +1,332 @@
|
||||
"""
|
||||
Integration tests for YouTube download flow.
|
||||
|
||||
Tests the full flow from API endpoint to download completion, verifying that
|
||||
both entry points (audio search selection and direct URL submission) correctly
|
||||
use the YouTubeDownloadService.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_youtube_service():
|
||||
"""Create a mock YouTubeDownloadService."""
|
||||
mock = MagicMock()
|
||||
mock.download = AsyncMock(return_value="uploads/test_job/audio/test.flac")
|
||||
mock.download_by_id = AsyncMock(return_value="uploads/test_job/audio/test.flac")
|
||||
mock.is_remote_enabled = MagicMock(return_value=True)
|
||||
mock._extract_video_id = MagicMock(return_value="test_video_id")
|
||||
return mock
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_services():
|
||||
"""Mock common services used across tests."""
|
||||
mocks = {
|
||||
'job_manager': MagicMock(),
|
||||
'storage_service': MagicMock(),
|
||||
'worker_service': MagicMock(),
|
||||
'auth_service': MagicMock(),
|
||||
}
|
||||
|
||||
# Configure job_manager mock
|
||||
mock_job = MagicMock()
|
||||
mock_job.job_id = "test_job_123"
|
||||
mock_job.status = "pending"
|
||||
mock_job.state_data = {}
|
||||
mocks['job_manager'].create_job = MagicMock(return_value=mock_job)
|
||||
mocks['job_manager'].get_job = MagicMock(return_value=mock_job)
|
||||
mocks['job_manager'].transition_to_state = MagicMock()
|
||||
mocks['job_manager'].update_job = MagicMock()
|
||||
mocks['job_manager'].fail_job = MagicMock()
|
||||
|
||||
# Configure worker service mock
|
||||
mocks['worker_service'].trigger_audio_worker = AsyncMock()
|
||||
mocks['worker_service'].trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
return mocks
|
||||
|
||||
|
||||
class TestAudioSearchYouTubeFlow:
|
||||
"""Tests for YouTube downloads via audio search selection."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_search_select_youtube_uses_youtube_service(self, mock_youtube_service, mock_services):
|
||||
"""
|
||||
When selecting a YouTube result from audio search,
|
||||
should use YouTubeDownloadService instead of direct download.
|
||||
"""
|
||||
# Configure mock job with search results
|
||||
mock_job = mock_services['job_manager'].get_job.return_value
|
||||
mock_job.status = "awaiting_audio_selection"
|
||||
mock_job.state_data = {
|
||||
'audio_search_results': [{
|
||||
'index': 0,
|
||||
'title': 'Test Song',
|
||||
'artist': 'Test Artist',
|
||||
'provider': 'YouTube',
|
||||
'source_id': 'dQw4w9WgXcQ',
|
||||
'url': 'https://www.youtube.com/watch?v=dQw4w9WgXcQ',
|
||||
}],
|
||||
'audio_search_count': 1,
|
||||
}
|
||||
|
||||
# Import the route function
|
||||
from backend.api.routes.audio_search import _download_and_start_processing
|
||||
|
||||
# Patch dependencies
|
||||
with patch(
|
||||
'backend.api.routes.audio_search.get_youtube_download_service',
|
||||
return_value=mock_youtube_service
|
||||
), patch(
|
||||
'backend.api.routes.audio_search.job_manager',
|
||||
mock_services['job_manager']
|
||||
), patch(
|
||||
'backend.api.routes.audio_search.storage_service',
|
||||
mock_services['storage_service']
|
||||
):
|
||||
# Create a mock audio_search_service
|
||||
mock_audio_search_service = MagicMock()
|
||||
mock_audio_search_service.is_remote_enabled = MagicMock(return_value=True)
|
||||
|
||||
# Create a mock background_tasks
|
||||
mock_background_tasks = MagicMock()
|
||||
|
||||
# Call the download function
|
||||
result = await _download_and_start_processing(
|
||||
job_id="test_job_123",
|
||||
selection_index=0,
|
||||
audio_search_service=mock_audio_search_service,
|
||||
background_tasks=mock_background_tasks,
|
||||
)
|
||||
|
||||
# Verify YouTubeDownloadService was used
|
||||
mock_youtube_service.download_by_id.assert_called_once()
|
||||
call_args = mock_youtube_service.download_by_id.call_args
|
||||
|
||||
assert call_args.kwargs['video_id'] == 'dQw4w9WgXcQ'
|
||||
assert call_args.kwargs['job_id'] == 'test_job_123'
|
||||
assert call_args.kwargs['artist'] == 'Test Artist'
|
||||
assert call_args.kwargs['title'] == 'Test Song'
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_torrent_download_does_not_use_youtube_service(self, mock_youtube_service, mock_services):
|
||||
"""
|
||||
When selecting a torrent result (RED/OPS), should NOT use YouTubeDownloadService.
|
||||
"""
|
||||
mock_job = mock_services['job_manager'].get_job.return_value
|
||||
mock_job.status = "awaiting_audio_selection"
|
||||
mock_job.state_data = {
|
||||
'audio_search_results': [{
|
||||
'index': 0,
|
||||
'title': 'Test Song',
|
||||
'artist': 'Test Artist',
|
||||
'provider': 'RED',
|
||||
'source_id': '12345',
|
||||
'target_file': 'test.flac',
|
||||
}],
|
||||
'audio_search_count': 1,
|
||||
}
|
||||
|
||||
from backend.api.routes.audio_search import _download_and_start_processing
|
||||
|
||||
# Mock audio_search_service for torrent download
|
||||
mock_audio_search_service = MagicMock()
|
||||
mock_audio_search_service.is_remote_enabled = MagicMock(return_value=True)
|
||||
mock_result = MagicMock()
|
||||
mock_result.filepath = "gs://bucket/uploads/test/audio/test.flac"
|
||||
mock_audio_search_service.download_by_id = MagicMock(return_value=mock_result)
|
||||
|
||||
with patch(
|
||||
'backend.api.routes.audio_search.get_youtube_download_service',
|
||||
return_value=mock_youtube_service
|
||||
), patch(
|
||||
'backend.api.routes.audio_search.job_manager',
|
||||
mock_services['job_manager']
|
||||
), patch(
|
||||
'backend.api.routes.audio_search.storage_service',
|
||||
mock_services['storage_service']
|
||||
):
|
||||
mock_background_tasks = MagicMock()
|
||||
|
||||
result = await _download_and_start_processing(
|
||||
job_id="test_job_123",
|
||||
selection_index=0,
|
||||
audio_search_service=mock_audio_search_service,
|
||||
background_tasks=mock_background_tasks,
|
||||
)
|
||||
|
||||
# YouTubeDownloadService should NOT be called for torrent
|
||||
mock_youtube_service.download_by_id.assert_not_called()
|
||||
|
||||
# audio_search_service.download_by_id should be called instead
|
||||
mock_audio_search_service.download_by_id.assert_called_once()
|
||||
|
||||
|
||||
class TestDirectUrlSubmissionFlow:
|
||||
"""Tests for direct YouTube URL submission via /api/jobs/url endpoint."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_job_from_youtube_url_downloads_immediately(
|
||||
self, mock_youtube_service, mock_services
|
||||
):
|
||||
"""
|
||||
When creating a job from a YouTube URL, should download audio
|
||||
immediately (before triggering workers) using YouTubeDownloadService.
|
||||
"""
|
||||
from backend.api.routes.file_upload import _is_youtube_url
|
||||
|
||||
# Verify helper function works
|
||||
assert _is_youtube_url("https://www.youtube.com/watch?v=test") is True
|
||||
assert _is_youtube_url("https://youtu.be/test") is True
|
||||
assert _is_youtube_url("https://vimeo.com/test") is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_non_youtube_url_triggers_worker_download(self, mock_services):
|
||||
"""
|
||||
When creating a job from a non-YouTube URL (e.g., Vimeo),
|
||||
should trigger audio worker to handle the download.
|
||||
"""
|
||||
from backend.api.routes.file_upload import _is_youtube_url
|
||||
|
||||
# Non-YouTube URLs should not match
|
||||
assert _is_youtube_url("https://vimeo.com/123456") is False
|
||||
assert _is_youtube_url("https://soundcloud.com/artist/track") is False
|
||||
|
||||
|
||||
class TestYouTubeDownloadErrorHandling:
|
||||
"""Tests for error handling in YouTube download flow."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_youtube_download_failure_fails_job(self, mock_services):
|
||||
"""
|
||||
When YouTube download fails, should properly fail the job
|
||||
with a clear error message.
|
||||
"""
|
||||
from backend.services.youtube_download_service import (
|
||||
YouTubeDownloadService,
|
||||
YouTubeDownloadError,
|
||||
)
|
||||
|
||||
# Create a service that will fail
|
||||
mock_client = MagicMock()
|
||||
mock_client.download_by_id = AsyncMock(
|
||||
side_effect=Exception("Bot detection triggered")
|
||||
)
|
||||
|
||||
with patch(
|
||||
'backend.services.youtube_download_service.get_flacfetch_client',
|
||||
return_value=mock_client
|
||||
):
|
||||
from backend.services.youtube_download_service import reset_youtube_download_service
|
||||
reset_youtube_download_service()
|
||||
|
||||
service = YouTubeDownloadService()
|
||||
|
||||
with pytest.raises(YouTubeDownloadError) as exc:
|
||||
await service.download(
|
||||
url="https://youtu.be/abcDEF12345",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
assert "Remote download failed" in str(exc.value)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_invalid_youtube_url_raises_clear_error(self):
|
||||
"""
|
||||
When given an invalid YouTube URL (no video ID extractable),
|
||||
should raise a clear error.
|
||||
"""
|
||||
from backend.services.youtube_download_service import (
|
||||
YouTubeDownloadService,
|
||||
YouTubeDownloadError,
|
||||
reset_youtube_download_service,
|
||||
)
|
||||
|
||||
with patch(
|
||||
'backend.services.youtube_download_service.get_flacfetch_client',
|
||||
return_value=None # No remote client
|
||||
):
|
||||
reset_youtube_download_service()
|
||||
service = YouTubeDownloadService()
|
||||
|
||||
with pytest.raises(YouTubeDownloadError) as exc:
|
||||
await service.download(
|
||||
url="not-a-valid-youtube-url",
|
||||
job_id="job123",
|
||||
)
|
||||
|
||||
assert "Could not extract video ID" in str(exc.value)
|
||||
|
||||
|
||||
class TestRemoteVsLocalDownload:
|
||||
"""Tests verifying correct routing between remote and local download."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_remote_download_when_configured(self):
|
||||
"""
|
||||
When FLACFETCH_API_URL is configured, should use remote download.
|
||||
"""
|
||||
mock_client = MagicMock()
|
||||
mock_client.download_by_id = AsyncMock(return_value="download_123")
|
||||
mock_client.wait_for_download = AsyncMock(return_value={
|
||||
"status": "complete",
|
||||
"gcs_path": "gs://bucket/uploads/job_test/audio/test.flac",
|
||||
})
|
||||
|
||||
with patch(
|
||||
'backend.services.youtube_download_service.get_flacfetch_client',
|
||||
return_value=mock_client
|
||||
):
|
||||
from backend.services.youtube_download_service import (
|
||||
YouTubeDownloadService,
|
||||
reset_youtube_download_service,
|
||||
)
|
||||
reset_youtube_download_service()
|
||||
|
||||
service = YouTubeDownloadService()
|
||||
assert service.is_remote_enabled() is True
|
||||
|
||||
result = await service.download(
|
||||
url="https://youtu.be/abcDEF12345",
|
||||
job_id="job_test",
|
||||
)
|
||||
|
||||
# Should have called remote client
|
||||
mock_client.download_by_id.assert_called_once()
|
||||
assert "uploads/job_test/audio/" in result
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_local_download_when_remote_not_configured(self):
|
||||
"""
|
||||
When FLACFETCH_API_URL is not configured, should fall back to local.
|
||||
"""
|
||||
with patch(
|
||||
'backend.services.youtube_download_service.get_flacfetch_client',
|
||||
return_value=None # No remote client
|
||||
):
|
||||
from backend.services.youtube_download_service import (
|
||||
YouTubeDownloadService,
|
||||
reset_youtube_download_service,
|
||||
)
|
||||
reset_youtube_download_service()
|
||||
|
||||
service = YouTubeDownloadService()
|
||||
assert service.is_remote_enabled() is False
|
||||
|
||||
# Mock the local download to avoid actually calling yt_dlp
|
||||
with patch.object(
|
||||
service, '_download_local',
|
||||
new_callable=AsyncMock,
|
||||
return_value="uploads/job/audio/test.wav"
|
||||
) as mock_local:
|
||||
result = await service.download(
|
||||
url="https://youtu.be/abcDEF12345",
|
||||
job_id="job_test",
|
||||
)
|
||||
|
||||
mock_local.assert_called_once()
|
||||
assert result == "uploads/job/audio/test.wav"
|
||||
@@ -0,0 +1,352 @@
|
||||
"""
|
||||
Unit tests for admin delete job outputs endpoint.
|
||||
|
||||
Tests the POST /api/admin/jobs/{job_id}/delete-outputs endpoint that allows admins
|
||||
to delete distributed outputs (YouTube, Dropbox, Google Drive) while preserving
|
||||
the job record.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
from datetime import datetime
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_complete_job():
|
||||
"""Create a mock job in COMPLETE status with distribution data."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = "complete"
|
||||
job.dropbox_path = "/Karaoke/Organized"
|
||||
job.outputs_deleted_at = None
|
||||
job.outputs_deleted_by = None
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtu.be/abc123",
|
||||
"brand_code": "NOMAD-1234",
|
||||
"dropbox_link": "https://dropbox.com/...",
|
||||
"gdrive_files": {"mp4": "file_id_1", "mp4_720p": "file_id_2"},
|
||||
}
|
||||
job.timeline = []
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_no_outputs():
|
||||
"""Create a mock job in COMPLETE status without distribution data."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-456"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = "complete"
|
||||
job.dropbox_path = None
|
||||
job.outputs_deleted_at = None
|
||||
job.outputs_deleted_by = None
|
||||
job.state_data = {}
|
||||
job.timeline = []
|
||||
return job
|
||||
|
||||
|
||||
class TestDeleteOutputsSuccess:
|
||||
"""Tests for successful output deletion."""
|
||||
|
||||
def test_delete_outputs_success(self, client, mock_complete_job):
|
||||
"""Test successfully deleting outputs from a complete job."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert "outputs_deleted_at" in data
|
||||
|
||||
def test_delete_outputs_clears_state_data_keys(self, client, mock_complete_job):
|
||||
"""Test that output-related state_data keys are listed as cleared."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Should list cleared keys
|
||||
assert "youtube_url" in data["cleared_state_data"]
|
||||
assert "brand_code" in data["cleared_state_data"]
|
||||
assert "dropbox_link" in data["cleared_state_data"]
|
||||
assert "gdrive_files" in data["cleared_state_data"]
|
||||
|
||||
def test_delete_outputs_job_without_distribution(self, client, mock_job_no_outputs):
|
||||
"""Test deleting outputs from job that has no distribution data."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_no_outputs
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-456/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
# All services should be skipped
|
||||
assert data["deleted_services"]["youtube"]["status"] == "skipped"
|
||||
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
|
||||
assert data["deleted_services"]["gdrive"]["status"] == "skipped"
|
||||
# No keys to clear
|
||||
assert data["cleared_state_data"] == []
|
||||
|
||||
|
||||
class TestDeleteOutputsValidation:
|
||||
"""Tests for validation on delete outputs endpoint."""
|
||||
|
||||
def test_rejects_non_terminal_status(self, client, mock_complete_job):
|
||||
"""Test that jobs not in terminal states are rejected."""
|
||||
mock_complete_job.status = "encoding"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "terminal" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_awaiting_review_status(self, client, mock_complete_job):
|
||||
"""Test that AWAITING_REVIEW is rejected (not terminal)."""
|
||||
mock_complete_job.status = "awaiting_review"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_accepts_prep_complete_status(self, client, mock_complete_job):
|
||||
"""Test that prep_complete is accepted as terminal."""
|
||||
mock_complete_job.status = "prep_complete"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_failed_status(self, client, mock_complete_job):
|
||||
"""Test that failed is accepted as terminal."""
|
||||
mock_complete_job.status = "failed"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_rejects_already_deleted(self, client, mock_complete_job):
|
||||
"""Test that already-deleted outputs cannot be deleted again."""
|
||||
mock_complete_job.outputs_deleted_at = datetime(2026, 1, 9, 12, 0, 0)
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "already deleted" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_404_for_missing_job(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/nonexistent/delete-outputs")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
class TestDeleteOutputsServices:
|
||||
"""Tests for service deletion behavior."""
|
||||
|
||||
def test_handles_missing_youtube(self, client, mock_complete_job):
|
||||
"""Test handling when job has no YouTube URL."""
|
||||
mock_complete_job.state_data = {"brand_code": "NOMAD-1234"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["deleted_services"]["youtube"]["status"] == "skipped"
|
||||
|
||||
def test_handles_missing_dropbox_path(self, client, mock_complete_job):
|
||||
"""Test handling when job has no dropbox_path."""
|
||||
mock_complete_job.dropbox_path = None
|
||||
mock_complete_job.state_data = {"youtube_url": "https://youtu.be/abc123"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
|
||||
|
||||
|
||||
class TestDeleteOutputsLogging:
|
||||
"""Tests for logging on delete outputs endpoint."""
|
||||
|
||||
def test_logs_admin_action(self, client, mock_complete_job):
|
||||
"""Test that admin delete action is logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin" in log_message.lower() or "deleted" in log_message.lower()
|
||||
|
||||
|
||||
class TestDeleteOutputsAuthorization:
|
||||
"""Tests for authorization on the delete outputs endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_complete_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/delete-outputs",
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
Unit tests for admin email endpoints.
|
||||
|
||||
Tests the completion message and send email API endpoints.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
# The router already has prefix="/admin", so we add /api prefix
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
return ("admin@example.com", "admin", 1)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.audio_hash = "hash123"
|
||||
job.review_token = "review123"
|
||||
job.instrumental_token = "inst123"
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=test123",
|
||||
"dropbox_link": "https://dropbox.com/folder/test",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
|
||||
|
||||
def test_returns_completion_message(self, client, mock_job):
|
||||
"""Test successful completion message retrieval."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
# Setup mocks
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["message"] == "Your video is ready!"
|
||||
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
|
||||
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["youtube_url"] is None
|
||||
assert data["dropbox_url"] is None
|
||||
|
||||
def test_default_subject_without_song_info(self, client, mock_job):
|
||||
"""Test default subject when no artist/title."""
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["subject"] == "Your karaoke video is ready!"
|
||||
|
||||
|
||||
class TestSendCompletionEmail:
|
||||
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
|
||||
|
||||
def test_sends_email_successfully(self, client, mock_job):
|
||||
"""Test successful email sending."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": True},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["success"] is True
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["to_email"] == "customer@example.com"
|
||||
|
||||
# Verify email was sent with correct params
|
||||
mock_es.send_job_completion.assert_called_once_with(
|
||||
to_email="customer@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_code="NOMAD-1234",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
def test_sends_email_without_cc(self, client, mock_job):
|
||||
"""Test email sending without CC."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": False},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify CC was not included
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["cc_admin"] is False
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_returns_500_when_email_fails(self, client, mock_job):
|
||||
"""Test 500 when email sending fails."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = False # Email failed
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "Failed to send email" in response.json()["detail"]
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_any_email_string(self, client, mock_job):
|
||||
"""Test that any string is currently accepted as email (no validation).
|
||||
|
||||
Documents current behavior: FastAPI/Pydantic doesn't validate email
|
||||
format by default. If validation is added later, this test should
|
||||
be updated to expect 422 for invalid emails.
|
||||
"""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "invalid-email"}, # Not a valid email format
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Currently accepts any string - endpoint succeeds
|
||||
assert response.status_code == 200
|
||||
# Verify the invalid email was passed through
|
||||
mock_es.send_job_completion.assert_called_once()
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["to_email"] == "invalid-email"
|
||||
|
||||
|
||||
class TestIdleReminderEndpoint:
|
||||
"""Tests for the internal idle reminder endpoint."""
|
||||
|
||||
def test_sends_reminder_when_still_idle(self, mock_job):
|
||||
"""Test that reminder is sent when user is still idle."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": False,
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.firestore = Mock() # Mock firestore for update_job call
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed and return "sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
|
||||
def test_skips_reminder_when_already_sent(self, mock_job):
|
||||
"""Test that reminder is skipped if already sent."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": True, # Already sent
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed with "already_sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
@@ -0,0 +1,337 @@
|
||||
"""
|
||||
Unit tests for admin job files endpoint.
|
||||
|
||||
Tests the GET /api/admin/jobs/{job_id}/files endpoint that returns
|
||||
all files for a job with signed download URLs.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_files():
|
||||
"""Create a mock job with comprehensive file_urls."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/jobs/test-job-123/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test-job-123/stems/instrumental_clean.flac",
|
||||
"vocals": "gs://bucket/jobs/test-job-123/stems/vocals.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/test-job-123/lyrics/corrections.json",
|
||||
"lrc": "gs://bucket/jobs/test-job-123/lyrics/output.lrc",
|
||||
},
|
||||
"finals": {
|
||||
"lossy_720p_mp4": "gs://bucket/jobs/test-job-123/finals/video_720p.mp4",
|
||||
},
|
||||
"youtube": {
|
||||
"url": "https://youtube.com/watch?v=xyz789",
|
||||
"video_id": "xyz789",
|
||||
},
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_no_files():
|
||||
"""Create a mock job with empty file_urls."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-empty"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Empty Artist"
|
||||
job.title = "Empty Song"
|
||||
job.status = JobStatus.PENDING
|
||||
job.file_urls = {}
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_partial_files():
|
||||
"""Create a mock job with only some files (partial processing)."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-partial"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Partial Artist"
|
||||
job.title = "Partial Song"
|
||||
job.status = JobStatus.SEPARATING_STAGE1
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/jobs/test-job-partial/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test-job-partial/stems/instrumental_clean.flac",
|
||||
},
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetJobFiles:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/files endpoint."""
|
||||
|
||||
def test_returns_files_with_signed_urls(self, client, mock_job_with_files):
|
||||
"""Test that endpoint returns all files with signed URLs."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
# Setup JobManager mock
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Setup StorageService mock to return predictable signed URLs
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/files",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["artist"] == "Test Artist"
|
||||
assert data["title"] == "Test Song"
|
||||
assert "files" in data
|
||||
assert data["total_files"] > 0
|
||||
|
||||
def test_returns_correct_file_count(self, client, mock_job_with_files):
|
||||
"""Test that file count matches actual GCS files (excluding non-GCS entries)."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
# Should have: input, stems.instrumental_clean, stems.vocals,
|
||||
# lyrics.corrections, lyrics.lrc, finals.lossy_720p_mp4
|
||||
# NOT youtube.url or youtube.video_id (not GCS paths)
|
||||
assert data["total_files"] == 6
|
||||
|
||||
def test_handles_nested_file_structure(self, client, mock_job_with_files):
|
||||
"""Test that nested file_urls structure is properly traversed."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Check that files from different categories are present
|
||||
categories = {f["category"] for f in files}
|
||||
assert "input" in categories or any(f["category"] == "" and f["file_key"] == "input" for f in files)
|
||||
assert "stems" in categories
|
||||
assert "lyrics" in categories
|
||||
assert "finals" in categories
|
||||
|
||||
def test_file_info_structure(self, client, mock_job_with_files):
|
||||
"""Test that each file has correct info structure."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Every file should have these fields
|
||||
for file_info in files:
|
||||
assert "name" in file_info
|
||||
assert "path" in file_info
|
||||
assert "download_url" in file_info
|
||||
assert "category" in file_info
|
||||
assert "file_key" in file_info
|
||||
# download_url should be a signed URL
|
||||
assert file_info["download_url"].startswith("https://signed.url/")
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/files",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_empty_files_for_new_job(self, client, mock_job_no_files):
|
||||
"""Test that new jobs with no files return empty list."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_no_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-empty/files")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-empty"
|
||||
assert data["files"] == []
|
||||
assert data["total_files"] == 0
|
||||
|
||||
def test_handles_partial_processing(self, client, mock_job_partial_files):
|
||||
"""Test job with only some files processed."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_partial_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-partial/files")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Should have input + stems.instrumental_clean = 2 files
|
||||
assert data["total_files"] == 2
|
||||
|
||||
def test_skips_non_gcs_urls(self, client, mock_job_with_files):
|
||||
"""Test that non-GCS URLs (like youtube URLs) are skipped."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Ensure no youtube URLs or video IDs are in the files
|
||||
for file_info in files:
|
||||
assert "youtube.com" not in file_info["path"]
|
||||
assert file_info["path"].startswith("gs://")
|
||||
|
||||
def test_signed_url_expiration(self, client, mock_job_with_files):
|
||||
"""Test that signed URLs are generated with appropriate expiration."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.return_value = "https://signed.url/test"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
# Verify signed URLs were requested with expiration (default 120 minutes)
|
||||
assert mock_storage.generate_signed_url.called
|
||||
# Check that expiration_minutes was passed (could be any reasonable value)
|
||||
call_kwargs = mock_storage.generate_signed_url.call_args_list[0][1]
|
||||
if "expiration_minutes" in call_kwargs:
|
||||
assert call_kwargs["expiration_minutes"] >= 60
|
||||
|
||||
|
||||
class TestGetJobFilesAuthorization:
|
||||
"""Tests for authorization on the files endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job_with_files):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
# Reset the dependency override to test auth
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/files",
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
# Restore the original override
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -0,0 +1,639 @@
|
||||
"""
|
||||
Unit tests for admin job reset endpoint.
|
||||
|
||||
Tests the POST /api/admin/jobs/{job_id}/reset endpoint that allows admins
|
||||
to reset a job to a specific state for re-processing.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
from datetime import datetime
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job in COMPLETE status for testing."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.theme_id = "nomad"
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/stems/clean.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/lyrics/corrections.json",
|
||||
},
|
||||
}
|
||||
job.state_data = {
|
||||
"audio_search_results": [{"provider": "test"}],
|
||||
"audio_selection": {"index": 0},
|
||||
"review_complete": True,
|
||||
"corrected_lyrics": {"lines": []},
|
||||
"instrumental_selection": "clean",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
job.timeline = []
|
||||
job.created_at = datetime(2026, 1, 9, 10, 0, 0)
|
||||
job.updated_at = datetime(2026, 1, 9, 12, 0, 0)
|
||||
return job
|
||||
|
||||
|
||||
class TestResetJobToPending:
|
||||
"""Tests for resetting job to PENDING state."""
|
||||
|
||||
def test_reset_to_pending_success(self, client, mock_job):
|
||||
"""Test resetting a job to PENDING state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["new_status"] == "pending"
|
||||
assert "pending" in data["message"].lower()
|
||||
|
||||
def test_reset_to_pending_clears_state_data(self, client, mock_job):
|
||||
"""Test that resetting to PENDING clears appropriate state_data keys."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify update_job was called with cleared state_data keys
|
||||
call_args = mock_jm.update_job.call_args
|
||||
update_data = call_args[0][1]
|
||||
# Verify status was set
|
||||
assert update_data["status"] == "pending"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingAudioSelection:
|
||||
"""Tests for resetting job to AWAITING_AUDIO_SELECTION state."""
|
||||
|
||||
def test_reset_to_awaiting_audio_selection(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_AUDIO_SELECTION state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_audio_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_audio_selection"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingReview:
|
||||
"""Tests for resetting job to AWAITING_REVIEW state."""
|
||||
|
||||
def test_reset_to_awaiting_review(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_REVIEW state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_review"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingInstrumentalSelection:
|
||||
"""Tests for resetting job to AWAITING_INSTRUMENTAL_SELECTION state."""
|
||||
|
||||
def test_reset_to_awaiting_instrumental_selection(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_INSTRUMENTAL_SELECTION state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_instrumental_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_instrumental_selection"
|
||||
|
||||
|
||||
class TestResetJobValidation:
|
||||
"""Tests for reset endpoint validation."""
|
||||
|
||||
def test_rejects_invalid_target_state(self, client, mock_job):
|
||||
"""Test that invalid target states are rejected."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "encoding"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "invalid" in response.json()["detail"].lower() or "not allowed" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_complete_as_target(self, client, mock_job):
|
||||
"""Test that COMPLETE is not a valid reset target."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "complete"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_rejects_failed_as_target(self, client, mock_job):
|
||||
"""Test that FAILED is not a valid reset target."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "failed"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_missing_target_state(self, client, mock_job):
|
||||
"""Test that target_state is required."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={},
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestResetJobLogging:
|
||||
"""Tests for logging on the reset endpoint."""
|
||||
|
||||
def test_logs_admin_reset_action(self, client, mock_job):
|
||||
"""Test that admin reset actions are logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify logging was called
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin" in log_message.lower() or "reset" in log_message.lower()
|
||||
|
||||
|
||||
class TestResetJobTimeline:
|
||||
"""Tests for timeline event on reset."""
|
||||
|
||||
def test_adds_timeline_event_on_reset(self, client, mock_job):
|
||||
"""Test that a timeline event is added when job is reset."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Timeline event should be included in response
|
||||
assert "timeline_event" in data or data.get("message")
|
||||
|
||||
|
||||
class TestResetJobAuthorization:
|
||||
"""Tests for authorization on the reset endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
class TestResetJobClearsStateData:
|
||||
"""Tests for state_data clearing based on target state."""
|
||||
|
||||
def test_reset_to_pending_clears_all_processing_data(self, client, mock_job):
|
||||
"""Test that resetting to PENDING clears audio search results."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# The cleared_data field should list what was cleared
|
||||
data = response.json()
|
||||
assert "cleared_data" in data or "state_data" in str(data)
|
||||
|
||||
def test_reset_to_awaiting_review_preserves_audio_data(self, client, mock_job):
|
||||
"""Test that resetting to AWAITING_REVIEW preserves audio stems."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_reset_to_instrumental_preserves_review_data(self, client, mock_job):
|
||||
"""Test that resetting to AWAITING_INSTRUMENTAL_SELECTION preserves review data."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_instrumental_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
class TestResetJobToInstrumentalSelected:
|
||||
"""Tests for resetting job to INSTRUMENTAL_SELECTED state (reprocess video)."""
|
||||
|
||||
def test_reset_to_instrumental_selected_success(self, client, mock_job):
|
||||
"""Test resetting a job to INSTRUMENTAL_SELECTED state."""
|
||||
import asyncio
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.worker_service.get_worker_service') as mock_ws:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock worker service with async method
|
||||
mock_worker_service = Mock()
|
||||
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
|
||||
mock_ws.return_value = mock_worker_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "instrumental_selected"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "instrumental_selected"
|
||||
|
||||
def test_reset_to_instrumental_selected_triggers_video_worker(self, client, mock_job):
|
||||
"""Test that resetting to INSTRUMENTAL_SELECTED triggers video worker."""
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.worker_service.get_worker_service') as mock_ws:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock worker service
|
||||
mock_worker_service = Mock()
|
||||
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
|
||||
mock_ws.return_value = mock_worker_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "instrumental_selected"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify worker was triggered
|
||||
mock_worker_service.trigger_video_worker.assert_called_once_with("test-job-123")
|
||||
# Verify message indicates worker was triggered
|
||||
data = response.json()
|
||||
assert "video worker triggered" in data["message"]
|
||||
|
||||
def test_reset_to_instrumental_selected_message_on_worker_failure(self, client, mock_job):
|
||||
"""Test message when video worker trigger fails."""
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.worker_service.get_worker_service') as mock_ws:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock worker service failure
|
||||
mock_worker_service = Mock()
|
||||
mock_worker_service.trigger_video_worker = AsyncMock(return_value=False)
|
||||
mock_ws.return_value = mock_worker_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "instrumental_selected"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Reset still succeeds even if worker trigger fails
|
||||
data = response.json()
|
||||
assert data["new_status"] == "instrumental_selected"
|
||||
# Message should NOT say worker was triggered
|
||||
assert "video worker triggered" not in data["message"]
|
||||
|
||||
def test_reset_to_instrumental_selected_clears_video_state(self, client, mock_job):
|
||||
"""Test that INSTRUMENTAL_SELECTED reset clears video/encoding/distribution state."""
|
||||
from unittest.mock import AsyncMock
|
||||
|
||||
mock_job.state_data = {
|
||||
"instrumental_selection": "clean",
|
||||
"lyrics_metadata": {"has_countdown_padding": True},
|
||||
"video_progress": {"stage": "running"},
|
||||
"brand_code": "NOMAD-1234",
|
||||
"youtube_url": "https://youtube.com/watch?v=xxx",
|
||||
"dropbox_link": "https://dropbox.com/xxx",
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.worker_service.get_worker_service') as mock_ws:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_worker_service = Mock()
|
||||
mock_worker_service.trigger_video_worker = AsyncMock(return_value=True)
|
||||
mock_ws.return_value = mock_worker_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "instrumental_selected"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# These keys should be marked as cleared
|
||||
cleared = data.get("cleared_data", [])
|
||||
# At minimum, video_progress and brand_code should be cleared
|
||||
# (they exist in the mock job's state_data)
|
||||
assert any(key in ["video_progress", "brand_code", "youtube_url", "dropbox_link"]
|
||||
for key in cleared)
|
||||
|
||||
|
||||
class TestResetJobClearsErrorState:
|
||||
"""Tests that reset clears error_message and error_details."""
|
||||
|
||||
def test_reset_clears_error_message(self, client, mock_job):
|
||||
"""Test that resetting a job clears error_message field."""
|
||||
# Set job to failed state with error message
|
||||
mock_job.status = JobStatus.FAILED
|
||||
mock_job.error_message = "Previous error message"
|
||||
mock_job.error_details = {"code": "TEST_ERROR"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# The endpoint should issue an update that includes DELETE_FIELD for error fields
|
||||
# We verify this indirectly through the success of the operation
|
||||
|
||||
|
||||
class TestClearWorkersEndpoint:
|
||||
"""Tests for the POST /api/admin/jobs/{job_id}/clear-workers endpoint."""
|
||||
|
||||
def test_clear_workers_success(self, client, mock_job):
|
||||
"""Test clearing worker progress keys."""
|
||||
mock_job.state_data = {
|
||||
"render_progress": {"stage": "complete"},
|
||||
"video_progress": {"stage": "complete"},
|
||||
"encoding_progress": {"stage": "complete"},
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert "cleared_keys" in data
|
||||
# Should have cleared the progress keys that exist
|
||||
assert any(k in data["cleared_keys"] for k in
|
||||
["render_progress", "video_progress", "encoding_progress"])
|
||||
|
||||
def test_clear_workers_no_progress_keys(self, client, mock_job):
|
||||
"""Test clearing workers when no progress keys exist."""
|
||||
mock_job.state_data = {"other_data": "value"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert len(data["cleared_keys"]) == 0
|
||||
|
||||
def test_clear_workers_job_not_found(self, client):
|
||||
"""Test clearing workers for non-existent job."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/nonexistent/clear-workers")
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_clear_workers_clears_all_worker_types(self, client, mock_job):
|
||||
"""Test that all defined worker progress keys can be cleared."""
|
||||
# Set all possible worker progress keys
|
||||
mock_job.state_data = {
|
||||
"audio_progress": {"stage": "complete"},
|
||||
"lyrics_progress": {"stage": "complete"},
|
||||
"render_progress": {"stage": "complete"},
|
||||
"screens_progress": {"stage": "complete"},
|
||||
"video_progress": {"stage": "complete"},
|
||||
"encoding_progress": {"stage": "complete"},
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/clear-workers")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Should have cleared all 6 progress keys
|
||||
assert len(data["cleared_keys"]) == 6
|
||||
|
||||
|
||||
class TestResetIncludesEncodingProgress:
|
||||
"""Tests that reset includes encoding_progress in cleared keys."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_encoding_progress(self, client, mock_job):
|
||||
"""Test that resetting to awaiting_review clears encoding_progress."""
|
||||
mock_job.state_data = {
|
||||
"encoding_progress": {"stage": "complete"},
|
||||
"video_progress": {"stage": "complete"},
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# encoding_progress should be in cleared_data
|
||||
assert "encoding_progress" in data.get("cleared_data", [])
|
||||
@@ -0,0 +1,326 @@
|
||||
"""
|
||||
Unit tests for admin job update endpoint.
|
||||
|
||||
Tests the PATCH /api/admin/jobs/{job_id} endpoint that allows admins
|
||||
to update editable job fields.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job for testing."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Original Artist"
|
||||
job.title = "Original Title"
|
||||
job.status = JobStatus.AWAITING_REVIEW
|
||||
job.theme_id = "nomad"
|
||||
job.enable_cdg = False
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = True
|
||||
job.customer_email = None
|
||||
job.customer_notes = None
|
||||
job.brand_prefix = None
|
||||
job.non_interactive = False
|
||||
job.prep_only = False
|
||||
job.discord_webhook_url = None
|
||||
job.youtube_description = None
|
||||
job.youtube_description_template = None
|
||||
job.file_urls = {}
|
||||
job.state_data = {}
|
||||
job.created_at = "2026-01-09T10:00:00Z"
|
||||
job.updated_at = "2026-01-09T10:30:00Z"
|
||||
return job
|
||||
|
||||
|
||||
class TestUpdateJob:
|
||||
"""Tests for PATCH /api/admin/jobs/{job_id} endpoint."""
|
||||
|
||||
def test_update_artist_field(self, client, mock_job):
|
||||
"""Test updating the artist field."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert "artist" in data["updated_fields"]
|
||||
|
||||
# Verify update_job was called with correct args
|
||||
mock_jm.update_job.assert_called_once()
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][0] == "test-job-123"
|
||||
assert call_args[0][1]["artist"] == "New Artist"
|
||||
|
||||
def test_update_multiple_fields(self, client, mock_job):
|
||||
"""Test updating multiple fields at once."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"artist": "New Artist",
|
||||
"title": "New Title",
|
||||
"theme_id": "default",
|
||||
"enable_cdg": True,
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert len(data["updated_fields"]) == 4
|
||||
assert "artist" in data["updated_fields"]
|
||||
assert "title" in data["updated_fields"]
|
||||
assert "theme_id" in data["updated_fields"]
|
||||
assert "enable_cdg" in data["updated_fields"]
|
||||
|
||||
def test_update_user_email(self, client, mock_job):
|
||||
"""Test updating user_email field."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"user_email": "newuser@example.com"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["user_email"] == "newuser@example.com"
|
||||
|
||||
def test_update_boolean_fields(self, client, mock_job):
|
||||
"""Test updating boolean fields."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"enable_cdg": True,
|
||||
"enable_txt": True,
|
||||
"enable_youtube_upload": False,
|
||||
"non_interactive": True,
|
||||
"prep_only": True,
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["enable_cdg"] is True
|
||||
assert call_args[0][1]["enable_txt"] is True
|
||||
assert call_args[0][1]["enable_youtube_upload"] is False
|
||||
|
||||
def test_rejects_non_editable_job_id(self, client, mock_job):
|
||||
"""Test that job_id cannot be updated."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"job_id": "new-job-id"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_non_editable_created_at(self, client, mock_job):
|
||||
"""Test that created_at cannot be updated."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"created_at": "2025-01-01T00:00:00Z"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_status_update(self, client, mock_job):
|
||||
"""Test that status cannot be updated via PATCH (use reset endpoint)."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"status": "complete"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_state_data_update(self, client, mock_job):
|
||||
"""Test that state_data cannot be updated directly."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"state_data": {"foo": "bar"}},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/nonexistent-job",
|
||||
json={"artist": "New Artist"},
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_400_for_empty_update(self, client, mock_job):
|
||||
"""Test 400 when no valid fields are provided."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "no valid" in response.json()["detail"].lower()
|
||||
|
||||
def test_logs_admin_changes(self, client, mock_job):
|
||||
"""Test that admin changes are logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify logging was called
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin@example.com" in log_message.lower() or "admin" in log_message.lower()
|
||||
|
||||
def test_update_customer_fields(self, client, mock_job):
|
||||
"""Test updating made-for-you customer fields."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"customer_email": "customer@example.com",
|
||||
"customer_notes": "Special request: add extra countdown",
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["customer_email"] == "customer@example.com"
|
||||
assert call_args[0][1]["customer_notes"] == "Special request: add extra countdown"
|
||||
|
||||
|
||||
class TestUpdateJobAuthorization:
|
||||
"""Tests for authorization on the update endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -5,7 +5,7 @@ Tests the audio search service, API routes, and job model fields.
|
||||
"""
|
||||
import pytest
|
||||
import os
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.models.job import Job, JobCreate, JobStatus, STATE_TRANSITIONS
|
||||
@@ -845,7 +845,8 @@ class TestAudioSearchThemeSupport:
|
||||
This is the key behavior: selecting a theme should automatically
|
||||
enable CDG and TXT output formats.
|
||||
"""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# When theme_id is set, enable_cdg/enable_txt should default to True
|
||||
request = AudioSearchRequest(
|
||||
@@ -855,7 +856,7 @@ class TestAudioSearchThemeSupport:
|
||||
# enable_cdg and enable_txt are None (not specified)
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
@@ -864,7 +865,8 @@ class TestAudioSearchThemeSupport:
|
||||
|
||||
def test_audio_search_request_no_theme_no_cdg_txt(self):
|
||||
"""Test that without theme_id, CDG/TXT defaults to disabled."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
@@ -872,7 +874,7 @@ class TestAudioSearchThemeSupport:
|
||||
# No theme_id, no enable_cdg, no enable_txt
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
@@ -881,7 +883,8 @@ class TestAudioSearchThemeSupport:
|
||||
|
||||
def test_explicit_cdg_txt_overrides_theme_default(self):
|
||||
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# Theme set (would default to True), but explicitly disabled
|
||||
request = AudioSearchRequest(
|
||||
@@ -892,7 +895,7 @@ class TestAudioSearchThemeSupport:
|
||||
enable_txt=False,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
@@ -901,7 +904,8 @@ class TestAudioSearchThemeSupport:
|
||||
|
||||
def test_explicit_cdg_txt_enables_without_theme(self):
|
||||
"""Test that explicit True enables CDG/TXT even without theme."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# No theme (would default to False), but explicitly enabled
|
||||
request = AudioSearchRequest(
|
||||
@@ -911,7 +915,7 @@ class TestAudioSearchThemeSupport:
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
@@ -1215,44 +1219,48 @@ class TestAudioSearchApiRouteDownload:
|
||||
assert 'gcs_path' in call_kwargs
|
||||
assert call_kwargs['gcs_path'] == "uploads/job123/audio/"
|
||||
|
||||
def test_local_youtube_download_does_not_pass_gcs_path(self, mock_job_manager, mock_storage_service):
|
||||
"""Test that YouTube downloads don't use gcs_path (download locally, upload manually)."""
|
||||
def test_youtube_remote_download_uses_youtube_service(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads use YouTubeDownloadService.
|
||||
|
||||
After the consolidation refactor, all YouTube downloads go through
|
||||
YouTubeDownloadService instead of audio_search_service.download_by_id().
|
||||
"""
|
||||
from backend.api.routes.audio_search import _download_and_start_processing
|
||||
import asyncio
|
||||
import tempfile
|
||||
|
||||
# Setup mock job with YouTube search result
|
||||
|
||||
# Setup mock job with YouTube search result that has source_id
|
||||
mock_job = Mock()
|
||||
mock_job.state_data = {
|
||||
'audio_search_results': [{
|
||||
'title': 'Unwanted',
|
||||
'artist': 'Avril Lavigne',
|
||||
'provider': 'YouTube', # NOT a torrent source
|
||||
'title': 'Demons',
|
||||
'artist': 'Kenny Chesney',
|
||||
'provider': 'YouTube',
|
||||
'quality': 'Opus 128kbps',
|
||||
'source_id': 'qmTjyGH9ro4', # YouTube video ID (11 chars)
|
||||
'url': 'https://www.youtube.com/watch?v=qmTjyGH9ro4',
|
||||
}]
|
||||
}
|
||||
mock_job.audio_search_artist = 'Avril Lavigne'
|
||||
mock_job.audio_search_title = 'Unwanted'
|
||||
mock_job.audio_search_artist = 'Kenny Chesney'
|
||||
mock_job.audio_search_title = 'Demons'
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
# Create mock audio search service with remote client (but YouTube doesn't use it)
|
||||
|
||||
# Create mock audio search service (not used for YouTube anymore)
|
||||
mock_audio_service = Mock()
|
||||
mock_audio_service.is_remote_enabled.return_value = True
|
||||
|
||||
# Create a temp file to simulate downloaded file
|
||||
temp_file = tempfile.NamedTemporaryFile(suffix='.opus', delete=False)
|
||||
temp_file.write(b'fake audio data')
|
||||
temp_file.close()
|
||||
|
||||
try:
|
||||
# Mock download to return local path
|
||||
mock_download_result = Mock()
|
||||
mock_download_result.filepath = temp_file.name
|
||||
mock_audio_service.download.return_value = mock_download_result
|
||||
|
||||
# Create mock background tasks
|
||||
mock_bg_tasks = Mock()
|
||||
|
||||
|
||||
# Create mock YouTubeDownloadService
|
||||
mock_youtube_service = Mock()
|
||||
mock_youtube_service.download_by_id = AsyncMock(
|
||||
return_value="uploads/job456/audio/Kenny Chesney - Demons.webm"
|
||||
)
|
||||
mock_youtube_service._extract_video_id = Mock(return_value="qmTjyGH9ro4")
|
||||
|
||||
# Create mock background tasks
|
||||
mock_bg_tasks = Mock()
|
||||
|
||||
# Patch get_youtube_download_service to return our mock
|
||||
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
|
||||
# Run the async function
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
@@ -1266,22 +1274,76 @@ class TestAudioSearchApiRouteDownload:
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# YouTube should NOT have gcs_path in kwargs
|
||||
mock_audio_service.download.assert_called_once()
|
||||
call_kwargs = mock_audio_service.download.call_args.kwargs
|
||||
|
||||
# For YouTube, gcs_path should NOT be set (or should be None)
|
||||
assert call_kwargs.get('gcs_path') is None
|
||||
|
||||
# Instead, storage service should be called to upload
|
||||
mock_storage_service.upload_fileobj.assert_called_once()
|
||||
finally:
|
||||
import os
|
||||
|
||||
# CRITICAL: Verify YouTubeDownloadService.download_by_id was called
|
||||
mock_youtube_service.download_by_id.assert_called_once()
|
||||
call_kwargs = mock_youtube_service.download_by_id.call_args.kwargs
|
||||
|
||||
# Verify correct arguments
|
||||
assert call_kwargs['video_id'] == 'qmTjyGH9ro4'
|
||||
assert call_kwargs['job_id'] == 'job456'
|
||||
assert call_kwargs['artist'] == 'Kenny Chesney'
|
||||
assert call_kwargs['title'] == 'Demons'
|
||||
|
||||
# audio_search_service.download_by_id should NOT be called for YouTube
|
||||
mock_audio_service.download_by_id.assert_not_called()
|
||||
|
||||
def test_youtube_download_always_uses_youtube_service(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads always use YouTubeDownloadService,
|
||||
regardless of remote enabled status. The service handles fallback internally.
|
||||
"""
|
||||
from backend.api.routes.audio_search import _download_and_start_processing
|
||||
import asyncio
|
||||
|
||||
# Setup mock job with YouTube search result (use valid 11-char video ID)
|
||||
mock_job = Mock()
|
||||
mock_job.state_data = {
|
||||
'audio_search_results': [{
|
||||
'title': 'Unwanted',
|
||||
'artist': 'Avril Lavigne',
|
||||
'provider': 'YouTube',
|
||||
'quality': 'Opus 128kbps',
|
||||
'source_id': 'abc123xyz45', # Valid 11-char ID
|
||||
'url': 'https://www.youtube.com/watch?v=abc123xyz45',
|
||||
}]
|
||||
}
|
||||
mock_job.audio_search_artist = 'Avril Lavigne'
|
||||
mock_job.audio_search_title = 'Unwanted'
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
# Create mock audio search service (not used for YouTube)
|
||||
mock_audio_service = Mock()
|
||||
mock_audio_service.is_remote_enabled.return_value = False
|
||||
|
||||
# Create mock YouTubeDownloadService
|
||||
mock_youtube_service = Mock()
|
||||
mock_youtube_service.download_by_id = AsyncMock(
|
||||
return_value="uploads/job789/audio/Avril Lavigne - Unwanted.opus"
|
||||
)
|
||||
mock_youtube_service._extract_video_id = Mock(return_value="abc123xyz45")
|
||||
|
||||
# Create mock background tasks
|
||||
mock_bg_tasks = Mock()
|
||||
|
||||
# Patch get_youtube_download_service to return our mock
|
||||
with patch('backend.api.routes.audio_search.get_youtube_download_service', return_value=mock_youtube_service):
|
||||
# Run the async function
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
os.unlink(temp_file.name)
|
||||
except:
|
||||
pass
|
||||
result = loop.run_until_complete(
|
||||
_download_and_start_processing(
|
||||
job_id="job789",
|
||||
selection_index=0,
|
||||
audio_search_service=mock_audio_service,
|
||||
background_tasks=mock_bg_tasks,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# YouTubeDownloadService.download_by_id should always be called for YouTube
|
||||
mock_youtube_service.download_by_id.assert_called_once()
|
||||
|
||||
def test_handles_gcs_path_response_correctly(self, mock_job_manager, mock_storage_service):
|
||||
"""Test that GCS path responses are parsed correctly."""
|
||||
@@ -1338,41 +1400,43 @@ class TestAudioSearchApiRouteDownload:
|
||||
|
||||
assert gcs_path_set, "input_media_gcs_path was not set in job update"
|
||||
|
||||
def test_remote_disabled_always_uses_local(self, mock_job_manager, mock_storage_service):
|
||||
"""Test that when remote is disabled, even torrent sources use local download."""
|
||||
def test_torrent_download_requires_remote(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that torrent sources (RED/OPS) require remote flacfetch.
|
||||
|
||||
Torrent downloads can only happen on the flacfetch VM which has:
|
||||
- BitTorrent peer connectivity
|
||||
- Private tracker credentials
|
||||
- Long-running seeding capability
|
||||
|
||||
Cloud Run cannot download torrents, so remote must be enabled.
|
||||
"""
|
||||
from backend.api.routes.audio_search import _download_and_start_processing
|
||||
from backend.services.audio_search_service import DownloadError
|
||||
import asyncio
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
|
||||
# Setup mock job with RED search result
|
||||
mock_job = Mock()
|
||||
mock_job.state_data = {
|
||||
'audio_search_results': [{
|
||||
'title': 'Test',
|
||||
'artist': 'Test',
|
||||
'provider': 'RED', # Torrent source, but remote is disabled
|
||||
'provider': 'RED', # Torrent source requires remote
|
||||
'quality': 'FLAC',
|
||||
}]
|
||||
}
|
||||
mock_job_manager.get_job.return_value = mock_job
|
||||
|
||||
# Create temp file
|
||||
temp_file = tempfile.NamedTemporaryFile(suffix='.flac', delete=False)
|
||||
temp_file.write(b'fake')
|
||||
temp_file.close()
|
||||
|
||||
|
||||
# Create mock audio search service WITHOUT remote client
|
||||
mock_audio_service = Mock()
|
||||
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
# Create mock audio search service WITHOUT remote client
|
||||
mock_audio_service = Mock()
|
||||
mock_audio_service.is_remote_enabled.return_value = False # REMOTE DISABLED
|
||||
|
||||
mock_download_result = Mock()
|
||||
mock_download_result.filepath = temp_file.name
|
||||
mock_audio_service.download.return_value = mock_download_result
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
result = loop.run_until_complete(
|
||||
# Should raise an error because RED requires remote
|
||||
with pytest.raises(Exception) as exc_info:
|
||||
loop.run_until_complete(
|
||||
_download_and_start_processing(
|
||||
job_id="job_no_remote",
|
||||
selection_index=0,
|
||||
@@ -1380,19 +1444,7 @@ class TestAudioSearchApiRouteDownload:
|
||||
background_tasks=Mock(),
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# When remote is disabled, gcs_path should NOT be passed
|
||||
mock_audio_service.download.assert_called_once()
|
||||
call_kwargs = mock_audio_service.download.call_args.kwargs
|
||||
assert call_kwargs.get('gcs_path') is None
|
||||
|
||||
# And storage should upload manually
|
||||
mock_storage_service.upload_fileobj.assert_called_once()
|
||||
# Verify the error message mentions remote flacfetch
|
||||
assert "remote flacfetch" in str(exc_info.value).lower() or "flacfetch" in str(exc_info.value).lower()
|
||||
finally:
|
||||
import os
|
||||
try:
|
||||
os.unlink(temp_file.name)
|
||||
except:
|
||||
pass
|
||||
loop.close()
|
||||
@@ -0,0 +1,409 @@
|
||||
"""
|
||||
Contract tests for countdown padding state synchronization.
|
||||
|
||||
These tests verify the data contract between:
|
||||
- render_video_worker (WRITER) - must set has_countdown_padding after adding countdown
|
||||
- video_worker (READER) - must read and use has_countdown_padding
|
||||
|
||||
The bug this test suite catches:
|
||||
- render_video_worker adds countdown (deferred from lyrics phase) but doesn't update state
|
||||
- video_worker reads lyrics_metadata.has_countdown_padding which is still False
|
||||
- Result: instrumental is not padded, causing audio desync
|
||||
|
||||
These tests will FAIL if either side of the contract is broken.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestCountdownStateSchema:
|
||||
"""Tests that the lyrics_metadata schema supports countdown padding fields."""
|
||||
|
||||
def test_lyrics_metadata_can_store_has_countdown_padding(self):
|
||||
"""lyrics_metadata must support has_countdown_padding (bool) field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
|
||||
def test_lyrics_metadata_can_store_countdown_padding_seconds(self):
|
||||
"""lyrics_metadata must support countdown_padding_seconds (float) field."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'countdown_padding_seconds': 3.0
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
def test_lyrics_metadata_defaults_to_no_countdown(self):
|
||||
"""Default state should have has_countdown_padding=False."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.RENDERING_VIDEO,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding', False) is False
|
||||
|
||||
|
||||
class TestRenderVideoWorkerWriterContract:
|
||||
"""
|
||||
WRITER CONTRACT: render_video_worker must update job state when countdown is added.
|
||||
|
||||
When CountdownProcessor.process() returns padding_added=True:
|
||||
- render_video_worker MUST call job_manager.update_state_data to set
|
||||
lyrics_metadata.has_countdown_padding = True
|
||||
lyrics_metadata.countdown_padding_seconds = <padding_seconds>
|
||||
"""
|
||||
|
||||
def test_render_video_updates_state_when_countdown_added(self):
|
||||
"""
|
||||
WRITER CONTRACT TEST: When countdown is added, state MUST be updated.
|
||||
|
||||
This test verifies the state update logic that should execute in render_video_worker
|
||||
after CountdownProcessor adds padding. We test the logic directly to avoid the
|
||||
complexity of mocking the full worker function.
|
||||
|
||||
The actual code in render_video_worker.py should look like:
|
||||
if padding_added:
|
||||
existing_lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
existing_lyrics_metadata['has_countdown_padding'] = True
|
||||
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
|
||||
job_manager.update_state_data(job_id, 'lyrics_metadata', existing_lyrics_metadata)
|
||||
"""
|
||||
# Simulate the inputs from CountdownProcessor
|
||||
padding_added = True
|
||||
padding_seconds = 3.0
|
||||
|
||||
# Simulate existing job state
|
||||
existing_state_data = {
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0
|
||||
}
|
||||
}
|
||||
|
||||
# This is the exact logic that should be in render_video_worker.py
|
||||
if padding_added:
|
||||
existing_lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
|
||||
existing_lyrics_metadata['has_countdown_padding'] = True
|
||||
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
|
||||
|
||||
# Verify the state was updated correctly
|
||||
assert existing_lyrics_metadata['has_countdown_padding'] is True, (
|
||||
"WRITER CONTRACT: has_countdown_padding must be True when countdown added"
|
||||
)
|
||||
assert existing_lyrics_metadata['countdown_padding_seconds'] == 3.0, (
|
||||
f"WRITER CONTRACT: countdown_padding_seconds must match padding_seconds"
|
||||
)
|
||||
|
||||
def test_render_video_worker_source_contains_state_update(self):
|
||||
"""
|
||||
WRITER CONTRACT TEST: Verify the state update code exists in render_video_worker.py.
|
||||
|
||||
This is a source code inspection test that ensures the critical state update
|
||||
code is present and hasn't been accidentally removed or refactored incorrectly.
|
||||
"""
|
||||
import inspect
|
||||
from backend.workers import render_video_worker
|
||||
|
||||
# Get the source code of the module
|
||||
source = inspect.getsource(render_video_worker)
|
||||
|
||||
# Verify the critical state update patterns exist
|
||||
assert "existing_lyrics_metadata['has_countdown_padding'] = True" in source, (
|
||||
"WRITER CONTRACT VIOLATION: render_video_worker.py must contain "
|
||||
"code to set has_countdown_padding=True when countdown is added"
|
||||
)
|
||||
|
||||
assert "existing_lyrics_metadata['countdown_padding_seconds']" in source, (
|
||||
"WRITER CONTRACT VIOLATION: render_video_worker.py must contain "
|
||||
"code to set countdown_padding_seconds when countdown is added"
|
||||
)
|
||||
|
||||
assert "job_manager.update_state_data(job_id, 'lyrics_metadata'" in source, (
|
||||
"WRITER CONTRACT VIOLATION: render_video_worker.py must call "
|
||||
"job_manager.update_state_data to persist countdown state"
|
||||
)
|
||||
|
||||
def test_render_video_no_state_update_when_countdown_not_needed(self):
|
||||
"""
|
||||
When countdown is NOT needed, state should NOT be updated to True.
|
||||
|
||||
This ensures we don't incorrectly set has_countdown_padding when
|
||||
the song starts after 3 seconds and doesn't need countdown.
|
||||
"""
|
||||
# Simulate the inputs when countdown is NOT needed
|
||||
padding_added = False
|
||||
padding_seconds = 0.0
|
||||
|
||||
# Simulate existing job state
|
||||
existing_state_data = {
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0
|
||||
}
|
||||
}
|
||||
|
||||
# The logic should NOT update state when padding_added is False
|
||||
if padding_added:
|
||||
existing_lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
|
||||
existing_lyrics_metadata['has_countdown_padding'] = True
|
||||
existing_lyrics_metadata['countdown_padding_seconds'] = padding_seconds
|
||||
|
||||
# Verify state was NOT changed
|
||||
lyrics_metadata = existing_state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding') is False, (
|
||||
"State should remain False when countdown not added"
|
||||
)
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 0.0, (
|
||||
"countdown_padding_seconds should remain 0 when countdown not added"
|
||||
)
|
||||
|
||||
|
||||
class TestVideoWorkerReaderContract:
|
||||
"""
|
||||
READER CONTRACT: video_worker must read and use countdown state for instrumental padding.
|
||||
|
||||
When lyrics_metadata.has_countdown_padding = True:
|
||||
- video_worker MUST extract countdown_padding_seconds
|
||||
- video_worker MUST pass countdown_padding_seconds to encoding backend
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def job_with_countdown(self):
|
||||
"""Create a job where countdown padding was added."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
file_urls={
|
||||
'instrumental_clean': 'gs://bucket/instrumental.flac',
|
||||
'video_output': 'gs://bucket/video.mp4',
|
||||
},
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def job_without_countdown(self):
|
||||
"""Create a job where countdown padding was NOT added."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
file_urls={
|
||||
'instrumental_clean': 'gs://bucket/instrumental.flac',
|
||||
'video_output': 'gs://bucket/video.mp4',
|
||||
},
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_video_worker_reads_countdown_padding_from_state(self, job_with_countdown):
|
||||
"""
|
||||
READER CONTRACT: video_worker must read has_countdown_padding from lyrics_metadata.
|
||||
"""
|
||||
# This mirrors the logic in video_worker.py
|
||||
lyrics_metadata = job_with_countdown.state_data.get('lyrics_metadata', {})
|
||||
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0
|
||||
|
||||
def test_video_worker_returns_none_when_no_countdown(self, job_without_countdown):
|
||||
"""
|
||||
READER CONTRACT: video_worker returns None when has_countdown_padding=False.
|
||||
"""
|
||||
lyrics_metadata = job_without_countdown.state_data.get('lyrics_metadata', {})
|
||||
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
def test_video_worker_passes_countdown_to_karaoke_finalise(self, job_with_countdown):
|
||||
"""
|
||||
READER CONTRACT: video_worker must pass countdown_padding_seconds to KaraokeFinalise.
|
||||
|
||||
This verifies the standard encoding path uses countdown info by testing the
|
||||
extraction logic that would be used to pass the value to KaraokeFinalise.
|
||||
|
||||
Note: This is a focused unit test that verifies the contract without running
|
||||
the full complex worker function. The actual integration is tested in
|
||||
emulator tests.
|
||||
"""
|
||||
# Simulate the exact code path from video_worker.py lines 541-544
|
||||
lyrics_metadata = job_with_countdown.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# CRITICAL ASSERTION: The value that would be passed to KaraokeFinalise
|
||||
assert countdown_padding_seconds is not None, (
|
||||
"READER CONTRACT VIOLATION: countdown_padding_seconds should be extracted "
|
||||
"when has_countdown_padding is True"
|
||||
)
|
||||
assert countdown_padding_seconds == 3.0, (
|
||||
f"READER CONTRACT VIOLATION: Expected countdown_padding_seconds=3.0, "
|
||||
f"got {countdown_padding_seconds}"
|
||||
)
|
||||
|
||||
def test_karaoke_finalise_accepts_countdown_parameter(self):
|
||||
"""
|
||||
READER CONTRACT: KaraokeFinalise constructor must accept countdown_padding_seconds.
|
||||
|
||||
This verifies that the encoding layer can receive the countdown info.
|
||||
"""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
import inspect
|
||||
|
||||
# Get the signature of KaraokeFinalise.__init__
|
||||
sig = inspect.signature(KaraokeFinalise.__init__)
|
||||
param_names = list(sig.parameters.keys())
|
||||
|
||||
assert 'countdown_padding_seconds' in param_names, (
|
||||
"READER CONTRACT VIOLATION: KaraokeFinalise does not accept "
|
||||
"countdown_padding_seconds parameter. This is required for audio sync."
|
||||
)
|
||||
|
||||
|
||||
class TestEndToEndStateFlow:
|
||||
"""
|
||||
Tests that verify the complete state flow from render_video_worker to video_worker.
|
||||
"""
|
||||
|
||||
def test_state_written_by_render_can_be_read_by_video(self):
|
||||
"""
|
||||
State written by render_video_worker format can be read by video_worker logic.
|
||||
|
||||
This ensures both sides use the same state format.
|
||||
"""
|
||||
# Simulate state written by render_video_worker
|
||||
state_from_render = {
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
}
|
||||
|
||||
# Create job with this state
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data=state_from_render
|
||||
)
|
||||
|
||||
# Simulate video_worker reading logic (from video_worker.py lines 539-545)
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# Verify the state can be correctly read
|
||||
assert countdown_padding_seconds == 3.0, (
|
||||
"State format mismatch: render_video_worker writes in a format "
|
||||
"that video_worker cannot correctly read"
|
||||
)
|
||||
|
||||
def test_missing_lyrics_metadata_handled_gracefully(self):
|
||||
"""
|
||||
video_worker should handle missing lyrics_metadata gracefully.
|
||||
"""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={} # No lyrics_metadata at all
|
||||
)
|
||||
|
||||
# This should not raise an exception
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
def test_countdown_padding_seconds_defaults_to_3_when_missing(self):
|
||||
"""
|
||||
If has_countdown_padding=True but countdown_padding_seconds is missing,
|
||||
default to 3.0 seconds (the standard countdown duration).
|
||||
"""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
# countdown_padding_seconds intentionally missing
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0
|
||||
@@ -65,39 +65,87 @@ class TestDropboxService:
|
||||
def test_get_next_brand_code_first_track(self):
|
||||
"""Test brand code calculation when no existing tracks."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
with patch.object(service, 'list_folders', return_value=[]):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0001"
|
||||
|
||||
def test_get_next_brand_code_sequential(self):
|
||||
"""Test brand code calculation with existing tracks."""
|
||||
def test_get_next_brand_code_legacy_gaps_preserved(self):
|
||||
"""Test that gaps below 1000 are NOT filled (legacy behavior)."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"NOMAD-0002 - Artist2 - Song2",
|
||||
"NOMAD-0005 - Artist3 - Song3", # Gap in sequence
|
||||
"OTHER-0001 - Different Brand", # Different brand
|
||||
"NOMAD-0005 - Artist3 - Song3", # Gap at 0003, 0004 - should be preserved
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0006"
|
||||
assert brand_code == "NOMAD-0006" # Does NOT fill legacy gap
|
||||
|
||||
def test_get_next_brand_code_fills_gap_above_1000(self):
|
||||
"""Test that gaps at 1001+ ARE filled."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-1001 - Artist1 - Song1",
|
||||
"NOMAD-1002 - Artist2 - Song2",
|
||||
"NOMAD-1005 - Artist3 - Song3", # Gap at 1003, 1004
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-1003" # Fills gap at 1003
|
||||
|
||||
def test_get_next_brand_code_no_gaps_above_1000(self):
|
||||
"""Test brand code calculation with no gaps above 1000 returns max+1."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-1001 - Artist1 - Song1",
|
||||
"NOMAD-1002 - Artist2 - Song2",
|
||||
"NOMAD-1003 - Artist3 - Song3",
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-1004"
|
||||
|
||||
def test_get_next_brand_code_mixed_legacy_and_new(self):
|
||||
"""Test with both legacy (<1000) and new (>=1000) codes."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"NOMAD-0005 - Artist2 - Song2", # Legacy gap - preserved
|
||||
"NOMAD-1001 - Artist3 - Song3",
|
||||
"NOMAD-1003 - Artist4 - Song4", # Gap at 1002 - should be filled
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-1002" # Fills gap at 1002, ignores legacy gap
|
||||
|
||||
def test_get_next_brand_code_different_prefix(self):
|
||||
"""Test brand code calculation with different brand prefix."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
existing_folders = [
|
||||
"NOMAD-0001 - Artist1 - Song1",
|
||||
"TEST-0010 - Artist2 - Song2",
|
||||
"NOMAD-0002 - Artist2 - Song2",
|
||||
"TEST-0001 - Artist3 - Song3",
|
||||
"TEST-0002 - Artist4 - Song4",
|
||||
]
|
||||
with patch.object(service, 'list_folders', return_value=existing_folders):
|
||||
# NOMAD sequence: 1,2 (below 1000) -> next is 3
|
||||
brand_code = service.get_next_brand_code("/test/path", "NOMAD")
|
||||
assert brand_code == "NOMAD-0003"
|
||||
# TEST sequence: 1,2 (below 1000) -> next is 3
|
||||
brand_code = service.get_next_brand_code("/test/path", "TEST")
|
||||
assert brand_code == "TEST-0011"
|
||||
assert brand_code == "TEST-0003"
|
||||
|
||||
def test_factory_function_returns_instance(self):
|
||||
"""Test that get_dropbox_service returns a service instance."""
|
||||
|
||||
@@ -222,30 +222,31 @@ class TestGetNextBrandCode:
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code(self, mock_sm_client_class):
|
||||
"""Test calculating next brand code from existing folders."""
|
||||
"""Test calculating next brand code - fills gaps only above 1000."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
# Mock list_folders to return existing codes
|
||||
|
||||
# Mock list_folders to return existing codes with a gap at 1003
|
||||
with patch.object(service, "list_folders") as mock_list:
|
||||
mock_list.return_value = [
|
||||
"NOMAD-1161",
|
||||
"NOMAD-1162",
|
||||
"NOMAD-1163",
|
||||
"NOMAD-1001",
|
||||
"NOMAD-1002",
|
||||
"NOMAD-1004", # Gap at 1003 - should be filled
|
||||
"NOMAD-1005",
|
||||
"Other Folder",
|
||||
"NOMAD-0001",
|
||||
]
|
||||
|
||||
|
||||
next_code = service.get_next_brand_code("/path", "NOMAD")
|
||||
|
||||
assert next_code == "NOMAD-1164"
|
||||
|
||||
# Should fill the gap at 1003 (gaps >= 1001 are filled)
|
||||
assert next_code == "NOMAD-1003"
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_get_next_brand_code_empty_folder(self, mock_sm_client_class):
|
||||
|
||||
@@ -0,0 +1,725 @@
|
||||
"""
|
||||
Unit tests for email service.
|
||||
|
||||
Tests the new job completion and action reminder email methods,
|
||||
as well as CC support.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.email_service import (
|
||||
EmailService,
|
||||
EmailProvider,
|
||||
ConsoleEmailProvider,
|
||||
SendGridEmailProvider,
|
||||
get_email_service,
|
||||
)
|
||||
|
||||
|
||||
class TestConsoleEmailProvider:
|
||||
"""Tests for console email provider."""
|
||||
|
||||
def test_send_email_basic(self):
|
||||
"""Test basic email logging."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test content</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_text_content(self):
|
||||
"""Test email with text content."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
text_content="Test plain text",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email with CC recipients."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestSendGridEmailProvider:
|
||||
"""Tests for SendGrid email provider."""
|
||||
|
||||
def test_send_email_success(self):
|
||||
"""Test successful email sending via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
from_name="Test Sender",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at the sendgrid module level since it's imported inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_client.send.assert_called_once()
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email sending with CC via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_failure_status(self):
|
||||
"""Test email sending failure due to bad status."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 400
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_email_exception(self):
|
||||
"""Test email sending exception handling."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_sg.return_value.send.side_effect = Exception("API error")
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestEmailServiceJobCompletion:
|
||||
"""Tests for job completion email method."""
|
||||
|
||||
def test_send_job_completion_basic(self):
|
||||
"""Test basic job completion email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_job_completion_with_song_info(self):
|
||||
"""Test job completion email includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_job_completion_with_brand_code(self):
|
||||
"""Test job completion email includes brand code in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
brand_code="NOMAD-1178",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
|
||||
def test_send_job_completion_default_subject(self):
|
||||
"""Test job completion email default subject without song info."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "karaoke video is ready" in subject.lower()
|
||||
|
||||
def test_send_job_completion_with_cc(self):
|
||||
"""Test job completion email with CC to admin."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
|
||||
|
||||
def test_send_job_completion_without_cc(self):
|
||||
"""Test job completion email without CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=False,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
def test_send_job_completion_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_job_completion_includes_plain_text(self):
|
||||
"""Test that plain text content is included."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
message = "Your video is ready!"
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content=message,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('text_content') == message
|
||||
|
||||
|
||||
class TestEmailServiceActionReminder:
|
||||
"""Tests for action reminder email method."""
|
||||
|
||||
def test_send_action_reminder_lyrics(self):
|
||||
"""Test lyrics action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review your lyrics",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "lyrics" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_instrumental(self):
|
||||
"""Test instrumental action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please select instrumental",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "instrumental" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_with_song_info(self):
|
||||
"""Test action reminder includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_action_reminder_unknown_type(self):
|
||||
"""Test action reminder with unknown type still sends."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please take action",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Action needed" in subject
|
||||
|
||||
def test_send_action_reminder_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_action_reminder_no_cc(self):
|
||||
"""Test that action reminders don't have CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
# Action reminders should not have CC
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
|
||||
class TestEmailServiceConfiguration:
|
||||
"""Tests for email service configuration."""
|
||||
|
||||
def test_uses_sendgrid_when_configured(self):
|
||||
"""Test that SendGrid is used when API key is set."""
|
||||
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, SendGridEmailProvider)
|
||||
|
||||
def test_uses_console_when_not_configured(self):
|
||||
"""Test that console is used when no API key."""
|
||||
with patch.dict('os.environ', {}, clear=True):
|
||||
# Remove SENDGRID_API_KEY if it exists
|
||||
import os
|
||||
original = os.environ.pop('SENDGRID_API_KEY', None)
|
||||
try:
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, ConsoleEmailProvider)
|
||||
finally:
|
||||
if original:
|
||||
os.environ['SENDGRID_API_KEY'] = original
|
||||
|
||||
def test_is_configured_true_for_sendgrid(self):
|
||||
"""Test is_configured returns True for SendGrid."""
|
||||
service = EmailService()
|
||||
service.provider = SendGridEmailProvider("key", "from@example.com")
|
||||
assert service.is_configured() is True
|
||||
|
||||
def test_is_configured_false_for_console(self):
|
||||
"""Test is_configured returns False for console."""
|
||||
service = EmailService()
|
||||
service.provider = ConsoleEmailProvider()
|
||||
assert service.is_configured() is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_email_service_returns_same_instance(self):
|
||||
"""Test that get_email_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.email_service as es
|
||||
es._email_service = None
|
||||
|
||||
service1 = get_email_service()
|
||||
service2 = get_email_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestCCFunctionality:
|
||||
"""Integration tests for CC functionality across providers."""
|
||||
|
||||
def test_console_provider_logs_cc(self, caplog):
|
||||
"""Test that console provider logs CC recipients."""
|
||||
import logging
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
provider = ConsoleEmailProvider()
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
# Check that CC was logged
|
||||
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
|
||||
|
||||
def test_sendgrid_provider_adds_cc_recipients(self):
|
||||
"""Test that SendGrid provider adds CC recipients to message."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at sendgrid module level since imports are inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
# We need to capture the Mail object
|
||||
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
|
||||
mock_mail = MagicMock()
|
||||
mock_mail_class.return_value = mock_mail
|
||||
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
# Verify add_cc was called
|
||||
mock_mail.add_cc.assert_called()
|
||||
|
||||
|
||||
class TestMadeForYouOrderConfirmation:
|
||||
"""Tests for send_made_for_you_order_confirmation method."""
|
||||
|
||||
def test_send_order_confirmation_success(self):
|
||||
"""Test successful order confirmation email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-123",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
# Verify BCC is set
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('bcc_emails') == ["done@nomadkaraoke.com"]
|
||||
|
||||
def test_send_order_confirmation_subject_format(self):
|
||||
"""Test order confirmation has correct subject format."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
job_id="test-job-456",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Order Confirmed" in subject
|
||||
assert "Seether" in subject
|
||||
assert "Tonight" in subject
|
||||
|
||||
def test_send_order_confirmation_with_notes(self):
|
||||
"""Test order confirmation includes notes."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-789",
|
||||
notes="Wedding anniversary!",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "Wedding anniversary!" in html_content
|
||||
|
||||
def test_send_order_confirmation_includes_delivery_promise(self):
|
||||
"""Test order confirmation mentions delivery timeframe."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content', '')
|
||||
text_content = call_kwargs.get('text_content', '')
|
||||
# Should mention delivery timeline
|
||||
assert "24" in html_content or "24" in text_content
|
||||
|
||||
def test_send_order_confirmation_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = False
|
||||
|
||||
result = service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-def",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestMadeForYouAdminNotification:
|
||||
"""Tests for send_made_for_you_admin_notification method."""
|
||||
|
||||
def test_send_admin_notification_success(self):
|
||||
"""Test successful admin notification email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_admin_notification_subject_format(self):
|
||||
"""Test admin notification has correct subject format."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
# Subject format: "Karaoke Order: {artist} - {title} [ID: {job_id}]"
|
||||
assert "Karaoke Order" in subject
|
||||
assert "Seether" in subject
|
||||
assert "Tonight" in subject
|
||||
|
||||
def test_send_admin_notification_includes_customer_email(self):
|
||||
"""Test admin notification includes customer email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="vip@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "vip@example.com" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_job_link(self):
|
||||
"""Test admin notification includes job link."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="abc-def-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "abc-def-123" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_audio_count(self):
|
||||
"""Test admin notification includes audio source count."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
audio_source_count=5,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "5" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_notes(self):
|
||||
"""Test admin notification includes customer notes."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
notes="Rush order please!",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "Rush order please!" in html_content
|
||||
|
||||
def test_send_admin_notification_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = False
|
||||
|
||||
result = service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
@@ -0,0 +1,298 @@
|
||||
"""
|
||||
Unit tests for EmailValidationService.
|
||||
|
||||
Tests email normalization, disposable domain detection, and blocklist management.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
# Mock Google Cloud before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.storage'] = MagicMock()
|
||||
|
||||
|
||||
class TestEmailNormalization:
|
||||
"""Test email normalization logic."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
# Default: empty blocklist config
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
# =========================================================================
|
||||
# Gmail Normalization Tests
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_gmail_removes_dots(self, email_service):
|
||||
"""Test that dots are removed from Gmail local part."""
|
||||
result = email_service.normalize_email("j.o.h.n@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_gmail_removes_plus_suffix(self, email_service):
|
||||
"""Test that +tag is removed from Gmail local part."""
|
||||
result = email_service.normalize_email("john+spam@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_gmail_removes_dots_and_plus(self, email_service):
|
||||
"""Test both dots and +tag are removed from Gmail."""
|
||||
result = email_service.normalize_email("j.o.h.n+newsletter@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_googlemail_treated_like_gmail(self, email_service):
|
||||
"""Test that googlemail.com is normalized like gmail.com."""
|
||||
result = email_service.normalize_email("j.o.h.n+test@googlemail.com")
|
||||
assert result == "john@googlemail.com"
|
||||
|
||||
def test_normalize_gmail_to_lowercase(self, email_service):
|
||||
"""Test Gmail addresses are lowercased."""
|
||||
result = email_service.normalize_email("JOHN.DOE+Test@GMAIL.COM")
|
||||
assert result == "johndoe@gmail.com"
|
||||
|
||||
# =========================================================================
|
||||
# Non-Gmail Normalization Tests
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_non_gmail_preserves_dots(self, email_service):
|
||||
"""Test dots are preserved for non-Gmail domains."""
|
||||
result = email_service.normalize_email("j.o.h.n@example.com")
|
||||
assert result == "j.o.h.n@example.com"
|
||||
|
||||
def test_normalize_non_gmail_preserves_plus(self, email_service):
|
||||
"""Test +tag is preserved for non-Gmail domains."""
|
||||
result = email_service.normalize_email("john+tag@company.com")
|
||||
assert result == "john+tag@company.com"
|
||||
|
||||
def test_normalize_non_gmail_to_lowercase(self, email_service):
|
||||
"""Test non-Gmail addresses are lowercased."""
|
||||
result = email_service.normalize_email("John.Doe@Example.Com")
|
||||
assert result == "john.doe@example.com"
|
||||
|
||||
def test_normalize_strips_whitespace(self, email_service):
|
||||
"""Test whitespace is stripped."""
|
||||
result = email_service.normalize_email(" john@example.com ")
|
||||
assert result == "john@example.com"
|
||||
|
||||
# =========================================================================
|
||||
# Edge Cases
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_empty_string(self, email_service):
|
||||
"""Test empty string returns empty."""
|
||||
result = email_service.normalize_email("")
|
||||
assert result == ""
|
||||
|
||||
def test_normalize_none_returns_empty(self, email_service):
|
||||
"""Test None returns empty string."""
|
||||
result = email_service.normalize_email(None)
|
||||
assert result == ""
|
||||
|
||||
def test_normalize_invalid_no_at_sign(self, email_service):
|
||||
"""Test email without @ is returned as-is (lowercase)."""
|
||||
result = email_service.normalize_email("notanemail")
|
||||
assert result == "notanemail"
|
||||
|
||||
|
||||
class TestDisposableDomainDetection:
|
||||
"""Test disposable domain detection."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
# Clear any cached blocklist
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_detect_known_disposable_domain(self, email_service):
|
||||
"""Test known disposable domains are detected."""
|
||||
# These are in DEFAULT_DISPOSABLE_DOMAINS
|
||||
assert email_service.is_disposable_domain("user@tempmail.com") is True
|
||||
assert email_service.is_disposable_domain("user@mailinator.com") is True
|
||||
assert email_service.is_disposable_domain("user@guerrillamail.com") is True
|
||||
|
||||
def test_detect_legitimate_domain(self, email_service):
|
||||
"""Test legitimate domains are not flagged."""
|
||||
assert email_service.is_disposable_domain("user@gmail.com") is False
|
||||
assert email_service.is_disposable_domain("user@yahoo.com") is False
|
||||
assert email_service.is_disposable_domain("user@company.com") is False
|
||||
|
||||
def test_detect_case_insensitive(self, email_service):
|
||||
"""Test domain detection is case-insensitive."""
|
||||
assert email_service.is_disposable_domain("user@TEMPMAIL.COM") is True
|
||||
assert email_service.is_disposable_domain("user@TempMail.Com") is True
|
||||
|
||||
def test_detect_invalid_email_returns_false(self, email_service):
|
||||
"""Test invalid email returns False."""
|
||||
assert email_service.is_disposable_domain("notanemail") is False
|
||||
assert email_service.is_disposable_domain("") is False
|
||||
|
||||
|
||||
class TestBlocklistManagement:
|
||||
"""Test blocklist management functionality."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client with blocklist."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = True
|
||||
mock_doc.to_dict.return_value = {
|
||||
"disposable_domains": ["custom-temp.com"],
|
||||
"blocked_emails": ["spammer@example.com"],
|
||||
"blocked_ips": ["192.168.1.100"],
|
||||
}
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
# Clear any cached blocklist
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_is_email_blocked(self, email_service):
|
||||
"""Test blocked email detection."""
|
||||
assert email_service.is_email_blocked("spammer@example.com") is True
|
||||
assert email_service.is_email_blocked("legitimate@example.com") is False
|
||||
|
||||
def test_is_email_blocked_case_insensitive(self, email_service):
|
||||
"""Test blocked email detection is case-insensitive."""
|
||||
assert email_service.is_email_blocked("SPAMMER@example.com") is True
|
||||
assert email_service.is_email_blocked("Spammer@Example.Com") is True
|
||||
|
||||
def test_is_ip_blocked(self, email_service):
|
||||
"""Test blocked IP detection."""
|
||||
assert email_service.is_ip_blocked("192.168.1.100") is True
|
||||
assert email_service.is_ip_blocked("10.0.0.1") is False
|
||||
|
||||
def test_custom_disposable_domain_added(self, email_service):
|
||||
"""Test custom disposable domains from Firestore are included."""
|
||||
assert email_service.is_disposable_domain("user@custom-temp.com") is True
|
||||
|
||||
def test_blocklist_caching(self, email_service, mock_db):
|
||||
"""Test blocklist is cached."""
|
||||
# First call
|
||||
email_service.is_disposable_domain("user@test.com")
|
||||
# Second call should use cache
|
||||
email_service.is_disposable_domain("user@test2.com")
|
||||
|
||||
# Should only call Firestore once due to caching
|
||||
assert mock_db.collection.return_value.document.return_value.get.call_count == 1
|
||||
|
||||
|
||||
class TestBetaEnrollmentValidation:
|
||||
"""Test beta enrollment validation."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = True
|
||||
mock_doc.to_dict.return_value = {
|
||||
"disposable_domains": [],
|
||||
"blocked_emails": ["blocked@example.com"],
|
||||
"blocked_ips": [],
|
||||
}
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_validate_legitimate_email(self, email_service):
|
||||
"""Test legitimate email passes validation."""
|
||||
is_valid, error = email_service.validate_email_for_beta("user@gmail.com")
|
||||
assert is_valid is True
|
||||
assert error == ""
|
||||
|
||||
def test_validate_disposable_email_rejected(self, email_service):
|
||||
"""Test disposable email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("user@tempmail.com")
|
||||
assert is_valid is False
|
||||
assert "Disposable email" in error
|
||||
|
||||
def test_validate_blocked_email_rejected(self, email_service):
|
||||
"""Test blocked email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("blocked@example.com")
|
||||
assert is_valid is False
|
||||
assert "not allowed" in error
|
||||
|
||||
def test_validate_invalid_format_rejected(self, email_service):
|
||||
"""Test invalid email format is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("notanemail")
|
||||
assert is_valid is False
|
||||
assert "Invalid email format" in error
|
||||
|
||||
def test_validate_empty_email_rejected(self, email_service):
|
||||
"""Test empty email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("")
|
||||
assert is_valid is False
|
||||
|
||||
|
||||
class TestIPHashing:
|
||||
"""Test IP address hashing."""
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
mock_db = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
return EmailValidationService(db=mock_db)
|
||||
|
||||
def test_hash_ip_consistent(self, email_service):
|
||||
"""Test IP hashing produces consistent results."""
|
||||
hash1 = email_service.hash_ip("192.168.1.1")
|
||||
hash2 = email_service.hash_ip("192.168.1.1")
|
||||
assert hash1 == hash2
|
||||
|
||||
def test_hash_ip_different_for_different_ips(self, email_service):
|
||||
"""Test different IPs produce different hashes."""
|
||||
hash1 = email_service.hash_ip("192.168.1.1")
|
||||
hash2 = email_service.hash_ip("192.168.1.2")
|
||||
assert hash1 != hash2
|
||||
|
||||
def test_hash_ip_is_sha256(self, email_service):
|
||||
"""Test IP hash is SHA-256 (64 hex chars)."""
|
||||
hash_result = email_service.hash_ip("192.168.1.1")
|
||||
assert len(hash_result) == 64
|
||||
assert all(c in "0123456789abcdef" for c in hash_result)
|
||||
@@ -0,0 +1,493 @@
|
||||
"""
|
||||
Unit tests for countdown padding in all encoding paths.
|
||||
|
||||
These tests verify that all encoding backends correctly handle countdown padding
|
||||
to ensure instrumental audio is synchronized with the countdown-padded vocals.
|
||||
|
||||
The tests cover:
|
||||
1. KaraokeFinalise - standard encoding path
|
||||
2. Local encoding backend
|
||||
3. GCE encoding backend
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch, call
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestKaraokeFinaliseCountdownPadding:
|
||||
"""Tests for KaraokeFinalise countdown padding logic."""
|
||||
|
||||
def test_countdown_padding_stored_in_instance(self):
|
||||
"""countdown_padding_seconds is stored when passed to constructor."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=3.0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
assert finalise.countdown_padding_seconds == 3.0
|
||||
|
||||
def test_pad_audio_file_method_signature(self):
|
||||
"""_pad_audio_file accepts input, output, and padding_seconds."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
import inspect
|
||||
|
||||
sig = inspect.signature(KaraokeFinalise._pad_audio_file)
|
||||
params = list(sig.parameters.keys())
|
||||
|
||||
# Should have self, input_audio, output_audio, padding_seconds
|
||||
assert 'input_audio' in params
|
||||
assert 'output_audio' in params
|
||||
assert 'padding_seconds' in params
|
||||
|
||||
@patch('subprocess.run')
|
||||
def test_pad_audio_file_calls_ffmpeg(self, mock_subprocess):
|
||||
"""_pad_audio_file uses ffmpeg to add silence padding."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
mock_subprocess.return_value = MagicMock(returncode=0)
|
||||
|
||||
finalise = KaraokeFinalise(non_interactive=True)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
input_audio = os.path.join(temp_dir, "input.flac")
|
||||
output_audio = os.path.join(temp_dir, "output.flac")
|
||||
|
||||
# Create a fake input file
|
||||
with open(input_audio, 'w') as f:
|
||||
f.write("fake")
|
||||
|
||||
finalise._pad_audio_file(input_audio, output_audio, 3.0)
|
||||
|
||||
# Verify subprocess was called (ffmpeg)
|
||||
assert mock_subprocess.called
|
||||
call_args = str(mock_subprocess.call_args)
|
||||
|
||||
# Should use adelay filter for padding
|
||||
assert 'adelay' in call_args
|
||||
|
||||
def test_remux_creates_padded_instrumental_when_needed(self):
|
||||
"""remux_with_instrumental creates padded instrumental if countdown active."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=3.0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
# Mock the _pad_audio_file and execute_command methods
|
||||
finalise._pad_audio_file = MagicMock()
|
||||
finalise.execute_command = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
|
||||
instrumental = os.path.join(temp_dir, "instrumental.flac")
|
||||
output = os.path.join(temp_dir, "output.mp4")
|
||||
|
||||
# Create fake files
|
||||
for f in [with_vocals, instrumental]:
|
||||
with open(f, 'w') as fp:
|
||||
fp.write("fake")
|
||||
|
||||
finalise.remux_with_instrumental(with_vocals, instrumental, output)
|
||||
|
||||
# Should have called _pad_audio_file because instrumental isn't "(Padded)"
|
||||
finalise._pad_audio_file.assert_called_once()
|
||||
call_args = finalise._pad_audio_file.call_args
|
||||
|
||||
# Verify padding amount
|
||||
assert call_args[0][2] == 3.0 # Third arg is padding_seconds
|
||||
|
||||
def test_remux_skips_padding_when_already_padded(self):
|
||||
"""remux_with_instrumental skips padding if file already has (Padded)."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=3.0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
finalise._pad_audio_file = MagicMock()
|
||||
finalise.execute_command = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
|
||||
# Note: "(Padded)" in the filename
|
||||
instrumental = os.path.join(temp_dir, "instrumental (Padded).flac")
|
||||
output = os.path.join(temp_dir, "output.mp4")
|
||||
|
||||
for f in [with_vocals, instrumental]:
|
||||
with open(f, 'w') as fp:
|
||||
fp.write("fake")
|
||||
|
||||
finalise.remux_with_instrumental(with_vocals, instrumental, output)
|
||||
|
||||
# Should NOT have called _pad_audio_file
|
||||
finalise._pad_audio_file.assert_not_called()
|
||||
|
||||
def test_remux_no_padding_when_countdown_is_none(self):
|
||||
"""remux_with_instrumental doesn't pad when countdown_padding_seconds is None."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=None,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
finalise._pad_audio_file = MagicMock()
|
||||
finalise.execute_command = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
|
||||
instrumental = os.path.join(temp_dir, "instrumental.flac")
|
||||
output = os.path.join(temp_dir, "output.mp4")
|
||||
|
||||
for f in [with_vocals, instrumental]:
|
||||
with open(f, 'w') as fp:
|
||||
fp.write("fake")
|
||||
|
||||
finalise.remux_with_instrumental(with_vocals, instrumental, output)
|
||||
|
||||
# Should NOT have called _pad_audio_file
|
||||
finalise._pad_audio_file.assert_not_called()
|
||||
|
||||
def test_remux_no_padding_when_countdown_is_zero(self):
|
||||
"""remux_with_instrumental doesn't pad when countdown_padding_seconds is 0."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
finalise._pad_audio_file = MagicMock()
|
||||
finalise.execute_command = MagicMock()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with_vocals = os.path.join(temp_dir, "with_vocals.mp4")
|
||||
instrumental = os.path.join(temp_dir, "instrumental.flac")
|
||||
output = os.path.join(temp_dir, "output.mp4")
|
||||
|
||||
for f in [with_vocals, instrumental]:
|
||||
with open(f, 'w') as fp:
|
||||
fp.write("fake")
|
||||
|
||||
finalise.remux_with_instrumental(with_vocals, instrumental, output)
|
||||
|
||||
# Should NOT have called _pad_audio_file
|
||||
finalise._pad_audio_file.assert_not_called()
|
||||
|
||||
|
||||
class TestCountdownProcessorAudioPadding:
|
||||
"""Tests for CountdownProcessor audio padding logic."""
|
||||
|
||||
def test_countdown_processor_constants(self):
|
||||
"""CountdownProcessor has expected padding constants."""
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
|
||||
assert hasattr(CountdownProcessor, 'COUNTDOWN_PADDING_SECONDS')
|
||||
assert CountdownProcessor.COUNTDOWN_PADDING_SECONDS == 3.0
|
||||
|
||||
assert hasattr(CountdownProcessor, 'COUNTDOWN_THRESHOLD_SECONDS')
|
||||
assert CountdownProcessor.COUNTDOWN_THRESHOLD_SECONDS == 3.0
|
||||
|
||||
def test_countdown_processor_process_signature(self):
|
||||
"""CountdownProcessor.process() has the correct signature for countdown handling.
|
||||
|
||||
This test verifies the process() method returns the expected types:
|
||||
- CorrectionResult (modified with shifted timestamps)
|
||||
- str (path to audio file, possibly padded)
|
||||
- bool (padding_added)
|
||||
- float (padding_seconds)
|
||||
"""
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
import inspect
|
||||
|
||||
sig = inspect.signature(CountdownProcessor.process)
|
||||
|
||||
# Check parameters
|
||||
params = list(sig.parameters.keys())
|
||||
assert 'correction_result' in params, "process() must accept correction_result"
|
||||
assert 'audio_filepath' in params, "process() must accept audio_filepath"
|
||||
|
||||
def test_countdown_processor_determines_padding_need(self):
|
||||
"""CountdownProcessor correctly determines if padding is needed.
|
||||
|
||||
Songs that start within 3 seconds should trigger countdown padding.
|
||||
This tests the decision logic without running actual ffmpeg.
|
||||
"""
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
|
||||
# Verify the CountdownProcessor has the padding threshold logic
|
||||
processor = CountdownProcessor.__new__(CountdownProcessor)
|
||||
|
||||
# Check the default padding is 3.0 seconds
|
||||
# Note: CountdownProcessor should have COUNTDOWN_DURATION constant
|
||||
import inspect
|
||||
source = inspect.getsource(CountdownProcessor)
|
||||
|
||||
# Verify countdown duration is defined
|
||||
assert 'COUNTDOWN_DURATION' in source or '3.0' in source, (
|
||||
"CountdownProcessor should define countdown duration (typically 3.0 seconds)"
|
||||
)
|
||||
|
||||
|
||||
class TestVideoWorkerCountdownPassthrough:
|
||||
"""Tests that video_worker passes countdown to encoding correctly."""
|
||||
|
||||
@pytest.fixture
|
||||
def job_with_countdown(self):
|
||||
"""Job with countdown padding enabled."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
file_urls={
|
||||
'instrumental_clean': 'gs://bucket/instrumental.flac',
|
||||
'video_output': 'gs://bucket/video.mp4',
|
||||
},
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
},
|
||||
'video_worker': {
|
||||
'instrumental_source': 'clean',
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def job_without_countdown(self):
|
||||
"""Job without countdown padding."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
file_urls={
|
||||
'instrumental_clean': 'gs://bucket/instrumental.flac',
|
||||
'video_output': 'gs://bucket/video.mp4',
|
||||
},
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
},
|
||||
'video_worker': {
|
||||
'instrumental_source': 'clean',
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_extract_countdown_with_padding(self, job_with_countdown):
|
||||
"""video_worker extracts countdown_padding_seconds when padding enabled."""
|
||||
job = job_with_countdown
|
||||
|
||||
# This mirrors the logic in video_worker.py
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0
|
||||
|
||||
def test_extract_countdown_without_padding(self, job_without_countdown):
|
||||
"""video_worker returns None when no countdown padding."""
|
||||
job = job_without_countdown
|
||||
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds is None
|
||||
|
||||
def test_karaoke_finalise_receives_countdown(self, job_with_countdown):
|
||||
"""KaraokeFinalise constructor receives countdown_padding_seconds.
|
||||
|
||||
This test verifies that:
|
||||
1. KaraokeFinalise accepts the countdown_padding_seconds parameter
|
||||
2. The video_worker code passes the extracted value to KaraokeFinalise
|
||||
"""
|
||||
# Verify the value that would be passed to KaraokeFinalise
|
||||
job = job_with_countdown
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0, (
|
||||
"countdown_padding_seconds should be 3.0 when has_countdown_padding is True"
|
||||
)
|
||||
|
||||
# Verify KaraokeFinalise accepts the parameter
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
import inspect
|
||||
sig = inspect.signature(KaraokeFinalise.__init__)
|
||||
assert 'countdown_padding_seconds' in sig.parameters, (
|
||||
"KaraokeFinalise must accept countdown_padding_seconds parameter"
|
||||
)
|
||||
|
||||
def test_karaoke_finalise_receives_none_when_no_countdown(self, job_without_countdown):
|
||||
"""KaraokeFinalise receives None for countdown when no padding needed.
|
||||
|
||||
This test verifies that when has_countdown_padding is False,
|
||||
the countdown_padding_seconds remains None (not passed to KaraokeFinalise).
|
||||
"""
|
||||
job = job_without_countdown
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
# Should be None when no countdown
|
||||
assert countdown_padding_seconds is None, (
|
||||
"countdown_padding_seconds should be None when has_countdown_padding is False"
|
||||
)
|
||||
|
||||
def test_video_worker_source_passes_countdown_to_karaoke_finalise(self):
|
||||
"""Verify video_worker.py source code passes countdown to KaraokeFinalise.
|
||||
|
||||
This is a source code inspection test to ensure the parameter is actually
|
||||
being passed in the KaraokeFinalise constructor call.
|
||||
"""
|
||||
import inspect
|
||||
from backend.workers import video_worker
|
||||
|
||||
source = inspect.getsource(video_worker)
|
||||
|
||||
# Verify the KaraokeFinalise instantiation includes countdown_padding_seconds
|
||||
assert 'countdown_padding_seconds=countdown_padding_seconds' in source, (
|
||||
"video_worker.py must pass countdown_padding_seconds to KaraokeFinalise"
|
||||
)
|
||||
|
||||
|
||||
class TestGCEEncodingCountdownPassthrough:
|
||||
"""Tests for GCE encoding countdown handling."""
|
||||
|
||||
def test_gce_encoding_config_structure(self):
|
||||
"""GCE encoding config can include countdown_padding_seconds."""
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": "Test",
|
||||
"title": "Song",
|
||||
"countdown_padding_seconds": 3.0,
|
||||
}
|
||||
|
||||
assert encoding_config.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_gce_backend_encodes_with_options(self):
|
||||
"""GCE backend passes options through to encoding service."""
|
||||
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={
|
||||
'job_id': 'test123',
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
)
|
||||
|
||||
# The options field should be accessible
|
||||
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
|
||||
class TestLocalEncodingCountdownPassthrough:
|
||||
"""Tests for local encoding countdown handling."""
|
||||
|
||||
def test_local_encoding_input_with_countdown(self):
|
||||
"""Local encoding receives EncodingInput with countdown in options."""
|
||||
from backend.services.encoding_interface import LocalEncodingBackend, EncodingInput
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
)
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
|
||||
# Backend should be able to access countdown from options
|
||||
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
|
||||
class TestCountdownTimingInvariant:
|
||||
"""Tests for the audio timing invariant."""
|
||||
|
||||
def test_padding_preserves_timing_relationship(self):
|
||||
"""
|
||||
INVARIANT: If vocals are padded by N seconds, instrumental must also
|
||||
be padded by N seconds to maintain sync.
|
||||
|
||||
Final timing relationship:
|
||||
- Original first word at T seconds
|
||||
- After padding: first word at T + N seconds
|
||||
- Instrumental must also start N seconds later (via padding)
|
||||
"""
|
||||
vocal_first_word_original = 0.5 # seconds
|
||||
padding_seconds = 3.0
|
||||
|
||||
vocal_first_word_after_padding = vocal_first_word_original + padding_seconds
|
||||
|
||||
# Instrumental should be padded by same amount
|
||||
instrumental_start_original = 0.0
|
||||
instrumental_start_after_padding = instrumental_start_original + padding_seconds
|
||||
|
||||
# The relative timing should be preserved
|
||||
relative_timing_before = vocal_first_word_original - instrumental_start_original
|
||||
relative_timing_after = vocal_first_word_after_padding - instrumental_start_after_padding
|
||||
|
||||
assert relative_timing_before == relative_timing_after, (
|
||||
"INVARIANT VIOLATION: Padding must preserve relative timing between "
|
||||
"vocals and instrumental"
|
||||
)
|
||||
|
||||
def test_countdown_threshold_makes_sense(self):
|
||||
"""
|
||||
Songs starting before threshold need countdown.
|
||||
Songs starting after threshold don't need countdown.
|
||||
"""
|
||||
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
|
||||
|
||||
threshold = CountdownProcessor.COUNTDOWN_THRESHOLD_SECONDS
|
||||
|
||||
# Song starting at 0.5s needs countdown
|
||||
early_start = 0.5
|
||||
assert early_start < threshold, "Early start should be before threshold"
|
||||
|
||||
# Song starting at 5.0s doesn't need countdown
|
||||
late_start = 5.0
|
||||
assert late_start > threshold, "Late start should be after threshold"
|
||||
|
||||
# Padding should match threshold
|
||||
padding = CountdownProcessor.COUNTDOWN_PADDING_SECONDS
|
||||
assert padding == threshold, (
|
||||
"Padding duration should match threshold to give singer exactly "
|
||||
"the countdown time they need"
|
||||
)
|
||||
@@ -0,0 +1,365 @@
|
||||
"""
|
||||
Contract tests for encoding interface countdown support.
|
||||
|
||||
These tests verify that all encoding backends correctly handle countdown padding.
|
||||
|
||||
The encoding backends must:
|
||||
1. Accept countdown_padding_seconds parameter
|
||||
2. Pad the instrumental audio when countdown_padding_seconds > 0
|
||||
3. Ensure audio sync between padded vocals and instrumental
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from datetime import datetime, UTC
|
||||
from unittest.mock import MagicMock, AsyncMock, patch
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestEncodingInputCountdownSupport:
|
||||
"""Tests that EncodingInput can carry countdown information."""
|
||||
|
||||
def test_encoding_input_options_can_contain_countdown(self):
|
||||
"""EncodingInput.options dict can carry countdown_padding_seconds."""
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/to/title.mov",
|
||||
karaoke_video_path="/path/to/karaoke.mov",
|
||||
instrumental_audio_path="/path/to/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={
|
||||
'countdown_padding_seconds': 3.0
|
||||
}
|
||||
)
|
||||
|
||||
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
def test_encoding_input_options_default_to_empty(self):
|
||||
"""EncodingInput.options defaults to empty dict (no countdown)."""
|
||||
from backend.services.encoding_interface import EncodingInput
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/to/title.mov",
|
||||
karaoke_video_path="/path/to/karaoke.mov",
|
||||
instrumental_audio_path="/path/to/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
)
|
||||
|
||||
assert encoding_input.options == {}
|
||||
assert encoding_input.options.get('countdown_padding_seconds') is None
|
||||
|
||||
|
||||
class TestKaraokeFinaliseCountdownContract:
|
||||
"""Contract tests for KaraokeFinalise countdown padding support."""
|
||||
|
||||
def test_karaoke_finalise_accepts_countdown_parameter(self):
|
||||
"""KaraokeFinalise __init__ must accept countdown_padding_seconds."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
import inspect
|
||||
|
||||
sig = inspect.signature(KaraokeFinalise.__init__)
|
||||
params = list(sig.parameters.keys())
|
||||
|
||||
assert 'countdown_padding_seconds' in params, (
|
||||
"CONTRACT VIOLATION: KaraokeFinalise must accept countdown_padding_seconds parameter"
|
||||
)
|
||||
|
||||
def test_karaoke_finalise_stores_countdown_value(self):
|
||||
"""KaraokeFinalise stores countdown_padding_seconds for later use."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=3.0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
assert finalise.countdown_padding_seconds == 3.0
|
||||
|
||||
def test_karaoke_finalise_handles_none_countdown(self):
|
||||
"""KaraokeFinalise handles None countdown gracefully."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=None,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
assert finalise.countdown_padding_seconds is None
|
||||
|
||||
def test_karaoke_finalise_handles_zero_countdown(self):
|
||||
"""KaraokeFinalise handles zero countdown (no padding needed)."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
assert finalise.countdown_padding_seconds == 0
|
||||
|
||||
|
||||
class TestGCEEncodingCountdownContract:
|
||||
"""Contract tests for GCE encoding countdown support."""
|
||||
|
||||
def test_gce_encoding_config_can_include_countdown(self):
|
||||
"""GCE encoding config dict structure supports countdown_padding_seconds."""
|
||||
# This mirrors how video_worker builds the config for GCE
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": "Test Artist",
|
||||
"title": "Test Song",
|
||||
"countdown_padding_seconds": 3.0, # MUST be supported
|
||||
}
|
||||
|
||||
assert 'countdown_padding_seconds' in encoding_config
|
||||
assert encoding_config['countdown_padding_seconds'] == 3.0
|
||||
|
||||
def test_gce_backend_receives_countdown_in_config(self):
|
||||
"""GCE encoding backend receives countdown_padding_seconds in config.
|
||||
|
||||
This test verifies that the GCEEncodingBackend can receive and process
|
||||
countdown_padding_seconds through the EncodingInput options.
|
||||
"""
|
||||
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
|
||||
|
||||
# Create an encoding input with countdown padding
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={
|
||||
'job_id': 'test123',
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
)
|
||||
|
||||
# Verify the options can store countdown_padding_seconds
|
||||
assert encoding_input.options.get('countdown_padding_seconds') == 3.0, (
|
||||
"EncodingInput.options must support countdown_padding_seconds"
|
||||
)
|
||||
|
||||
# Verify GCEEncodingBackend can be instantiated
|
||||
backend = GCEEncodingBackend()
|
||||
assert backend is not None
|
||||
|
||||
def test_gce_encoding_path_documented(self):
|
||||
"""Document the GCE encoding path's countdown handling.
|
||||
|
||||
NOTE: The encoding_interface.py is a general abstraction that doesn't
|
||||
directly handle countdown_padding_seconds. The countdown handling for
|
||||
GCE encoding is managed in video_worker.py which passes it in the
|
||||
encoding_config to the GCE worker.
|
||||
|
||||
This test verifies the encoding interface structure is correct.
|
||||
"""
|
||||
from backend.services.encoding_interface import GCEEncodingBackend, EncodingInput
|
||||
|
||||
# Verify the interface supports options which can carry countdown info
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={'countdown_padding_seconds': 3.0} # Can be passed via options
|
||||
)
|
||||
|
||||
assert 'countdown_padding_seconds' in encoding_input.options, (
|
||||
"EncodingInput.options should support countdown_padding_seconds"
|
||||
)
|
||||
|
||||
|
||||
class TestLocalEncodingCountdownContract:
|
||||
"""Contract tests for local encoding countdown support."""
|
||||
|
||||
def test_local_encoding_service_exists(self):
|
||||
"""LocalEncodingService class exists and can be imported."""
|
||||
from backend.services.encoding_interface import LocalEncodingBackend
|
||||
assert LocalEncodingBackend is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_local_backend_uses_encoding_input_options(self):
|
||||
"""Local encoding backend should use options from EncodingInput."""
|
||||
from backend.services.encoding_interface import LocalEncodingBackend, EncodingInput
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
|
||||
encoding_input = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/instrumental.flac",
|
||||
artist="Test",
|
||||
title="Song",
|
||||
output_dir="/output",
|
||||
options={
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
)
|
||||
|
||||
# Verify options are accessible
|
||||
assert encoding_input.options.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
|
||||
class TestVideoWorkerEncodingPathContracts:
|
||||
"""Tests that video_worker passes countdown to all encoding paths."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_countdown(self):
|
||||
"""Create a job with countdown padding."""
|
||||
return Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.ENCODING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
artist="Test",
|
||||
title="Test",
|
||||
file_urls={
|
||||
'instrumental_clean': 'gs://bucket/instrumental.flac',
|
||||
'video_output': 'gs://bucket/video.mp4',
|
||||
},
|
||||
state_data={
|
||||
'lyrics_metadata': {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
},
|
||||
'video_worker': {
|
||||
'instrumental_source': 'clean',
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
def test_countdown_extraction_logic(self, mock_job_with_countdown):
|
||||
"""video_worker countdown extraction logic works correctly."""
|
||||
job = mock_job_with_countdown
|
||||
|
||||
# This is the extraction logic from video_worker.py
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
countdown_padding_seconds = None
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
countdown_padding_seconds = lyrics_metadata.get('countdown_padding_seconds', 3.0)
|
||||
|
||||
assert countdown_padding_seconds == 3.0
|
||||
|
||||
def test_karaoke_finalise_path_receives_countdown(self, mock_job_with_countdown):
|
||||
"""Standard encoding path passes countdown to KaraokeFinalise.
|
||||
|
||||
This test verifies that:
|
||||
1. The job state contains countdown info
|
||||
2. KaraokeFinalise accepts countdown_padding_seconds
|
||||
3. video_worker.py source passes countdown to KaraokeFinalise
|
||||
"""
|
||||
job = mock_job_with_countdown
|
||||
|
||||
# Verify job has countdown state
|
||||
lyrics_metadata = job.state_data.get('lyrics_metadata', {})
|
||||
assert lyrics_metadata.get('has_countdown_padding') is True
|
||||
assert lyrics_metadata.get('countdown_padding_seconds') == 3.0
|
||||
|
||||
# Verify KaraokeFinalise accepts the parameter
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
import inspect
|
||||
sig = inspect.signature(KaraokeFinalise.__init__)
|
||||
assert 'countdown_padding_seconds' in sig.parameters, (
|
||||
"KaraokeFinalise must accept countdown_padding_seconds parameter"
|
||||
)
|
||||
|
||||
# Verify video_worker.py passes the parameter
|
||||
source = inspect.getsource(inspect.getmodule(KaraokeFinalise))
|
||||
# Note: The actual source check is in the video_worker tests
|
||||
|
||||
|
||||
class TestCountdownPaddingInvariant:
|
||||
"""
|
||||
Tests the critical invariant:
|
||||
If countdown was added to vocals, instrumental MUST be padded to match.
|
||||
"""
|
||||
|
||||
def test_invariant_countdown_implies_padding(self):
|
||||
"""
|
||||
INVARIANT: has_countdown_padding=True implies countdown_padding_seconds > 0
|
||||
"""
|
||||
# Valid states
|
||||
valid_state_with_countdown = {
|
||||
'has_countdown_padding': True,
|
||||
'countdown_padding_seconds': 3.0,
|
||||
}
|
||||
|
||||
valid_state_without_countdown = {
|
||||
'has_countdown_padding': False,
|
||||
'countdown_padding_seconds': 0.0,
|
||||
}
|
||||
|
||||
# Check invariant
|
||||
if valid_state_with_countdown['has_countdown_padding']:
|
||||
assert valid_state_with_countdown['countdown_padding_seconds'] > 0, (
|
||||
"INVARIANT VIOLATION: has_countdown_padding=True but padding_seconds=0"
|
||||
)
|
||||
|
||||
if not valid_state_without_countdown['has_countdown_padding']:
|
||||
assert valid_state_without_countdown['countdown_padding_seconds'] == 0.0, (
|
||||
"INVARIANT VIOLATION: has_countdown_padding=False but padding_seconds > 0"
|
||||
)
|
||||
|
||||
def test_render_video_worker_sets_valid_state(self):
|
||||
"""render_video_worker should never set invalid state (countdown=True but seconds=0).
|
||||
|
||||
This test verifies that the fix in render_video_worker.py sets both
|
||||
has_countdown_padding and countdown_padding_seconds together.
|
||||
"""
|
||||
# Simulate the state update from render_video_worker
|
||||
# When padding_added=True, BOTH fields must be set correctly
|
||||
padding_added = True
|
||||
padding_seconds = 3.0 # Always 3.0 from CountdownProcessor
|
||||
|
||||
state_data = {'lyrics_metadata': {}}
|
||||
|
||||
# This is the expected behavior from render_video_worker
|
||||
if padding_added:
|
||||
state_data['lyrics_metadata']['has_countdown_padding'] = True
|
||||
state_data['lyrics_metadata']['countdown_padding_seconds'] = padding_seconds
|
||||
|
||||
# Verify the invariant holds
|
||||
lyrics_metadata = state_data.get('lyrics_metadata', {})
|
||||
if lyrics_metadata.get('has_countdown_padding'):
|
||||
assert lyrics_metadata.get('countdown_padding_seconds', 0) > 0, (
|
||||
"INVARIANT VIOLATION: render_video_worker must set both "
|
||||
"has_countdown_padding and countdown_padding_seconds together"
|
||||
)
|
||||
|
||||
|
||||
class TestAudioPaddingFunction:
|
||||
"""Tests for the _pad_audio_file function in KaraokeFinalise."""
|
||||
|
||||
def test_pad_audio_file_function_exists(self):
|
||||
"""KaraokeFinalise has _pad_audio_file method."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(non_interactive=True)
|
||||
assert hasattr(finalise, '_pad_audio_file'), (
|
||||
"KaraokeFinalise must have _pad_audio_file method for instrumental padding"
|
||||
)
|
||||
|
||||
def test_remux_with_instrumental_checks_padding(self):
|
||||
"""remux_with_instrumental method handles countdown padding."""
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
|
||||
finalise = KaraokeFinalise(
|
||||
countdown_padding_seconds=3.0,
|
||||
non_interactive=True,
|
||||
)
|
||||
|
||||
# The method should exist and check countdown_padding_seconds
|
||||
assert hasattr(finalise, 'remux_with_instrumental'), (
|
||||
"KaraokeFinalise must have remux_with_instrumental method"
|
||||
)
|
||||
@@ -1128,7 +1128,8 @@ class TestFinaliseOnlyModels:
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (server applies default_enable_youtube_upload)
|
||||
assert request.enable_youtube_upload is None
|
||||
assert request.dropbox_path is None
|
||||
assert request.gdrive_folder_id is None
|
||||
|
||||
@@ -1680,17 +1681,19 @@ class TestUploadEndpointThemeSupport:
|
||||
)
|
||||
|
||||
def test_upload_endpoint_uses_resolve_cdg_txt_defaults(self):
|
||||
"""Verify the upload endpoint uses _resolve_cdg_txt_defaults for theme-based defaults."""
|
||||
"""Verify the upload endpoint uses resolve_cdg_txt_defaults for theme-based defaults."""
|
||||
import inspect
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
source_code = inspect.getsource(file_upload_module)
|
||||
|
||||
has_resolve_call = '_resolve_cdg_txt_defaults(' in source_code
|
||||
# Check for the centralized resolve_cdg_txt_defaults function (imported from job_defaults_service)
|
||||
has_resolve_call = 'resolve_cdg_txt_defaults(' in source_code
|
||||
|
||||
assert has_resolve_call, (
|
||||
"file_upload.py does not call _resolve_cdg_txt_defaults(). "
|
||||
"When theme_id is set, enable_cdg and enable_txt should default to True."
|
||||
"file_upload.py does not call resolve_cdg_txt_defaults(). "
|
||||
"Theme-driven CDG/TXT defaults (controlled by DEFAULT_ENABLE_CDG/DEFAULT_ENABLE_TXT "
|
||||
"settings) must be applied via the centralized job_defaults_service."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_has_optional_cdg_txt_params(self):
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
"""
|
||||
Tests for GCE Encoding Worker (gce_encoding/main.py).
|
||||
|
||||
These tests verify the file-finding logic in the GCE encoding worker,
|
||||
particularly ensuring that instrumental selection is respected.
|
||||
|
||||
NOTE: This test file is self-contained and doesn't import the full app
|
||||
to avoid dependency issues in CI/CD environments.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Copy the find_file function here to test it in isolation
|
||||
# This avoids importing the full gce_encoding module which has GCS dependencies
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
class TestFindFile:
|
||||
"""Test the find_file function used to locate input files."""
|
||||
|
||||
def test_find_file_single_match(self, tmp_path):
|
||||
"""Test finding a file with a single match."""
|
||||
# Create test file
|
||||
test_file = tmp_path / "test_instrumental.flac"
|
||||
test_file.touch()
|
||||
|
||||
result = find_file(tmp_path, "*instrumental*.flac")
|
||||
assert result is not None
|
||||
assert result.name == "test_instrumental.flac"
|
||||
|
||||
def test_find_file_no_match(self, tmp_path):
|
||||
"""Test finding a file with no matches returns None."""
|
||||
result = find_file(tmp_path, "*nonexistent*.flac")
|
||||
assert result is None
|
||||
|
||||
def test_find_file_priority_order(self, tmp_path):
|
||||
"""Test that find_file returns first matching pattern."""
|
||||
# Create multiple files that match different patterns
|
||||
first = tmp_path / "first_pattern.flac"
|
||||
second = tmp_path / "second_pattern.flac"
|
||||
first.touch()
|
||||
second.touch()
|
||||
|
||||
# First pattern should win
|
||||
result = find_file(tmp_path, "*first*.flac", "*second*.flac")
|
||||
assert result.name == "first_pattern.flac"
|
||||
|
||||
|
||||
class TestInstrumentalSelection:
|
||||
"""Test that instrumental selection is properly respected."""
|
||||
|
||||
def test_clean_instrumental_prioritized_when_selected(self, tmp_path):
|
||||
"""Test that clean instrumental is found when 'clean' is selected.
|
||||
|
||||
When both instrumentals exist and 'clean' is selected,
|
||||
the clean version should be found.
|
||||
"""
|
||||
# Create both instrumental files (as they exist in GCS)
|
||||
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
|
||||
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'clean' selection
|
||||
# (mimics the logic in run_encoding when instrumental_selection == 'clean')
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "Clean" in result.name
|
||||
|
||||
def test_backing_instrumental_prioritized_when_selected(self, tmp_path):
|
||||
"""Test that backing instrumental is found when 'with_backing' is selected.
|
||||
|
||||
When both instrumentals exist and 'with_backing' is selected,
|
||||
the backing version should be found.
|
||||
|
||||
This test would have caught the bug where the GCE worker always
|
||||
used the clean instrumental regardless of user selection.
|
||||
"""
|
||||
# Create both instrumental files (as they exist in GCS)
|
||||
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
|
||||
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'with_backing' selection
|
||||
# (mimics the logic in run_encoding when instrumental_selection == 'with_backing')
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "Backing" in result.name
|
||||
|
||||
def test_backing_instrumental_with_stem_naming(self, tmp_path):
|
||||
"""Test finding backing instrumental with GCS stem naming convention.
|
||||
|
||||
In GCS, stems are named like 'stems/instrumental_with_backing.flac'
|
||||
"""
|
||||
stems_dir = tmp_path / "stems"
|
||||
stems_dir.mkdir()
|
||||
|
||||
clean = stems_dir / "instrumental_clean.flac"
|
||||
backing = stems_dir / "instrumental_with_backing.flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'with_backing' selection
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "with_backing" in result.name
|
||||
|
||||
def test_fallback_to_generic_when_specific_not_found(self, tmp_path):
|
||||
"""Test fallback to generic pattern when specific not found."""
|
||||
# Only create a generically named instrumental
|
||||
generic = tmp_path / "instrumental.flac"
|
||||
generic.touch()
|
||||
|
||||
# Both clean and backing selections should fall back
|
||||
for selection in ["clean", "with_backing"]:
|
||||
if selection == "with_backing":
|
||||
patterns = (
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
else:
|
||||
patterns = (
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
result = find_file(tmp_path, *patterns)
|
||||
assert result is not None
|
||||
assert result.name == "instrumental.flac"
|
||||
|
||||
|
||||
class TestInstrumentalSelectionRegression:
|
||||
"""Regression tests for instrumental selection bug.
|
||||
|
||||
Bug: User selected 'with_backing' instrumental in the UI but the final
|
||||
video used the clean instrumental instead.
|
||||
|
||||
Root cause: The GCE worker's find_file() always prioritized
|
||||
'*instrumental_clean*' patterns before generic patterns, ignoring
|
||||
the user's selection.
|
||||
"""
|
||||
|
||||
def test_bug_scenario_both_instrumentals_present(self, tmp_path):
|
||||
"""Recreate the exact bug scenario where both instrumentals exist.
|
||||
|
||||
This is the key regression test. When both instrumentals are downloaded
|
||||
from GCS (which happens with the GCE worker), the correct one must
|
||||
be selected based on the user's choice.
|
||||
"""
|
||||
# Setup: Both instrumentals in stems/ (as downloaded from GCS)
|
||||
stems = tmp_path / "stems"
|
||||
stems.mkdir()
|
||||
|
||||
(stems / "instrumental_clean.flac").touch()
|
||||
(stems / "instrumental_with_backing.flac").touch()
|
||||
|
||||
# Also have properly named versions in the root (from _setup_working_directory)
|
||||
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
|
||||
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
|
||||
|
||||
# When user selected 'with_backing', the backing version MUST be found
|
||||
backing_patterns = (
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
result = find_file(tmp_path, *backing_patterns)
|
||||
|
||||
assert result is not None
|
||||
# Must find a backing-related file, not the clean one
|
||||
name = result.name.lower()
|
||||
assert "backing" in name or "with_backing" in name, \
|
||||
f"Expected backing instrumental but found: {result.name}"
|
||||
|
||||
def test_clean_selection_still_works(self, tmp_path):
|
||||
"""Verify clean selection still works correctly after the fix."""
|
||||
stems = tmp_path / "stems"
|
||||
stems.mkdir()
|
||||
|
||||
(stems / "instrumental_clean.flac").touch()
|
||||
(stems / "instrumental_with_backing.flac").touch()
|
||||
|
||||
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
|
||||
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
|
||||
|
||||
# When user selected 'clean', the clean version must be found
|
||||
clean_patterns = (
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
result = find_file(tmp_path, *clean_patterns)
|
||||
|
||||
assert result is not None
|
||||
name = result.name.lower()
|
||||
assert "clean" in name, f"Expected clean instrumental but found: {result.name}"
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Unit tests for user impersonation endpoint.
|
||||
|
||||
Tests the admin impersonation API that allows admins to view the app as other users.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
from backend.models.user import User, Session
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency - returns admin AuthResult."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin session valid",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
def get_mock_regular_user():
|
||||
"""Override for require_admin dependency - returns regular user (should fail)."""
|
||||
# This simulates what happens when a non-admin tries to access
|
||||
# In reality, require_admin raises 403, but we test the logic
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=5,
|
||||
message="User session valid",
|
||||
user_email="user@example.com",
|
||||
is_admin=False,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client with admin access."""
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
yield TestClient(app)
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_user_service():
|
||||
"""Create a mock user service."""
|
||||
service = Mock()
|
||||
return service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_target_user():
|
||||
"""Create a mock target user to impersonate."""
|
||||
user = Mock(spec=User)
|
||||
user.email = "target@example.com"
|
||||
user.credits = 5
|
||||
user.role = "user"
|
||||
user.is_active = True
|
||||
return user
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_session():
|
||||
"""Create a mock session."""
|
||||
session = Mock(spec=Session)
|
||||
session.token = "test-session-token-12345678901234567890"
|
||||
session.user_email = "target@example.com"
|
||||
return session
|
||||
|
||||
|
||||
class TestImpersonateUser:
|
||||
"""Tests for POST /api/admin/users/{email}/impersonate endpoint."""
|
||||
|
||||
def test_admin_can_impersonate_user(self, client, mock_target_user, mock_session):
|
||||
"""Admin gets session token for target user."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Override the dependency
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_token"] == mock_session.token
|
||||
assert data["user_email"] == "target@example.com"
|
||||
assert "impersonating" in data["message"].lower()
|
||||
|
||||
# Verify user lookup was called
|
||||
mock_service.get_user.assert_called_once_with("target@example.com")
|
||||
|
||||
# Verify session was created for target user
|
||||
mock_service.create_session.assert_called_once()
|
||||
call_args = mock_service.create_session.call_args
|
||||
assert call_args.kwargs["user_email"] == "target@example.com"
|
||||
assert "Impersonation by admin@nomadkaraoke.com" in call_args.kwargs["user_agent"]
|
||||
|
||||
def test_impersonate_nonexistent_user_returns_404(self, client):
|
||||
"""Target user must exist."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = None # User not found
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/nonexistent@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_cannot_impersonate_self(self, client, mock_target_user):
|
||||
"""Admin cannot impersonate themselves."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
# Set up user to be found (same as admin)
|
||||
mock_target_user.email = "admin@nomadkaraoke.com"
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/admin@nomadkaraoke.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "yourself" in response.json()["detail"].lower()
|
||||
|
||||
def test_impersonation_creates_valid_session(self, client, mock_target_user, mock_session):
|
||||
"""Returned token is a valid session for target user."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify create_session was called with correct user
|
||||
mock_service.create_session.assert_called_once()
|
||||
call_kwargs = mock_service.create_session.call_args.kwargs
|
||||
assert call_kwargs["user_email"] == "target@example.com"
|
||||
|
||||
def test_impersonation_logs_audit_trail(self, client, mock_target_user, mock_session):
|
||||
"""Impersonation is logged for security audit."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify logging was called with impersonation info
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "IMPERSONATION" in log_message
|
||||
assert "admin@nomadkaraoke.com" in log_message
|
||||
assert "target@example.com" in log_message
|
||||
|
||||
def test_email_is_normalized_to_lowercase(self, client, mock_target_user, mock_session):
|
||||
"""Email addresses are normalized to lowercase."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/TARGET@EXAMPLE.COM/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify user lookup was with lowercase
|
||||
mock_service.get_user.assert_called_once_with("target@example.com")
|
||||
|
||||
|
||||
class TestImpersonateUserNonAdmin:
|
||||
"""Tests to verify non-admins cannot impersonate."""
|
||||
|
||||
def test_non_admin_cannot_impersonate(self):
|
||||
"""
|
||||
Regular user gets 403 Forbidden.
|
||||
|
||||
Note: This is enforced by the require_admin dependency at the route level.
|
||||
We test that the dependency is correctly applied.
|
||||
"""
|
||||
# Create a fresh client without admin override
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(router, prefix="/api")
|
||||
|
||||
# Don't override require_admin - it should reject non-admin requests
|
||||
# In a real test, we'd mock the auth service to return a non-admin user
|
||||
# For now, we just verify the endpoint requires admin via dependency
|
||||
|
||||
# The require_admin dependency will reject requests without proper admin auth
|
||||
# This is tested implicitly by the fact that all other tests use admin override
|
||||
pass # Actual enforcement tested via integration tests
|
||||
@@ -128,6 +128,8 @@ class TestUserEmailExtraction:
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/jobs/create-from-url")
|
||||
# Mock tenant state (no tenant = default Nomad Karaoke)
|
||||
mock_request.state.tenant_config = None
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = CreateJobFromUrlRequest(
|
||||
@@ -138,13 +140,20 @@ class TestUserEmailExtraction:
|
||||
|
||||
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
|
||||
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
|
||||
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred, \
|
||||
patch('backend.api.routes.file_upload.get_youtube_download_service') as mock_yt:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-456"
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
|
||||
# Mock YouTube download service to return a GCS path
|
||||
from unittest.mock import AsyncMock
|
||||
mock_yt_service = Mock()
|
||||
mock_yt_service.download = AsyncMock(return_value="uploads/test-job-456/audio/test.webm")
|
||||
mock_yt.return_value = mock_yt_service
|
||||
|
||||
await create_job_from_url(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
@@ -168,6 +177,8 @@ class TestUserEmailExtraction:
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/audio-search/search")
|
||||
# Mock tenant state (no tenant = default Nomad Karaoke)
|
||||
mock_request.state.tenant_config = None
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = AudioSearchRequest(
|
||||
@@ -366,12 +377,13 @@ class TestAudioSearchCacheIndependence:
|
||||
assert 'state_data' in source, \
|
||||
"search_audio must use state_data for persistence"
|
||||
|
||||
def test_download_code_has_youtube_direct_download_branch(self):
|
||||
def test_download_code_uses_youtube_download_service(self):
|
||||
"""
|
||||
Verify that _download_and_start_processing has the YouTube direct download path.
|
||||
Verify that _download_and_start_processing uses YouTubeDownloadService.
|
||||
|
||||
This checks that the code path exists for downloading YouTube audio directly
|
||||
using the URL from state_data, avoiding the in-memory cache dependency.
|
||||
After the consolidation refactor (2025-01), all YouTube downloads go through
|
||||
YouTubeDownloadService instead of using download_from_url directly. This
|
||||
ensures consistent handling with remote flacfetch support.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
@@ -379,33 +391,28 @@ class TestAudioSearchCacheIndependence:
|
||||
# Get the source of _download_and_start_processing
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# Verify the YouTube direct download branch exists
|
||||
# Verify YouTubeDownloadService is used for YouTube
|
||||
assert "source_name == 'YouTube'" in source, \
|
||||
"_download_and_start_processing must check for YouTube source"
|
||||
assert 'download_url' in source, \
|
||||
"_download_and_start_processing must use download_url from state_data"
|
||||
assert 'download_from_url' in source, \
|
||||
"_download_and_start_processing must call download_from_url for YouTube"
|
||||
assert 'youtube_service' in source or 'get_youtube_download_service' in source, \
|
||||
"_download_and_start_processing must use YouTubeDownloadService for YouTube"
|
||||
|
||||
def test_youtube_download_branch_avoids_cache_dependency(self):
|
||||
def test_youtube_download_service_import_exists(self):
|
||||
"""
|
||||
Verify that the YouTube download path uses URL from state_data,
|
||||
not the in-memory cache that doesn't persist across Cloud Run instances.
|
||||
Verify that audio_search imports YouTubeDownloadService.
|
||||
|
||||
This ensures the consolidated YouTube download path is properly wired.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
source = inspect.getsource(audio_search)
|
||||
|
||||
# The fix specifically uses selected.get('url') which comes from state_data
|
||||
# rather than audio_search_service cache
|
||||
assert "selected.get('url')" in source or "download_url" in source, \
|
||||
"YouTube download must use URL from selected result (state_data)"
|
||||
|
||||
# Verify it doesn't rely on cache for YouTube
|
||||
# The branch is: elif source_name == 'YouTube' and download_url:
|
||||
assert "YouTube" in source and "download_url" in source, \
|
||||
"YouTube branch must check for download_url availability"
|
||||
# Verify YouTubeDownloadService is imported
|
||||
assert 'get_youtube_download_service' in source, \
|
||||
"audio_search must import get_youtube_download_service"
|
||||
assert 'YouTubeDownloadError' in source, \
|
||||
"audio_search must import YouTubeDownloadError"
|
||||
|
||||
|
||||
class TestTranscriptionTimeout:
|
||||
|
||||
@@ -33,13 +33,28 @@ def job_manager(mock_firestore_service):
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation logic."""
|
||||
|
||||
|
||||
def test_create_job_requires_theme_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that jobs without theme_id are rejected."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
# No theme_id - should fail
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="theme_id is required"):
|
||||
job_manager.create_job(job_create)
|
||||
|
||||
# Verify Firestore was NOT called
|
||||
mock_firestore_service.create_job.assert_not_called()
|
||||
|
||||
def test_create_job_with_url(self, job_manager, mock_firestore_service):
|
||||
"""Test creating a job with YouTube URL."""
|
||||
job_create = JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
# The actual create_job method creates the job and returns it
|
||||
@@ -59,7 +74,8 @@ class TestJobCreation:
|
||||
"""Test creating a job without URL (for file upload)."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
@@ -70,8 +86,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that each job gets a unique ID."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
# Create multiple jobs
|
||||
ids = []
|
||||
for i in range(5):
|
||||
@@ -89,8 +105,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
|
||||
"""Test that new jobs start with PENDING status."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
mock_firestore_service.create_job.return_value = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
@@ -105,13 +121,14 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
|
||||
"""Test that distribution settings are passed from JobCreate to Job.
|
||||
|
||||
|
||||
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
|
||||
discord_webhook_url were NOT being passed to the Job constructor.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad", # Required for all jobs
|
||||
brand_prefix="NOMAD",
|
||||
discord_webhook_url="https://discord.com/webhook/test",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
@@ -296,9 +313,154 @@ class TestJobFailure:
|
||||
assert call_args[1]['error_message'] == error_message
|
||||
|
||||
|
||||
class TestMadeForYouFieldMapping:
|
||||
"""
|
||||
CRITICAL: Test that made-for-you fields are properly copied from JobCreate to Job.
|
||||
|
||||
These tests verify that JobManager.create_job() properly maps all fields from
|
||||
JobCreate to the Job object that gets persisted to Firestore.
|
||||
|
||||
Bug context (2026-01-09): Production made-for-you orders were created with
|
||||
made_for_you=False, customer_email=None, customer_notes=None because
|
||||
JobManager.create_job() wasn't copying these fields from JobCreate to Job.
|
||||
"""
|
||||
|
||||
def test_create_job_copies_made_for_you_flag(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
CRITICAL: made_for_you flag must be copied from JobCreate to Job.
|
||||
|
||||
This flag is essential for:
|
||||
- Ownership transfer on job completion
|
||||
- Email suppression for intermediate states
|
||||
- Identifying made-for-you jobs in admin UI
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True, # This MUST be copied to the Job
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# CRITICAL ASSERTION: made_for_you must be True on the created Job
|
||||
assert job.made_for_you is True, \
|
||||
"made_for_you=True on JobCreate must be copied to Job"
|
||||
|
||||
# Also verify what was saved to Firestore
|
||||
mock_firestore_service.create_job.assert_called_once()
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.made_for_you is True, \
|
||||
"made_for_you=True must be persisted to Firestore"
|
||||
|
||||
def test_create_job_copies_customer_email(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
CRITICAL: customer_email must be copied from JobCreate to Job.
|
||||
|
||||
This field is essential for:
|
||||
- Ownership transfer on job completion (transferring to this email)
|
||||
- Sending completion email with download links to customer
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com", # This MUST be copied to the Job
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# CRITICAL ASSERTION: customer_email must be copied
|
||||
assert job.customer_email == "customer@example.com", \
|
||||
"customer_email on JobCreate must be copied to Job"
|
||||
|
||||
# Also verify what was saved to Firestore
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.customer_email == "customer@example.com", \
|
||||
"customer_email must be persisted to Firestore"
|
||||
|
||||
def test_create_job_copies_customer_notes(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
customer_notes must be copied from JobCreate to Job.
|
||||
|
||||
Customer notes contain special requests that admin needs to see.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
customer_notes="Please make this perfect for my wedding!",
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.customer_notes == "Please make this perfect for my wedding!", \
|
||||
"customer_notes on JobCreate must be copied to Job"
|
||||
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.customer_notes == "Please make this perfect for my wedding!", \
|
||||
"customer_notes must be persisted to Firestore"
|
||||
|
||||
def test_create_job_full_made_for_you_config(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
Test complete made-for-you job creation with all fields.
|
||||
|
||||
This is the realistic scenario: a made-for-you order creates a job with:
|
||||
- made_for_you=True
|
||||
- user_email=admin (owner during processing)
|
||||
- customer_email=customer (for final delivery)
|
||||
- customer_notes=notes (customer's special requests)
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Avril Lavigne",
|
||||
title="Complicated",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Anniversary gift!",
|
||||
# Distribution settings that should also be applied
|
||||
enable_youtube_upload=True,
|
||||
dropbox_path="/Production/Ready",
|
||||
brand_prefix="NOMAD",
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# Verify all made-for-you fields
|
||||
assert job.made_for_you is True
|
||||
assert job.user_email == "admin@nomadkaraoke.com"
|
||||
assert job.customer_email == "customer@example.com"
|
||||
assert job.customer_notes == "Anniversary gift!"
|
||||
|
||||
# Verify distribution settings too
|
||||
assert job.enable_youtube_upload is True
|
||||
assert job.dropbox_path == "/Production/Ready"
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
|
||||
def test_create_job_made_for_you_false_by_default(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
Regular jobs should have made_for_you=False by default.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
# No made_for_you specified - should default to False
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.made_for_you is False
|
||||
assert job.customer_email is None
|
||||
assert job.customer_notes is None
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion logic."""
|
||||
|
||||
|
||||
def test_delete_job(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting a job."""
|
||||
existing_job = Job(
|
||||
@@ -334,6 +496,78 @@ class TestJobDeletion:
|
||||
mock_firestore_service.delete_job.assert_called_once_with("test123")
|
||||
|
||||
|
||||
class TestDeleteStateDataKeys:
|
||||
"""Tests for delete_state_data_key and delete_state_data_keys methods.
|
||||
|
||||
These methods are used to clear worker progress keys from state_data,
|
||||
which is critical for allowing workers to re-run after admin resets.
|
||||
"""
|
||||
|
||||
def test_delete_state_data_key_success(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting a single key from state_data."""
|
||||
# Setup mock to return success
|
||||
mock_firestore_service.db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
|
||||
|
||||
result = job_manager.delete_state_data_key("test123", "render_progress")
|
||||
|
||||
assert result is True
|
||||
# Verify update was called with DELETE_FIELD for the key
|
||||
mock_job_ref.update.assert_called_once()
|
||||
call_args = mock_job_ref.update.call_args[0][0]
|
||||
assert "state_data.render_progress" in call_args
|
||||
|
||||
def test_delete_state_data_key_handles_exception(self, job_manager, mock_firestore_service):
|
||||
"""Test that delete_state_data_key handles exceptions gracefully."""
|
||||
mock_firestore_service.db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_job_ref.update.side_effect = Exception("Firestore error")
|
||||
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
|
||||
|
||||
result = job_manager.delete_state_data_key("test123", "render_progress")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_delete_state_data_keys_success(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting multiple keys from state_data in one operation."""
|
||||
mock_firestore_service.db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
|
||||
|
||||
keys_to_delete = ["render_progress", "video_progress", "encoding_progress"]
|
||||
result = job_manager.delete_state_data_keys("test123", keys_to_delete)
|
||||
|
||||
assert result == keys_to_delete
|
||||
# Verify single update was called with all keys
|
||||
mock_job_ref.update.assert_called_once()
|
||||
call_args = mock_job_ref.update.call_args[0][0]
|
||||
assert "state_data.render_progress" in call_args
|
||||
assert "state_data.video_progress" in call_args
|
||||
assert "state_data.encoding_progress" in call_args
|
||||
|
||||
def test_delete_state_data_keys_empty_list(self, job_manager, mock_firestore_service):
|
||||
"""Test that empty list returns empty without calling Firestore."""
|
||||
mock_firestore_service.db = Mock()
|
||||
|
||||
result = job_manager.delete_state_data_keys("test123", [])
|
||||
|
||||
assert result == []
|
||||
# Should NOT call Firestore for empty list
|
||||
mock_firestore_service.db.collection.assert_not_called()
|
||||
|
||||
def test_delete_state_data_keys_handles_exception(self, job_manager, mock_firestore_service):
|
||||
"""Test that delete_state_data_keys handles exceptions gracefully."""
|
||||
mock_firestore_service.db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_job_ref.update.side_effect = Exception("Firestore error")
|
||||
mock_firestore_service.db.collection.return_value.document.return_value = mock_job_ref
|
||||
|
||||
result = job_manager.delete_state_data_keys("test123", ["render_progress"])
|
||||
|
||||
assert result == []
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
"""
|
||||
Unit tests for job manager notification methods.
|
||||
|
||||
Tests the state transition triggers for email notifications.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestTriggerStateNotifications:
|
||||
"""Tests for _trigger_state_notifications method."""
|
||||
|
||||
def test_triggers_completion_email_on_complete(self):
|
||||
"""Test that completion email is triggered when job completes."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_review(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting review."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications(
|
||||
"job-123",
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
mock_schedule.assert_called_once_with(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
def test_skips_notification_when_no_job(self):
|
||||
"""Test that notification is skipped when job not found."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(return_value=None)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_skips_notification_when_no_user_email(self):
|
||||
"""Test that notification is skipped when job has no user email."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = None
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_does_not_trigger_on_other_states(self):
|
||||
"""Test that no notification is triggered for non-notification states."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
|
||||
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
|
||||
|
||||
# Test various non-notification states
|
||||
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
|
||||
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
|
||||
manager._trigger_state_notifications("job-123", status)
|
||||
|
||||
mock_complete.assert_not_called()
|
||||
mock_idle.assert_not_called()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate from notifications."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(side_effect=Exception("Database error"))
|
||||
|
||||
# Should not raise
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
|
||||
class TestScheduleCompletionEmail:
|
||||
"""Tests for _schedule_completion_email method."""
|
||||
|
||||
def test_extracts_urls_from_state_data(self):
|
||||
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = "Test Artist"
|
||||
mock_job.title = "Test Song"
|
||||
mock_job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=123",
|
||||
"dropbox_link": "https://dropbox.com/folder/abc",
|
||||
}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Run in sync context - this will start a daemon thread
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
# Give the thread a moment to execute
|
||||
import time
|
||||
time.sleep(0.1)
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
mock_job.state_data = None # None state_data
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_get_service.side_effect = Exception("Service error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
|
||||
class TestScheduleIdleReminder:
|
||||
"""Tests for _schedule_idle_reminder method."""
|
||||
|
||||
def test_updates_state_data_for_lyrics_review(self):
|
||||
"""Test that state_data is updated with blocking state info for lyrics."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"existing_key": "value"}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
assert update_call is not None
|
||||
updated_data = update_call[0][1] # Second positional arg
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'lyrics'
|
||||
assert state_data.get('reminder_sent') is False
|
||||
assert 'blocking_state_entered_at' in state_data
|
||||
assert state_data.get('existing_key') == 'value' # Preserved
|
||||
|
||||
def test_updates_state_data_for_instrumental_selection(self):
|
||||
"""Test that state_data is updated with blocking state info for instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
updated_data = update_call[0][1]
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'instrumental'
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify update was still called
|
||||
manager.firestore.update_job.assert_called_once()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
# Make firestore raise an exception
|
||||
manager.firestore.update_job.side_effect = Exception("Database error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
|
||||
class TestTransitionTriggersNotifications:
|
||||
"""Integration tests for transition_to_state triggering notifications."""
|
||||
|
||||
def test_transition_to_complete_triggers_notification(self):
|
||||
"""Test that transitioning to COMPLETE triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
|
||||
mock_job.status = JobStatus.ENCODING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
|
||||
|
||||
def test_transition_to_awaiting_review_triggers_notification(self):
|
||||
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
|
||||
mock_job.status = JobStatus.APPLYING_PADDING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
|
||||
@@ -0,0 +1,409 @@
|
||||
"""
|
||||
Unit tests for job notification service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, AsyncMock
|
||||
|
||||
from backend.services.job_notification_service import (
|
||||
JobNotificationService,
|
||||
get_job_notification_service,
|
||||
ENABLE_AUTO_EMAILS,
|
||||
FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
class TestURLBuilding:
|
||||
"""Tests for URL building methods."""
|
||||
|
||||
def test_build_review_url_basic(self):
|
||||
"""Test basic review URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123")
|
||||
|
||||
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
|
||||
|
||||
def test_build_review_url_ignores_legacy_params(self):
|
||||
"""Test review URL ignores legacy audio_hash and review_token params."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
|
||||
# Legacy params are still accepted but not used in the URL
|
||||
url = service._build_review_url("job-123", audio_hash="abc123", review_token="token456")
|
||||
|
||||
# URL should be the simple consolidated route
|
||||
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
|
||||
# No query params
|
||||
assert "?" not in url
|
||||
|
||||
def test_build_review_url_preserves_job_id_characters(self):
|
||||
"""Test that job ID is used directly in URL path."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
|
||||
# Note: job IDs with slashes would be unusual in real usage
|
||||
url = service._build_review_url("abc-def-123")
|
||||
|
||||
assert url == "https://gen.nomadkaraoke.com/app/jobs#/abc-def-123/review"
|
||||
|
||||
def test_build_instrumental_url_basic(self):
|
||||
"""Test basic instrumental URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123")
|
||||
|
||||
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
|
||||
|
||||
def test_build_instrumental_url_ignores_legacy_params(self):
|
||||
"""Test instrumental URL ignores legacy instrumental_token param."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
|
||||
# Legacy param is still accepted but not used in the URL
|
||||
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
|
||||
|
||||
# URL should be the simple consolidated route
|
||||
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
|
||||
# No query params
|
||||
assert "?" not in url
|
||||
|
||||
|
||||
class TestCompletionEmail:
|
||||
"""Tests for job completion email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_success(self):
|
||||
"""Test successful completion email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
service.email_service.send_job_completion.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_disabled(self):
|
||||
"""Test that completion email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_no_email(self):
|
||||
"""Test that completion email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_empty_email(self):
|
||||
"""Test that completion email is skipped when user email is empty."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_send_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = False
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_exception(self):
|
||||
"""Test handling of exceptions during email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.side_effect = Exception("Send error")
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_passes_cc_admin(self):
|
||||
"""Test that completion email is sent with CC to admin."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Verify cc_admin=True was passed
|
||||
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('cc_admin') is True
|
||||
|
||||
|
||||
class TestActionReminderEmail:
|
||||
"""Tests for action reminder email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_success(self):
|
||||
"""Test successful lyrics reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_lyrics.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_success(self):
|
||||
"""Test successful instrumental reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_instrumental.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_disabled(self):
|
||||
"""Test that reminder email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_no_email(self):
|
||||
"""Test that reminder email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_unknown_action_type(self):
|
||||
"""Test handling of unknown action type."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_includes_review_url(self):
|
||||
"""Test that lyrics reminder includes correct review URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
# Verify the review URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
|
||||
review_url = call_kwargs.get('review_url')
|
||||
assert review_url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_includes_url(self):
|
||||
"""Test that instrumental reminder includes correct URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
# Verify the instrumental URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
|
||||
instrumental_url = call_kwargs.get('instrumental_url')
|
||||
assert instrumental_url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for get_completion_message method."""
|
||||
|
||||
def test_get_completion_message_basic(self):
|
||||
"""Test basic completion message retrieval."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Your video is ready!"
|
||||
|
||||
result = service.get_completion_message(
|
||||
job_id="job-123",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result == "Your video is ready!"
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
|
||||
def test_get_completion_message_includes_feedback_url(self):
|
||||
"""Test that completion message includes feedback URL."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
# Verify feedback URL was passed
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert 'feedback_url' in call_kwargs
|
||||
|
||||
def test_get_completion_message_with_defaults(self):
|
||||
"""Test completion message with default values."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('job_id') == "job-123"
|
||||
# Other params should be None/default
|
||||
assert call_kwargs.get('name') is None
|
||||
assert call_kwargs.get('artist') is None
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_job_notification_service_returns_same_instance(self):
|
||||
"""Test that get_job_notification_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.job_notification_service as jns
|
||||
jns._job_notification_service = None
|
||||
|
||||
service1 = get_job_notification_service()
|
||||
service2 = get_job_notification_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
def test_service_initializes_with_dependencies(self):
|
||||
"""Test that service initializes with email and template services."""
|
||||
service = JobNotificationService()
|
||||
|
||||
assert service.email_service is not None
|
||||
assert service.template_service is not None
|
||||
assert service.frontend_url is not None
|
||||
assert service.backend_url is not None
|
||||
@@ -50,7 +50,16 @@ def mock_worker_service():
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(mock_job_manager, mock_worker_service):
|
||||
def mock_theme_service():
|
||||
"""Create a mock ThemeService that returns 'nomad' as default theme."""
|
||||
service = MagicMock()
|
||||
service.get_default_theme_id.return_value = "nomad"
|
||||
service.get_theme.return_value = None
|
||||
return service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(mock_job_manager, mock_worker_service, mock_theme_service):
|
||||
"""Create TestClient with mocked dependencies."""
|
||||
mock_creds = MagicMock()
|
||||
mock_creds.universe_domain = 'googleapis.com'
|
||||
@@ -63,6 +72,7 @@ def client(mock_job_manager, mock_worker_service):
|
||||
# Also patch JobManager class used in dependencies.py for auth checks
|
||||
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
|
||||
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
|
||||
patch('backend.api.routes.jobs.get_theme_service', return_value=mock_theme_service), \
|
||||
patch('backend.services.job_manager.JobManager', mock_job_manager_factory), \
|
||||
patch('backend.services.firestore_service.firestore'), \
|
||||
patch('backend.services.storage_service.storage'), \
|
||||
|
||||
@@ -421,3 +421,199 @@ class TestGetLocalEncodingService:
|
||||
service = get_local_encoding_service(dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
|
||||
|
||||
class TestLocalEncodingServiceCountdownPadding:
|
||||
"""Tests for countdown padding in LocalEncodingService."""
|
||||
|
||||
def test_encoding_config_accepts_countdown_padding(self):
|
||||
"""EncodingConfig dataclass accepts countdown_padding_seconds."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
countdown_padding_seconds=3.0,
|
||||
)
|
||||
assert config.countdown_padding_seconds == 3.0
|
||||
|
||||
def test_encoding_config_countdown_padding_default_none(self):
|
||||
"""EncodingConfig countdown_padding_seconds defaults to None."""
|
||||
config = EncodingConfig(
|
||||
title_video="/path/title.mov",
|
||||
karaoke_video="/path/karaoke.mov",
|
||||
instrumental_audio="/path/audio.flac",
|
||||
)
|
||||
assert config.countdown_padding_seconds is None
|
||||
|
||||
def test_pad_audio_file_method_exists(self):
|
||||
"""LocalEncodingService has _pad_audio_file method."""
|
||||
service = LocalEncodingService()
|
||||
assert hasattr(service, "_pad_audio_file")
|
||||
assert callable(service._pad_audio_file)
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_pad_audio_file_calls_ffmpeg_with_adelay(self, mock_run):
|
||||
"""_pad_audio_file uses ffmpeg adelay filter."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalEncodingService()
|
||||
result = service._pad_audio_file(
|
||||
"/input/audio.flac",
|
||||
"/output/audio_padded.flac",
|
||||
3.0
|
||||
)
|
||||
|
||||
assert result == "/output/audio_padded.flac"
|
||||
mock_run.assert_called_once()
|
||||
call_args = mock_run.call_args
|
||||
cmd = call_args[0][0]
|
||||
assert "ffmpeg" in cmd
|
||||
assert "-af" in cmd
|
||||
# adelay should be 3000ms (3.0 * 1000) for both channels
|
||||
af_index = cmd.index("-af")
|
||||
assert "adelay=3000|3000" in cmd[af_index + 1]
|
||||
|
||||
def test_pad_audio_file_dry_run_skips_execution(self):
|
||||
"""_pad_audio_file in dry run mode doesn't execute ffmpeg."""
|
||||
service = LocalEncodingService(dry_run=True)
|
||||
result = service._pad_audio_file(
|
||||
"/input/audio.flac",
|
||||
"/output/audio_padded.flac",
|
||||
3.0
|
||||
)
|
||||
# Should return output path without error (no subprocess called)
|
||||
assert result == "/output/audio_padded.flac"
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_pad_audio_file_raises_on_ffmpeg_error(self, mock_run):
|
||||
"""_pad_audio_file raises RuntimeError on FFmpeg failure."""
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="ffmpeg: error processing audio")
|
||||
|
||||
service = LocalEncodingService()
|
||||
with pytest.raises(RuntimeError, match="FFmpeg padding failed"):
|
||||
service._pad_audio_file("/input/audio.flac", "/output/audio_padded.flac", 3.0)
|
||||
|
||||
@patch.object(LocalEncodingService, "_pad_audio_file")
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
@patch("os.path.exists")
|
||||
def test_encode_all_formats_pads_when_countdown_set(
|
||||
self, mock_exists, mock_lossless, mock_remux, mock_pad
|
||||
):
|
||||
"""encode_all_formats pads instrumental when countdown_padding_seconds > 0."""
|
||||
mock_exists.return_value = False # Padded file doesn't exist
|
||||
mock_pad.return_value = "/output/instrumental (Padded).flac"
|
||||
mock_remux.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
countdown_padding_seconds=3.0,
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
# Should have called _pad_audio_file with 3.0 seconds
|
||||
mock_pad.assert_called_once()
|
||||
call_args = mock_pad.call_args
|
||||
assert call_args[0][0] == "/input/instrumental.flac"
|
||||
assert call_args[0][2] == 3.0
|
||||
|
||||
# remux should use the padded path
|
||||
mock_remux.assert_called_once()
|
||||
remux_args = mock_remux.call_args
|
||||
assert "(Padded)" in remux_args[0][1]
|
||||
|
||||
@patch.object(LocalEncodingService, "_pad_audio_file")
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
def test_encode_all_formats_skips_padding_when_no_countdown(
|
||||
self, mock_lossless, mock_remux, mock_pad
|
||||
):
|
||||
"""encode_all_formats skips padding when countdown_padding_seconds is None."""
|
||||
mock_remux.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
# No countdown_padding_seconds
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
# Should NOT have called _pad_audio_file
|
||||
mock_pad.assert_not_called()
|
||||
|
||||
# remux should use the original path
|
||||
mock_remux.assert_called_once()
|
||||
remux_args = mock_remux.call_args
|
||||
assert remux_args[0][1] == "/input/instrumental.flac"
|
||||
|
||||
@patch.object(LocalEncodingService, "_pad_audio_file")
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
def test_encode_all_formats_skips_already_padded(
|
||||
self, mock_lossless, mock_remux, mock_pad
|
||||
):
|
||||
"""encode_all_formats skips padding if file has (Padded) in name."""
|
||||
mock_remux.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental (Padded).flac", # Already padded
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
countdown_padding_seconds=3.0,
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
# Should NOT have called _pad_audio_file (already padded)
|
||||
mock_pad.assert_not_called()
|
||||
|
||||
# remux should use the original path (which is already padded)
|
||||
mock_remux.assert_called_once()
|
||||
remux_args = mock_remux.call_args
|
||||
assert remux_args[0][1] == "/input/instrumental (Padded).flac"
|
||||
|
||||
@patch.object(LocalEncodingService, "_pad_audio_file")
|
||||
@patch.object(LocalEncodingService, "remux_with_instrumental")
|
||||
@patch.object(LocalEncodingService, "encode_lossless_mp4")
|
||||
def test_encode_all_formats_skips_padding_when_zero(
|
||||
self, mock_lossless, mock_remux, mock_pad
|
||||
):
|
||||
"""encode_all_formats skips padding when countdown_padding_seconds is 0."""
|
||||
mock_remux.return_value = True
|
||||
mock_lossless.return_value = True
|
||||
|
||||
service = LocalEncodingService()
|
||||
config = EncodingConfig(
|
||||
title_video="/input/title.mov",
|
||||
karaoke_video="/input/karaoke.mov",
|
||||
instrumental_audio="/input/instrumental.flac",
|
||||
output_karaoke_mp4="/output/karaoke.mp4",
|
||||
output_lossless_4k_mp4="/output/lossless_4k.mp4",
|
||||
countdown_padding_seconds=0, # Zero, should skip padding
|
||||
)
|
||||
|
||||
result = service.encode_all_formats(config)
|
||||
|
||||
assert result.success is True
|
||||
# Should NOT have called _pad_audio_file (zero padding)
|
||||
mock_pad.assert_not_called()
|
||||
@@ -0,0 +1,567 @@
|
||||
"""
|
||||
Tests for LocalPreviewEncodingService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Hardware acceleration detection (NVENC)
|
||||
- FFmpeg filter path escaping
|
||||
- ASS filter building
|
||||
- Preview FFmpeg command construction
|
||||
- Preview encoding execution
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.local_preview_encoding_service import (
|
||||
LocalPreviewEncodingService,
|
||||
PreviewEncodingConfig,
|
||||
PreviewEncodingResult,
|
||||
get_local_preview_encoding_service,
|
||||
)
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_default_values(self):
|
||||
"""Test default initialization."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service._nvenc_available is None # Lazy detection
|
||||
assert service._video_encoder is None # Lazy detection
|
||||
assert service._hwaccel_flags is None # Lazy detection
|
||||
|
||||
def test_init_with_custom_logger(self):
|
||||
"""Test initialization with custom logger."""
|
||||
import logging
|
||||
custom_logger = logging.getLogger("test")
|
||||
service = LocalPreviewEncodingService(logger=custom_logger)
|
||||
assert service.logger is custom_logger
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceHWAccel:
|
||||
"""Test hardware acceleration detection."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_available(self, mock_run):
|
||||
"""Test NVENC detection when available."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is True
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_not_available(self, mock_run):
|
||||
"""Test fallback when NVENC not available."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_detect_nvenc_timeout(self, mock_run):
|
||||
"""Test NVENC detection timeout handling."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._detect_nvenc_support()
|
||||
|
||||
assert result is False
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_nvenc_available(self, mock_detect):
|
||||
"""Test configuration when NVENC is available."""
|
||||
mock_detect.return_value = True
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is True
|
||||
assert service._video_encoder == "h264_nvenc"
|
||||
assert "-hwaccel" in service._hwaccel_flags
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
|
||||
def test_configure_no_nvenc(self, mock_detect):
|
||||
"""Test configuration when NVENC not available."""
|
||||
mock_detect.return_value = False
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._configure_hardware_acceleration()
|
||||
|
||||
assert service._nvenc_available is False
|
||||
assert service._video_encoder == "libx264"
|
||||
assert service._hwaccel_flags == []
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceProperties:
|
||||
"""Test property accessors that trigger lazy detection."""
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_nvenc_available_triggers_detection(self, mock_config):
|
||||
"""Test that nvenc_available property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = None
|
||||
|
||||
# Access property
|
||||
_ = service.nvenc_available
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_video_encoder_triggers_detection(self, mock_config):
|
||||
"""Test that video_encoder property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._video_encoder = None
|
||||
|
||||
# Access property
|
||||
_ = service.video_encoder
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
|
||||
def test_hwaccel_flags_triggers_detection(self, mock_config):
|
||||
"""Test that hwaccel_flags property triggers detection."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._hwaccel_flags = None
|
||||
|
||||
# Access property
|
||||
_ = service.hwaccel_flags
|
||||
|
||||
mock_config.assert_called_once()
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServicePathEscaping:
|
||||
"""Test FFmpeg filter path escaping."""
|
||||
|
||||
def test_escape_simple_path(self):
|
||||
"""Test escaping a simple path."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/simple/path.ass")
|
||||
assert result == "/simple/path.ass"
|
||||
|
||||
def test_escape_path_with_spaces(self):
|
||||
"""Test escaping path with spaces."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/with spaces/file.ass")
|
||||
assert "\\ " in result # Spaces should be escaped
|
||||
|
||||
def test_escape_path_with_apostrophe(self):
|
||||
"""Test escaping path with apostrophe (common in song titles)."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path/I'm With You/file.ass")
|
||||
assert "\\\\\\'" in result # Apostrophe should be triple-backslash escaped
|
||||
|
||||
def test_escape_path_with_special_chars(self):
|
||||
"""Test escaping path with FFmpeg special characters."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path:with[special];chars,here.ass")
|
||||
# Should escape :,[];
|
||||
assert "\\:" in result
|
||||
assert "\\[" in result
|
||||
assert "\\]" in result
|
||||
assert "\\;" in result
|
||||
assert "\\," in result
|
||||
|
||||
def test_escape_path_with_backslashes(self):
|
||||
"""Test escaping path with existing backslashes."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._escape_ffmpeg_filter_path("/path\\with\\backslashes.ass")
|
||||
assert "\\\\" in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceASSFilter:
|
||||
"""Test ASS filter building."""
|
||||
|
||||
def test_build_ass_filter_simple(self):
|
||||
"""Test building ASS filter without font."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass")
|
||||
assert result.startswith("ass=")
|
||||
assert "fontsdir" not in result
|
||||
|
||||
def test_build_ass_filter_with_font(self):
|
||||
"""Test building ASS filter with custom font."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
font_path = os.path.join(tmpdir, "custom.ttf")
|
||||
with open(font_path, "w") as f:
|
||||
f.write("fake font")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", font_path)
|
||||
|
||||
assert "fontsdir=" in result
|
||||
|
||||
def test_build_ass_filter_nonexistent_font(self):
|
||||
"""Test building ASS filter with nonexistent font file."""
|
||||
service = LocalPreviewEncodingService()
|
||||
result = service._build_ass_filter("/path/to/file.ass", "/nonexistent/font.ttf")
|
||||
# Should not include fontsdir for nonexistent font
|
||||
assert "fontsdir" not in result
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceFFmpegCommand:
|
||||
"""Test FFmpeg command building."""
|
||||
|
||||
def test_build_command_solid_background_libx264(self):
|
||||
"""Test command building with solid background and libx264."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_color="black"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "ffmpeg" in cmd
|
||||
assert "-r" in cmd and "24" in cmd # Frame rate
|
||||
assert "-c:v" in cmd
|
||||
assert "libx264" in cmd
|
||||
assert "-preset" in cmd and "superfast" in cmd
|
||||
assert "-crf" in cmd and "28" in cmd
|
||||
assert "color=c=black" in " ".join(cmd)
|
||||
|
||||
def test_build_command_with_nvenc(self):
|
||||
"""Test command building with NVENC hardware acceleration."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = True
|
||||
service._video_encoder = "h264_nvenc"
|
||||
service._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "h264_nvenc" in cmd
|
||||
assert "-hwaccel" in cmd
|
||||
assert "-preset" in cmd and "p1" in cmd # Fastest NVENC preset
|
||||
|
||||
def test_build_command_with_background_image(self):
|
||||
"""Test command building with background image."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
bg_path = os.path.join(tmpdir, "background.png")
|
||||
with open(bg_path, "wb") as f:
|
||||
f.write(b"fake png")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path=bg_path
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-loop" in cmd
|
||||
assert "1" in cmd
|
||||
assert bg_path in cmd
|
||||
# Should have scale filter for background image
|
||||
vf_index = cmd.index("-vf")
|
||||
vf_value = cmd[vf_index + 1]
|
||||
assert "scale=" in vf_value
|
||||
assert "pad=" in vf_value
|
||||
|
||||
def test_build_command_resolution(self):
|
||||
"""Test that command uses correct preview resolution."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
cmd_str = " ".join(cmd)
|
||||
|
||||
# Resolution should be 480x270
|
||||
assert "480" in cmd_str
|
||||
assert "270" in cmd_str
|
||||
|
||||
def test_build_command_audio_settings(self):
|
||||
"""Test that command uses correct audio settings."""
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
|
||||
cmd = service._build_preview_ffmpeg_command(config)
|
||||
|
||||
assert "-c:a" in cmd
|
||||
assert "aac" in cmd
|
||||
assert "-b:a" in cmd
|
||||
assert "96k" in cmd
|
||||
|
||||
|
||||
class TestLocalPreviewEncodingServiceEncode:
|
||||
"""Test preview encoding execution."""
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_success(self, mock_run):
|
||||
"""Test successful preview encoding."""
|
||||
mock_run.return_value = MagicMock(returncode=0, stderr="")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
# Create input files
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is True
|
||||
assert result.output_path == output_path
|
||||
assert result.error is None
|
||||
mock_run.assert_called_once()
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_ffmpeg_failure(self, mock_run):
|
||||
"""Test preview encoding with FFmpeg failure."""
|
||||
mock_run.return_value = MagicMock(returncode=1, stderr="FFmpeg error message")
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "FFmpeg preview encoding failed" in result.error
|
||||
|
||||
def test_encode_preview_missing_ass_file(self):
|
||||
"""Test preview encoding with missing ASS file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/nonexistent/subs.ass",
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
def test_encode_preview_missing_audio_file(self):
|
||||
"""Test preview encoding with missing audio file."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path="/nonexistent/audio.flac",
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "not found" in result.error
|
||||
|
||||
@patch("subprocess.run")
|
||||
def test_encode_preview_timeout(self, mock_run):
|
||||
"""Test preview encoding timeout."""
|
||||
import subprocess
|
||||
mock_run.side_effect = subprocess.TimeoutExpired("ffmpeg", 300)
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ass_path = os.path.join(tmpdir, "subs.ass")
|
||||
audio_path = os.path.join(tmpdir, "audio.flac")
|
||||
output_path = os.path.join(tmpdir, "output.mp4")
|
||||
|
||||
with open(ass_path, "w") as f:
|
||||
f.write("[Script Info]\n")
|
||||
with open(audio_path, "wb") as f:
|
||||
f.write(b"fake audio")
|
||||
|
||||
service = LocalPreviewEncodingService()
|
||||
service._nvenc_available = False
|
||||
service._video_encoder = "libx264"
|
||||
service._hwaccel_flags = []
|
||||
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path=ass_path,
|
||||
audio_path=audio_path,
|
||||
output_path=output_path
|
||||
)
|
||||
|
||||
result = service.encode_preview(config)
|
||||
|
||||
assert result.success is False
|
||||
assert "timed out" in result.error
|
||||
|
||||
|
||||
class TestPreviewEncodingConfig:
|
||||
"""Test PreviewEncodingConfig dataclass."""
|
||||
|
||||
def test_config_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert config.ass_path == "/path/to/subs.ass"
|
||||
assert config.background_color == "black" # Default
|
||||
assert config.background_image_path is None # Optional
|
||||
|
||||
def test_config_all_fields(self):
|
||||
"""Test config with all fields."""
|
||||
config = PreviewEncodingConfig(
|
||||
ass_path="/path/to/subs.ass",
|
||||
audio_path="/path/to/audio.flac",
|
||||
output_path="/path/to/output.mp4",
|
||||
background_image_path="/path/to/bg.png",
|
||||
background_color="red",
|
||||
font_path="/path/to/font.ttf"
|
||||
)
|
||||
assert config.background_image_path == "/path/to/bg.png"
|
||||
assert config.background_color == "red"
|
||||
assert config.font_path == "/path/to/font.ttf"
|
||||
|
||||
|
||||
class TestPreviewEncodingResult:
|
||||
"""Test PreviewEncodingResult dataclass."""
|
||||
|
||||
def test_result_success(self):
|
||||
"""Test successful result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path="/path/to/output.mp4"
|
||||
)
|
||||
assert result.success is True
|
||||
assert result.error is None
|
||||
assert result.output_path == "/path/to/output.mp4"
|
||||
|
||||
def test_result_failure(self):
|
||||
"""Test failure result."""
|
||||
result = PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Something went wrong"
|
||||
)
|
||||
assert result.success is False
|
||||
assert result.error == "Something went wrong"
|
||||
assert result.output_path is None
|
||||
|
||||
|
||||
class TestGetLocalPreviewEncodingService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service = get_local_preview_encoding_service()
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, LocalPreviewEncodingService)
|
||||
|
||||
def test_get_service_returns_singleton(self):
|
||||
"""Test that factory function returns the same instance."""
|
||||
import backend.services.local_preview_encoding_service as module
|
||||
module._local_preview_encoding_service = None
|
||||
|
||||
service1 = get_local_preview_encoding_service()
|
||||
service2 = get_local_preview_encoding_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestPreviewEncodingConstants:
|
||||
"""Test that encoding constants match expected values."""
|
||||
|
||||
def test_preview_resolution(self):
|
||||
"""Test preview resolution constants."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_WIDTH == 480
|
||||
assert service.PREVIEW_HEIGHT == 270
|
||||
|
||||
def test_preview_fps(self):
|
||||
"""Test preview frame rate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_FPS == 24
|
||||
|
||||
def test_preview_audio_bitrate(self):
|
||||
"""Test preview audio bitrate constant."""
|
||||
service = LocalPreviewEncodingService()
|
||||
assert service.PREVIEW_AUDIO_BITRATE == "96k"
|
||||
File diff suppressed because it is too large.
Load diff
@@ -913,6 +913,145 @@ class TestJobWithFullTwoPhaseConfig:
|
||||
assert "title" in job.file_urls["screens"]
|
||||
|
||||
|
||||
class TestMadeForYouFields:
|
||||
"""Tests for made-for-you order tracking fields."""
|
||||
|
||||
def test_job_has_made_for_you_field(self):
|
||||
"""Test Job model has made_for_you field with default False."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
assert hasattr(job, 'made_for_you')
|
||||
assert job.made_for_you is False
|
||||
|
||||
def test_job_made_for_you_true(self):
|
||||
"""Test Job model can set made_for_you to True."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
made_for_you=True,
|
||||
)
|
||||
assert job.made_for_you is True
|
||||
|
||||
def test_job_has_customer_email_field(self):
|
||||
"""Test Job model has customer_email field with default None."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
assert hasattr(job, 'customer_email')
|
||||
assert job.customer_email is None
|
||||
|
||||
def test_job_customer_email_set(self):
|
||||
"""Test Job model can set customer_email."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
customer_email="customer@example.com",
|
||||
)
|
||||
assert job.customer_email == "customer@example.com"
|
||||
|
||||
def test_job_has_customer_notes_field(self):
|
||||
"""Test Job model has customer_notes field with default None."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
)
|
||||
assert hasattr(job, 'customer_notes')
|
||||
assert job.customer_notes is None
|
||||
|
||||
def test_job_customer_notes_set(self):
|
||||
"""Test Job model can set customer_notes."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
customer_notes="Please make it extra special!",
|
||||
)
|
||||
assert job.customer_notes == "Please make it extra special!"
|
||||
|
||||
def test_job_create_has_made_for_you_fields(self):
|
||||
"""Test JobCreate model has all made-for-you fields."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
assert hasattr(job_create, 'made_for_you')
|
||||
assert hasattr(job_create, 'customer_email')
|
||||
assert hasattr(job_create, 'customer_notes')
|
||||
assert job_create.made_for_you is False
|
||||
assert job_create.customer_email is None
|
||||
assert job_create.customer_notes is None
|
||||
|
||||
def test_job_create_with_made_for_you_config(self):
|
||||
"""Test JobCreate with full made-for-you configuration."""
|
||||
job_create = JobCreate(
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Wedding anniversary!",
|
||||
)
|
||||
assert job_create.made_for_you is True
|
||||
assert job_create.customer_email == "customer@example.com"
|
||||
assert job_create.customer_notes == "Wedding anniversary!"
|
||||
assert job_create.user_email == "admin@nomadkaraoke.com"
|
||||
|
||||
def test_made_for_you_serialization_roundtrip(self):
|
||||
"""Test made-for-you fields survive dict serialization."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Test notes",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
)
|
||||
|
||||
job_dict = job.model_dump()
|
||||
|
||||
assert job_dict['made_for_you'] is True
|
||||
assert job_dict['customer_email'] == "customer@example.com"
|
||||
assert job_dict['customer_notes'] == "Test notes"
|
||||
|
||||
def test_made_for_you_job_with_distribution_settings(self):
|
||||
"""Test made-for-you job with distribution settings."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
created_at=datetime.now(UTC),
|
||||
updated_at=datetime.now(UTC),
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
enable_youtube_upload=True,
|
||||
dropbox_path="/Production/Ready To Upload",
|
||||
gdrive_folder_id="1ABC123",
|
||||
brand_prefix="NOMAD",
|
||||
)
|
||||
|
||||
assert job.made_for_you is True
|
||||
assert job.enable_youtube_upload is True
|
||||
assert job.dropbox_path == "/Production/Ready To Upload"
|
||||
assert job.gdrive_folder_id == "1ABC123"
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
|
||||
@@ -201,7 +201,7 @@ class TestPackagingServiceCDGPackage:
|
||||
assert result[1] == mp3_file
|
||||
assert result[2] == cdg_file
|
||||
|
||||
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
|
||||
@patch("karaoke_gen.lyrics_transcriber.output.cdg.CDGGenerator")
|
||||
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
|
||||
"""Test CDG package creation using CDGGenerator."""
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
|
||||
Loaded 100 of 1022 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user