mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
382
Commits
No files matched your search
@@ -0,0 +1,393 @@
|
||||
# Plan: Admin Job Details Page
|
||||
|
||||
**Created:** 2026-01-09
|
||||
**Branch:** feat/sess-20260109-1738-admin-job-details
|
||||
**Status:** In Progress (Phase 1 Complete)
|
||||
|
||||
## Overview
|
||||
|
||||
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
|
||||
|
||||
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
|
||||
|
||||
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Must Have
|
||||
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
|
||||
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
|
||||
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
|
||||
- [ ] Download links for all existing files (generate signed URLs)
|
||||
- [ ] View worker logs with filtering by level/worker
|
||||
- [ ] Restart job from beginning (reset to PENDING)
|
||||
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
|
||||
|
||||
### Should Have
|
||||
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
|
||||
- [ ] View/edit raw JSON fields (state_data, file_urls)
|
||||
- [ ] Quick actions: send completion email, clear cache, impersonate user
|
||||
|
||||
### Nice to Have
|
||||
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
|
||||
- [ ] Diff view when editing fields
|
||||
- [ ] Audit log of admin changes
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Routing Fix (COMPLETED)
|
||||
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
|
||||
|
||||
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
|
||||
- Uses `useSearchParams()` to detect `?id=` parameter
|
||||
- Shows detail view when `selectedJobId` is present, list view otherwise
|
||||
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
|
||||
- Deleted old `/admin/jobs/detail/` directory
|
||||
|
||||
### Backend Enhancements
|
||||
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
|
||||
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
|
||||
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
|
||||
|
||||
### Frontend Structure
|
||||
The page will have collapsible sections:
|
||||
1. **Header**: Job ID, status badge, artist/title, quick actions
|
||||
2. **Overview Cards**: User, dates, progress, source info
|
||||
3. **Stage Timeline**: Visual breakdown of each stage with times
|
||||
4. **Files**: Downloadable links for all GCS files
|
||||
5. **Logs**: Filterable worker logs
|
||||
6. **Configuration**: Editable job settings
|
||||
7. **Raw Data**: JSON views of state_data, file_urls, full job
|
||||
|
||||
### Job Reset Logic
|
||||
Reset to specific states requires:
|
||||
- Clearing downstream state_data keys
|
||||
- Resetting status
|
||||
- Optionally clearing associated files
|
||||
- Adding timeline event for the reset
|
||||
|
||||
Valid reset targets:
|
||||
| Target State | Clears | Use Case |
|
||||
|--------------|--------|----------|
|
||||
| PENDING | Everything except input | Full restart |
|
||||
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
|
||||
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
|
||||
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
|
||||
|
||||
### Phase 1: Fix Routing (COMPLETED)
|
||||
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
|
||||
2. [x] Combined list and detail views in single component
|
||||
3. [x] Use `useSearchParams()` to detect `?id=` parameter
|
||||
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
|
||||
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
|
||||
6. [x] Verified: All 9 E2E tests pass, build succeeds
|
||||
|
||||
### Phase 2: Backend - File Downloads with Signed URLs
|
||||
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
|
||||
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
|
||||
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
|
||||
10. [ ] Traverse job.file_urls recursively
|
||||
11. [ ] Generate signed URLs for each GCS path
|
||||
12. [ ] Return structured response with file metadata
|
||||
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
|
||||
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
|
||||
|
||||
### Phase 3: Frontend - Enhanced File Display
|
||||
15. [ ] Create "Files" accordion section with grouped downloads
|
||||
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
|
||||
17. [ ] Show file name, type, download button
|
||||
18. [ ] Handle missing/unavailable files gracefully
|
||||
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
|
||||
20. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 4: Backend - Job Update Endpoint
|
||||
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
|
||||
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
|
||||
23. [ ] Define editable fields whitelist
|
||||
24. [ ] Validate field values (e.g., status must be valid enum)
|
||||
25. [ ] Log admin changes for audit trail
|
||||
26. [ ] Verify: Backend tests pass
|
||||
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
|
||||
|
||||
### Phase 5: Frontend - Editable Fields
|
||||
28. [ ] Create "Configuration" section with editable fields
|
||||
29. [ ] Use inline edit pattern (click to edit, enter to save)
|
||||
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
|
||||
31. [ ] Show validation errors inline
|
||||
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
|
||||
33. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 6: Backend - Job Reset Endpoint
|
||||
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
|
||||
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
|
||||
36. [ ] Accept `target_state` parameter
|
||||
37. [ ] Implement reset logic for each target state
|
||||
38. [ ] Clear appropriate state_data keys
|
||||
39. [ ] Optionally trigger workers after reset
|
||||
40. [ ] Verify: Backend tests pass
|
||||
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
|
||||
|
||||
### Phase 7: Frontend - Reset Actions
|
||||
42. [ ] Add "Reset Job" dropdown/menu in header
|
||||
43. [ ] Show available reset targets based on current state
|
||||
44. [ ] Confirm dialog explaining what will be cleared
|
||||
45. [ ] Show success/error toast after reset
|
||||
46. [ ] **E2E Test**: Add reset confirmation dialog test
|
||||
47. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 8: Enhanced Stage Timeline
|
||||
48. [ ] Parse timeline events to calculate stage durations
|
||||
49. [ ] Display stages as cards or timeline items
|
||||
50. [ ] Show duration for each stage
|
||||
51. [ ] Highlight current/blocking stages
|
||||
52. [ ] **E2E Test**: Add timeline display test
|
||||
|
||||
### Phase 9: Final Verification
|
||||
53. [ ] Run full backend test suite: `make test-backend`
|
||||
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
|
||||
55. [ ] Manual production testing checklist (see Testing Strategy)
|
||||
56. [ ] Verify CI passes on PR
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
|
||||
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
|
||||
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
|
||||
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
|
||||
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
|
||||
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Based on `docs/TESTING.md` guidelines, this feature requires:
|
||||
|
||||
### Test Types Required
|
||||
|
||||
| Type | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
|
||||
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
|
||||
| Manual Production E2E | N/A | Final validation on staging/prod |
|
||||
|
||||
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
|
||||
|
||||
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
|
||||
|
||||
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
|
||||
```python
|
||||
def test_get_job_files_returns_signed_urls_for_all_files():
|
||||
"""Test that all files in file_urls get signed URLs."""
|
||||
|
||||
def test_get_job_files_handles_nested_file_urls():
|
||||
"""Test recursive traversal of stems.instrumental_clean, etc."""
|
||||
|
||||
def test_get_job_files_handles_missing_files_gracefully():
|
||||
"""Test files that don't exist in GCS are omitted or marked."""
|
||||
|
||||
def test_get_job_files_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_get_job_files_returns_404_for_missing_job():
|
||||
"""Test invalid job_id returns 404."""
|
||||
```
|
||||
|
||||
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
|
||||
```python
|
||||
def test_update_job_allows_editable_fields():
|
||||
"""Test artist, title, user_email, theme_id can be updated."""
|
||||
|
||||
def test_update_job_rejects_non_editable_fields():
|
||||
"""Test job_id, created_at, etc. cannot be updated."""
|
||||
|
||||
def test_update_job_validates_status_enum():
|
||||
"""Test invalid status value returns 422."""
|
||||
|
||||
def test_update_job_validates_theme_id_exists():
|
||||
"""Test non-existent theme_id returns 422."""
|
||||
|
||||
def test_update_job_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_update_job_returns_updated_job():
|
||||
"""Test response includes updated job data."""
|
||||
```
|
||||
|
||||
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
|
||||
```python
|
||||
def test_reset_to_pending_clears_all_state_data():
|
||||
"""Test full reset clears everything except input."""
|
||||
|
||||
def test_reset_to_audio_selection_clears_search_results():
|
||||
"""Test audio selection reset clears audio_search_results."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_review_data():
|
||||
"""Test review reset clears screens, videos, review data."""
|
||||
|
||||
def test_reset_to_instrumental_selection_clears_finals():
|
||||
"""Test instrumental reset clears encoding results."""
|
||||
|
||||
def test_reset_adds_timeline_event():
|
||||
"""Test reset adds admin action to timeline."""
|
||||
|
||||
def test_reset_invalid_target_returns_422():
|
||||
"""Test invalid target_state returns validation error."""
|
||||
|
||||
def test_reset_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_reset_from_complete_to_pending_works():
|
||||
"""Test can reset completed job back to start."""
|
||||
```
|
||||
|
||||
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
|
||||
|
||||
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
|
||||
|
||||
test.describe('Admin Job Detail Page', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await setAdminAuthToken(page);
|
||||
});
|
||||
|
||||
test('navigates from jobs list to detail page', async ({ page }) => {
|
||||
// Mock jobs list and single job endpoints
|
||||
await page.route('**/api/admin/jobs', route => route.fulfill({
|
||||
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
|
||||
}));
|
||||
await page.route('**/api/jobs/test123', route => route.fulfill({
|
||||
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
|
||||
}));
|
||||
|
||||
await page.goto('/admin/jobs');
|
||||
await page.click('text=test123');
|
||||
await expect(page).toHaveURL('/admin/jobs/test123');
|
||||
await expect(page.getByText('Job test123')).toBeVisible();
|
||||
});
|
||||
|
||||
test('displays all job metadata sections', async ({ page }) => {
|
||||
// Test overview cards, timeline, logs accordion, etc.
|
||||
});
|
||||
|
||||
test('files section shows downloadable links', async ({ page }) => {
|
||||
// Mock files endpoint, verify download buttons appear
|
||||
});
|
||||
|
||||
test('edit field inline and save', async ({ page }) => {
|
||||
// Click edit, change value, press Enter, verify PATCH called
|
||||
});
|
||||
|
||||
test('reset job shows confirmation dialog', async ({ page }) => {
|
||||
// Click reset, verify dialog, confirm, verify POST called
|
||||
});
|
||||
|
||||
test('handles API errors gracefully', async ({ page }) => {
|
||||
// Mock 500 response, verify error toast shown
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test File Organization
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── test_admin_job_detail.py # NEW - all backend tests for this feature
|
||||
├── test_admin_api.py # Existing admin tests (keep separate)
|
||||
└── conftest.py # Shared fixtures
|
||||
|
||||
frontend/e2e/
|
||||
├── fixtures/
|
||||
│ ├── test-helper.ts # Add setAdminAuthToken helper
|
||||
│ └── mock-responses.ts # Add admin job detail mocks
|
||||
└── regression/
|
||||
└── admin-job-detail.spec.ts # NEW - E2E regression tests
|
||||
```
|
||||
|
||||
### Mocking Guidelines
|
||||
|
||||
**Backend - What to mock:**
|
||||
- `StorageService.generate_signed_url()` - return predictable test URLs
|
||||
- `StorageService.file_exists()` - control which files "exist"
|
||||
- Firestore operations via existing `mock_firestore` fixture
|
||||
|
||||
**Backend - What NOT to mock:**
|
||||
- `JobManager` logic - test the real reset/update logic
|
||||
- Validation logic - test real Pydantic validation
|
||||
- Authorization checks - test real `require_admin` dependency
|
||||
|
||||
**Frontend - What to mock:**
|
||||
- All API responses via Playwright route interception
|
||||
- Time (if testing token expiry)
|
||||
|
||||
**Frontend - What NOT to mock:**
|
||||
- React component behavior
|
||||
- User interactions (click, type, etc.)
|
||||
|
||||
### CI Requirements
|
||||
|
||||
This feature will be validated by existing CI checks:
|
||||
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
|
||||
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
|
||||
- `frontend-build` - Must pass (verifies new page compiles)
|
||||
|
||||
### Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Backend tests only (for this feature)
|
||||
poetry run pytest backend/tests/test_admin_job_detail.py -v
|
||||
|
||||
# All backend tests
|
||||
make test-backend
|
||||
|
||||
# Frontend E2E regression (includes new tests)
|
||||
cd frontend && npm run test:e2e
|
||||
|
||||
# Single E2E test file
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
|
||||
|
||||
# With browser visible for debugging
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
|
||||
```
|
||||
|
||||
### Manual Production Testing Checklist
|
||||
|
||||
After deployment, manually verify on production:
|
||||
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
|
||||
- [ ] All file download links work (valid signed URLs)
|
||||
- [ ] Edit artist/title → refresh → changes persisted
|
||||
- [ ] Reset job to pending → job status shows "pending"
|
||||
- [ ] Logs section loads and filters work
|
||||
- [ ] Error states handled (invalid job ID shows friendly error)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
|
||||
- **Recommendation**: Delete it. No external links depend on it.
|
||||
|
||||
2. [ ] Should job reset delete GCS files or just clear state_data references?
|
||||
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
|
||||
|
||||
3. [ ] Should edits be immediately saved or require explicit "Save" button?
|
||||
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
|
||||
2. Backend: New endpoints are additive, won't affect existing functionality
|
||||
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
|
||||
|
||||
## Notes
|
||||
|
||||
- The existing detail page has good foundational code - we're enhancing rather than replacing
|
||||
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
|
||||
- Job reset is powerful - consider adding confirmation dialogs and logging
|
||||
@@ -0,0 +1,218 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -0,0 +1,106 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
@@ -0,0 +1,401 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -0,0 +1,317 @@
|
||||
# Plan: White Label B2B Karaoke Portals
|
||||
|
||||
**Created:** 2026-01-08
|
||||
**Branch:** feat/sess-20260108-0358-white-label-portal
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
|
||||
|
||||
**First customer: Vocal Star**
|
||||
- Subdomain: `vocalstar.nomadkaraoke.com`
|
||||
- Workflow: Upload original + instrumental audio (no audio search)
|
||||
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
|
||||
- Distribution: Download only (no Dropbox/GDrive/YouTube)
|
||||
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
|
||||
- [ ] Per-tenant frontend branding (logo, colors, metadata)
|
||||
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
|
||||
- [ ] Per-tenant default theme (always use Vocal Star theme)
|
||||
- [ ] Per-tenant allowed email domains for magic link auth
|
||||
- [ ] Per-tenant fixed API token support (for automation)
|
||||
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
|
||||
- [ ] Users scoped to tenant
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Minimal code duplication - single codebase serves all tenants
|
||||
- [ ] Easy to add new tenants via config, not code changes
|
||||
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
|
||||
- [ ] No performance impact on existing single-tenant flow
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decision: Config-Driven Multi-Tenancy
|
||||
|
||||
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
|
||||
|
||||
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
|
||||
2. **Frontend detects subdomain** and fetches tenant config from API
|
||||
3. **Backend middleware** extracts tenant from request and applies config
|
||||
4. **Firestore collections** get `tenant_id` field for isolation
|
||||
|
||||
This approach:
|
||||
- Requires no database migrations for new tenants
|
||||
- Configs can be managed via simple JSON files
|
||||
- Easy to test locally with query param override (`?tenant=vocalstar`)
|
||||
- Can evolve to admin UI later if needed
|
||||
|
||||
### Tenant Config Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vocalstar",
|
||||
"name": "Vocal Star",
|
||||
"subdomain": "vocalstar.nomadkaraoke.com",
|
||||
"branding": {
|
||||
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
|
||||
"primary_color": "#ffff00",
|
||||
"secondary_color": "#006CF9",
|
||||
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
|
||||
"site_title": "Vocal Star Karaoke Generator"
|
||||
},
|
||||
"features": {
|
||||
"audio_search": false,
|
||||
"file_upload": true,
|
||||
"youtube_upload": false,
|
||||
"dropbox_upload": false,
|
||||
"gdrive_upload": false,
|
||||
"theme_selection": false
|
||||
},
|
||||
"defaults": {
|
||||
"theme_id": "vocalstar",
|
||||
"distribution_mode": "download_only"
|
||||
},
|
||||
"auth": {
|
||||
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
|
||||
"fixed_tokens": ["<will be in Secret Manager>"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### 1. Subdomain Detection (Frontend)
|
||||
- Next.js middleware checks `request.headers.host`
|
||||
- If subdomain matches tenant, store tenant ID in cookie/context
|
||||
- Fetch `/api/tenant/config` to get branding/features
|
||||
- Apply CSS variables dynamically for colors
|
||||
- Swap logo component based on config
|
||||
|
||||
#### 2. Tenant Middleware (Backend)
|
||||
- FastAPI middleware extracts tenant from:
|
||||
- `X-Tenant-ID` header (set by frontend)
|
||||
- Or subdomain from `Host` header
|
||||
- Or query param `?tenant=` (dev only)
|
||||
- Attaches `tenant_id` to request state
|
||||
- Tenant config loaded from GCS (cached 5 min like themes)
|
||||
|
||||
#### 3. Auth Enhancement
|
||||
- `AuthService.validate_token_full()` already returns `user_email`
|
||||
- Add tenant validation: check if user's email domain is in tenant's allowed list
|
||||
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
|
||||
- Magic links include tenant context for proper redirect
|
||||
|
||||
#### 4. Data Isolation
|
||||
- Add `tenant_id: Optional[str]` to User and Job models
|
||||
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
|
||||
- Query filters: `where('tenant_id', '==', tenant_id or None)`
|
||||
- No schema migration needed - new field, optional
|
||||
|
||||
#### 5. Theme Integration
|
||||
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
|
||||
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
|
||||
- Tenant config specifies `default_theme_id: "vocalstar"`
|
||||
- When `theme_selection: false`, UI hides theme picker and uses default
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Tenant Foundation (Day 1-2)
|
||||
|
||||
1. [ ] **Create tenant model and config loader**
|
||||
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
|
||||
- Create `backend/services/tenant_service.py` with GCS config loading and caching
|
||||
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
|
||||
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
|
||||
|
||||
2. [ ] **Add tenant middleware**
|
||||
- Create `backend/api/middleware/tenant.py`
|
||||
- Extract tenant from subdomain or header
|
||||
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
|
||||
- Add `get_tenant()` dependency for routes
|
||||
|
||||
3. [ ] **Update data models**
|
||||
- Add `tenant_id: Optional[str] = None` to `User` model
|
||||
- Add `tenant_id: Optional[str] = None` to `Job` model
|
||||
- Update Firestore indexes in Pulumi for tenant-scoped queries
|
||||
|
||||
4. [ ] **Update auth for tenant awareness**
|
||||
- Modify `AuthService` to validate email domain against tenant config
|
||||
- Add tenant context to magic link generation
|
||||
- Support per-tenant fixed tokens in `auth_tokens` collection
|
||||
|
||||
### Phase 2: Upload Vocal Star Theme (Day 2)
|
||||
|
||||
5. [ ] **Prepare Vocal Star theme for GCS**
|
||||
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
|
||||
- Upload background images to `themes/vocalstar/assets/`
|
||||
- Upload Oswald font to `themes/vocalstar/assets/`
|
||||
- Upload `style_params.json` to `themes/vocalstar/`
|
||||
- Create preview image and upload to `themes/vocalstar/preview.png`
|
||||
- Add Vocal Star to `themes/_metadata.json`
|
||||
|
||||
6. [ ] **Create Vocal Star tenant config**
|
||||
- Create `tenants/vocalstar/config.json` in GCS
|
||||
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
|
||||
- Upload favicon to `tenants/vocalstar/favicon.ico`
|
||||
|
||||
### Phase 3: Frontend Tenant Support (Day 3-4)
|
||||
|
||||
7. [ ] **Add tenant context provider**
|
||||
- Create `frontend/lib/tenant.ts` with tenant config types
|
||||
- Create `frontend/hooks/useTenant.ts` for tenant state
|
||||
- Create `TenantProvider` component that fetches config on mount
|
||||
- Detect subdomain in `_app.tsx` or middleware
|
||||
|
||||
8. [ ] **Dynamic branding**
|
||||
- Create `TenantLogo` component that renders tenant or default logo
|
||||
- Update `globals.css` to support CSS variable overrides
|
||||
- Add `applyTenantColors()` function to set CSS variables from config
|
||||
- Update page metadata (title, favicon) from tenant config
|
||||
|
||||
9. [ ] **Feature-flagged UI**
|
||||
- Wrap audio search in `{tenant.features.audio_search && ...}`
|
||||
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
|
||||
- Wrap distribution options based on tenant features
|
||||
- Show simplified form for Vocal Star (just file upload fields)
|
||||
|
||||
10. [ ] **Update magic link flow**
|
||||
- Pass tenant context when requesting magic link
|
||||
- Magic link redirect goes to tenant subdomain
|
||||
- Verify endpoint extracts tenant from link context
|
||||
|
||||
### Phase 4: Backend Feature Enforcement (Day 4-5)
|
||||
|
||||
11. [ ] **Enforce tenant features in API**
|
||||
- Audio search endpoint: return 403 if `audio_search: false`
|
||||
- File upload: check `file_upload: true`
|
||||
- Distribution settings: ignore/error if disabled features requested
|
||||
- Theme selection: use default if `theme_selection: false`
|
||||
|
||||
12. [ ] **Job creation with tenant context**
|
||||
- `JobCreate` handler sets `tenant_id` from request state
|
||||
- Distribution config respects tenant defaults
|
||||
- Theme defaults to tenant's default theme
|
||||
|
||||
13. [ ] **Query scoping**
|
||||
- Update `JobManager.get_jobs()` to filter by tenant_id
|
||||
- Update `UserService` queries to include tenant_id
|
||||
- Admin endpoints bypass tenant filter (see all tenants)
|
||||
|
||||
### Phase 5: Infrastructure & Deployment (Day 5-6)
|
||||
|
||||
14. [ ] **DNS setup**
|
||||
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
|
||||
- Configure Cloudflare Pages to accept wildcard subdomain
|
||||
- Add SSL certificate for subdomain
|
||||
|
||||
15. [ ] **Backend CORS update**
|
||||
- Add `vocalstar.nomadkaraoke.com` to allowed origins
|
||||
- Or use pattern matching for `*.nomadkaraoke.com`
|
||||
|
||||
16. [ ] **Pulumi updates**
|
||||
- Add Firestore indexes for tenant_id queries
|
||||
- Add any new secrets (tenant fixed tokens)
|
||||
|
||||
### Phase 6: Testing & Documentation (Day 6-7)
|
||||
|
||||
17. [ ] **Testing**
|
||||
- Unit tests for TenantService config loading
|
||||
- Unit tests for tenant middleware
|
||||
- E2E test for Vocal Star flow (upload audio, generate, download)
|
||||
- Test magic link works for @vocal-star.com email
|
||||
|
||||
18. [ ] **Documentation**
|
||||
- Update CLAUDE.md with tenant info
|
||||
- Create `docs/WHITE-LABEL.md` with tenant setup guide
|
||||
- Document how to add new tenants
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/tenant.py` | Create | Tenant config models |
|
||||
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
|
||||
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
|
||||
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
|
||||
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
|
||||
| `backend/models/user.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/models/job.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
|
||||
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
|
||||
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/main.py` | Modify | Add tenant middleware |
|
||||
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
|
||||
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
|
||||
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
|
||||
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
|
||||
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
|
||||
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
|
||||
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
|
||||
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
|
||||
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `TenantService`: Config loading, caching, validation
|
||||
- `TenantMiddleware`: Subdomain extraction, header extraction
|
||||
- `AuthService`: Tenant email domain validation
|
||||
- `JobManager`: Tenant-scoped queries
|
||||
|
||||
### Integration Tests (Emulator)
|
||||
- Create job with tenant context
|
||||
- Query jobs filtered by tenant
|
||||
- Auth with tenant-allowed email domain
|
||||
- Auth rejection for wrong email domain
|
||||
|
||||
### E2E Tests
|
||||
- Vocal Star happy path: Upload audio files, review lyrics, download video
|
||||
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
|
||||
- Feature hiding: Verify audio search not shown on Vocal Star portal
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] **Q: Should tenant configs be in GCS or Firestore?**
|
||||
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
|
||||
|
||||
- [x] **Q: How to handle localhost development for specific tenant?**
|
||||
A: Query param override `?tenant=vocalstar` for local testing
|
||||
|
||||
- [x] **Q: Should admin portal be tenant-aware?**
|
||||
A: No - admins see all tenants. Admin routes bypass tenant filter.
|
||||
|
||||
- [x] **Q: Do we need tenant-specific email sender identity?**
|
||||
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
|
||||
|
||||
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
|
||||
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
|
||||
|
||||
1. Revert frontend to not check tenant
|
||||
2. Revert backend middleware to not extract tenant
|
||||
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
|
||||
4. No schema migration to reverse
|
||||
|
||||
For DNS: Simply remove CNAME record if needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
|
||||
- [ ] Portal shows Vocal Star branding (logo, colors)
|
||||
- [ ] Audio search is hidden; only file upload available
|
||||
- [ ] Theme is locked to Vocal Star style
|
||||
- [ ] Distribution options hidden; download only
|
||||
- [ ] Magic link works for @vocal-star.com emails
|
||||
- [ ] Fixed API token works for automation
|
||||
- [ ] Generated videos use Vocal Star theme
|
||||
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
|
||||
@@ -0,0 +1,222 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -0,0 +1,83 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
@@ -0,0 +1,115 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
+854
-166
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,173 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
+9
-1
@@ -12,6 +12,9 @@ tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
@@ -44,6 +47,7 @@ frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
frontend/e2e/screenshots/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
@@ -79,8 +83,9 @@ eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib
|
||||
# But allow frontend lib directories
|
||||
!frontend/lib/
|
||||
!buy-site/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
@@ -224,3 +229,6 @@ cython_debug/
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -1,131 +1,163 @@
|
||||
# Project Guidelines for AI Assistants
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
This document contains guidelines and rules for AI assistants working on this codebase.
|
||||
## Project Overview
|
||||
|
||||
## Core Principles
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
**Always be testing.** Proactively evaluate, read, run, fix, and improve tests whenever you touch any part of the system—frontend, backend, infrastructure. Every code change should be thoroughly tested. Leave everything better tested than you found it.
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
**Always be documenting.** Continuously read, write, improve, update, and organize markdown documentation. Reference existing docs and keep them accurate as the codebase evolves.
|
||||
## Quick Reference
|
||||
|
||||
**Build for maintainability.** Keep code SOLID, modular, and clean. Favor simplicity and clarity. Future developers (and AI agents) will thank you.
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Product vision & goals | `docs/PRODUCT-VISION.md` |
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
## Branching Workflow
|
||||
## Tech Stack
|
||||
|
||||
**Never commit directly to `main`.** Always work on a feature branch.
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
Create descriptive branch names like `feature/add-discord-notifications`, `fix/audio-sync-issue`, or `refactor/cleanup-api-routes`.
|
||||
## Essential Rules
|
||||
|
||||
### Using Git Worktrees
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
**For non-trivial changes** (new features, multi-file edits, refactors):
|
||||
- Create a git worktree in the parent folder before making any file changes:
|
||||
```bash
|
||||
git worktree add ../karaoke-gen-feat-xyz feature/xyz
|
||||
```
|
||||
- This keeps the main directory clean on `main` and enables parallel work on multiple features
|
||||
- Name the worktree folder descriptively, e.g., `../karaoke-gen-fix-audio-sync`
|
||||
### Testing (Required)
|
||||
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
|
||||
- Which test types are needed (unit, integration, E2E)
|
||||
- Where tests should be placed
|
||||
- What mocking approach to use
|
||||
- Coverage expectations
|
||||
- **Production E2E tests for user-facing features** - see below
|
||||
|
||||
**For trivial changes** (typos, single-line fixes):
|
||||
- Creating a branch in the current directory is acceptable
|
||||
- Use `git checkout -b fix/typo-xyz` before making changes
|
||||
|
||||
**Cleanup:** When done with a worktree, remove it with:
|
||||
```bash
|
||||
git worktree remove ../karaoke-gen-feat-xyz
|
||||
```
|
||||
|
||||
### Creating a Pull Request
|
||||
|
||||
Once all changes are complete, tests pass, and commits are pushed, create a PR with:
|
||||
|
||||
1. **Summary** - Brief overview of the changes
|
||||
2. **Changes Made** - List of specific modifications
|
||||
3. **Testing** - How changes were tested (unit tests, manual testing)
|
||||
4. **Related Issues** - Reference any GitHub issues (e.g., "Fixes #123")
|
||||
|
||||
Only merge after review and CI checks pass.
|
||||
|
||||
## Infrastructure Changes
|
||||
|
||||
All Google Cloud infrastructure (IAM, Cloud Storage, Cloud Run, Secret Manager, Artifact Registry, etc.) must be managed via **Pulumi IaC** in the `infrastructure/` folder.
|
||||
|
||||
### Rules
|
||||
|
||||
- **Reading/querying** with `gcloud` CLI is fine for investigation and debugging
|
||||
- **Temporary changes** via `gcloud` CLI or Console are acceptable for urgent fixes or experimentation, but **must be cleaned up** and properly codified in the IaC afterward
|
||||
- **Permanent changes** must go through Pulumi — never leave `gcloud` or Console changes as the final state
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Make changes in `infrastructure/`
|
||||
2. Run `pulumi preview` and verify the output
|
||||
3. Run `pulumi up` to apply
|
||||
|
||||
### Authentication Issues
|
||||
|
||||
If you encounter any authentication issues with `gcloud` or `pulumi` (e.g., expired tokens, permission errors), **stop and notify the user** so they can run the appropriate auth commands in their external terminal:
|
||||
|
||||
```bash
|
||||
gcloud auth login
|
||||
gcloud auth application-default login
|
||||
```
|
||||
|
||||
Do not attempt to work around auth issues or make assumptions about credentials.
|
||||
|
||||
## Version Management
|
||||
|
||||
When committing and pushing changes, always bump the patch version in `pyproject.toml`.
|
||||
|
||||
The version is located at `tool.poetry.version` in `pyproject.toml`. For example, if the current version is `0.70.3`, bump it to `0.70.4`.
|
||||
|
||||
This ensures every commit pushed to the repository has a unique version number for tracking and deployment purposes.
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
**Before committing and pushing ANY changes, you MUST run the full test suite.**
|
||||
|
||||
### Backend Test Command
|
||||
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
|
||||
|
||||
**Before committing**, run all tests:
|
||||
```bash
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
**Important for AI agents:**
|
||||
- Set a **10 minute timeout** when running tests (they can take a while with emulators)
|
||||
- Always pipe output through `tail -n 500` to limit output and preserve context window
|
||||
- The tail output will show test results and any failures at the end
|
||||
This single command:
|
||||
- Installs dependencies automatically (backend + frontend)
|
||||
- Runs all backend tests (unit, integration, emulator)
|
||||
- Runs all frontend tests (Jest unit + Playwright E2E)
|
||||
- Takes ~5-10 minutes
|
||||
|
||||
This runs all backend tests in order and fails fast if any test fails:
|
||||
1. Unit tests (karaoke_gen package) with coverage check
|
||||
2. Backend unit tests
|
||||
3. E2E integration tests with emulators (auto-starts/stops emulators)
|
||||
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
|
||||
|
||||
### Frontend Test Command
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
When working in the `frontend/` directory, run:
|
||||
### Infrastructure
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
This runs:
|
||||
1. **Unit tests** (Jest) - Component and utility tests
|
||||
2. **E2E tests** (Playwright) - Browser-based integration tests
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
**Note:** E2E tests require `KARAOKE_ACCESS_TOKEN` environment variable to be set in `frontend/.env.local`.
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
Individual test commands:
|
||||
- `npm run test:unit` - Run unit tests only
|
||||
- `npm run test:e2e` - Run E2E tests only
|
||||
- `npm run test:ci` - Run unit tests with coverage
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
### Pre-commit Checklist
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
- [ ] `make test` passes (for backend changes)
|
||||
- [ ] `cd frontend && npm run test:all` passes (for frontend changes)
|
||||
- [ ] Version in `pyproject.toml` has been bumped
|
||||
- [ ] `coderabbit review --plain` has been run to get comprehensive analysis and suggestions for cleaner, more maintainable code. Apply the feedback to improve accessibility, structure, and best practices.
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
**Do not commit or push code that fails tests. Fix any test failures before proceeding.**
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-PR Checklist
|
||||
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
@@ -1,34 +1,50 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
.PHONY: help install install-backend install-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Install dependencies (only if needed)
|
||||
install-backend:
|
||||
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
|
||||
echo "=== Installing backend dependencies ==="; \
|
||||
poetry install; \
|
||||
fi
|
||||
|
||||
install-frontend:
|
||||
@if [ ! -d "frontend/node_modules" ]; then \
|
||||
echo "=== Installing frontend dependencies ==="; \
|
||||
cd frontend && npm install; \
|
||||
fi
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit:
|
||||
test-unit: install-backend
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e:
|
||||
test-e2e: install-backend
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
@@ -42,8 +58,18 @@ test-e2e:
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
# Run all backend tests (unit + emulator)
|
||||
test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend tests (unit + E2E)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend tests ==="
|
||||
cd frontend && npm run test:all
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
|
||||
@@ -71,8 +71,40 @@ export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Whisper via RunPod
|
||||
Open-source alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
@@ -574,6 +606,44 @@ If the output video has quality problems:
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
[
|
||||
{
|
||||
"test_name": "Sound-alike error (wurld -> world)",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "ReplaceWord",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w1"
|
||||
],
|
||||
"replacement_text": "world",
|
||||
"confidence": 0.75,
|
||||
"reason": "Sound-alike error. Reference suggests: 'world'. The transcription 'wurld' is a phonetic misspelling or sound-alike of the word 'world' which appears in both reference sources (Genius and LRCLIB).",
|
||||
"gap_category": "GapCategory.SOUND_ALIKE",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Punctuation difference",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w2"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The gap text 'Hello' followed by the context 'is it me you're looking for' matches the Genius reference lyrics exactly, excluding punctuation. According to the guidelines, if any reference source matches the transcription text (ignoring punctuation), it should be classified as NO_ERROR.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Extra filler word",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "DeleteWord",
|
||||
"word_id": "w3",
|
||||
"word_ids": null,
|
||||
"replacement_text": null,
|
||||
"confidence": 0.8,
|
||||
"reason": "Extra filler word 'And' at sentence start not in reference. The transcription includes the filler word 'And' at the beginning of the phrase, which is not present in the reference lyrics. This matches the definition of EXTRA_WORDS.",
|
||||
"gap_category": "GapCategory.EXTRA_WORDS",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Complex multi-word error",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w7",
|
||||
"w8",
|
||||
"w9",
|
||||
"w10"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The transcription text 'gonna give you up' matches the Genius reference lyrics exactly when ignoring punctuation and capitalization. In context, the full phrase 'never gonna give you up' is correctly transcribed and aligns perfectly with the known lyrics and title of the song.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
+31
-1
@@ -10,14 +10,43 @@ FROM python:3.11-slim
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
ffmpeg \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
@@ -25,6 +54,7 @@ WORKDIR /app
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
|
||||
+182
-70
@@ -4,11 +4,11 @@ FastAPI dependencies for authentication and authorization.
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable
|
||||
from typing import Optional, Tuple, Callable, Union
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -66,70 +66,130 @@ async def require_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
AuthResult with full authentication context including:
|
||||
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
||||
- user_email: Email of authenticated user (if session/API key auth)
|
||||
- is_admin: Whether user has admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
# Log more details for debugging token issues
|
||||
|
||||
# Validate token using the full method
|
||||
auth_result = auth_service.validate_token_full(token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
|
||||
if not auth_result.is_valid:
|
||||
# Log auth failure with request_id for correlation
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
f"Authentication failed: {message}. "
|
||||
f"Token provided: {bool(token_str)}, "
|
||||
f"Token length: {len(token_str) if token_str else 0}, "
|
||||
f"Auth header present: {bool(auth_header)}, "
|
||||
f"Header prefix: {auth_header[:20] if auth_header else 'none'}..."
|
||||
"auth_failed",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"audit_type": "auth_event",
|
||||
"auth_message": auth_result.message,
|
||||
"token_provided": bool(token_str),
|
||||
"token_length": len(token_str) if token_str else 0,
|
||||
"auth_header_present": bool(auth_header),
|
||||
}
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"user_email": auth_result.user_email,
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
async def require_auth_legacy(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Legacy authentication dependency that returns tuple.
|
||||
|
||||
DEPRECATED: Use require_auth which returns AuthResult instead.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
"""
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
logger.info(f"Authenticated as {user_type} (remaining: {remaining_uses})")
|
||||
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
|
||||
Admin access is granted if:
|
||||
- Using an admin token from ADMIN_TOKENS env var
|
||||
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
||||
- User role is set to ADMIN in database
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
|
||||
AuthResult with admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
token, user_type, remaining_uses = auth_data
|
||||
|
||||
if user_type != UserType.ADMIN:
|
||||
logger.warning(f"Admin access denied for {user_type} user")
|
||||
if not auth_result.is_admin:
|
||||
logger.warning(
|
||||
f"Admin access denied for {auth_result.user_type} user "
|
||||
f"(email: {auth_result.user_email or 'unknown'})"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return token, user_type, remaining_uses
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
def optional_auth(
|
||||
@@ -168,14 +228,14 @@ def optional_auth(
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
@@ -189,37 +249,63 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Review access granted via full auth ({user_type}) for job {job_id}")
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Review access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
@@ -229,25 +315,25 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
@@ -258,14 +344,14 @@ require_review_auth = require_review_auth_factory()
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token)
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
@@ -279,37 +365,63 @@ def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
|
||||
if full_token:
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(full_token)
|
||||
if is_valid:
|
||||
logger.info(f"Instrumental access granted via full auth ({user_type}) for job {job_id}")
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Instrumental access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
@@ -319,25 +431,25 @@ def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -15,12 +15,13 @@ import asyncio
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from karaoke_gen.utils import normalize_text
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
@@ -32,10 +33,13 @@ from backend.services.audio_search_service import (
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -85,13 +89,17 @@ class AudioSearchRequest(BaseModel):
|
||||
# Auto-download mode
|
||||
auto_download: bool = Field(False, description="Automatically select best result and download")
|
||||
|
||||
# Processing options (CDG/TXT require style config, disabled by default)
|
||||
enable_cdg: bool = Field(False, description="Generate CDG+MP3 package (requires style config)")
|
||||
enable_txt: bool = Field(False, description="Generate TXT+MP3 package (requires style config)")
|
||||
# Theme configuration
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
@@ -103,6 +111,10 @@ class AudioSearchRequest(BaseModel):
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Display overrides (optional - if empty, search values are used for display)
|
||||
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
|
||||
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
@@ -113,6 +125,16 @@ class AudioSearchRequest(BaseModel):
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
|
||||
def normalize_text_fields(cls, v):
|
||||
"""Normalize text fields to standardize Unicode characters."""
|
||||
if v is not None and isinstance(v, str):
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
@@ -339,8 +361,31 @@ async def _download_and_start_processing(
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
)
|
||||
elif source_name == 'YouTube' and download_url:
|
||||
# For YouTube without remote, download directly using URL from state_data
|
||||
# This avoids relying on in-memory cache which doesn't persist across instances
|
||||
logger.info(f"Downloading YouTube audio directly from URL: {download_url}")
|
||||
from backend.workers.audio_worker import download_from_url
|
||||
|
||||
local_path = await download_from_url(
|
||||
download_url,
|
||||
temp_dir,
|
||||
selected.get('artist'),
|
||||
selected.get('title')
|
||||
)
|
||||
|
||||
if not local_path or not os.path.exists(local_path):
|
||||
raise DownloadError(f"Failed to download from YouTube: {download_url}")
|
||||
|
||||
# Create a result-like object
|
||||
class DownloadResult:
|
||||
def __init__(self, filepath):
|
||||
self.filepath = filepath
|
||||
|
||||
result = DownloadResult(local_path)
|
||||
else:
|
||||
# Fallback to search-based download
|
||||
# Fallback to search-based download (may fail if cache is empty)
|
||||
logger.warning(f"Falling back to cache-based download for {source_name} - this may fail on multi-instance deployments")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
@@ -403,21 +448,29 @@ async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
|
||||
|
||||
This endpoint:
|
||||
1. Creates a job in PENDING state
|
||||
2. Searches for audio using flacfetch
|
||||
3. Either returns search results for user selection, or
|
||||
4. If auto_download=True, automatically selects best and starts processing
|
||||
|
||||
|
||||
Use cases:
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.audio_search:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Audio search is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Apply default distribution settings
|
||||
settings = get_settings()
|
||||
@@ -426,7 +479,8 @@ async def search_audio(
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
@@ -461,13 +515,38 @@ async def search_audio(
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Determine display values - use display_* if provided, otherwise fall back to search values
|
||||
# Display values are used for title screens, filenames, YouTube, etc.
|
||||
# Search values (body.artist, body.title) are used for audio search
|
||||
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
|
||||
effective_display_title = body.display_title.strip() if body.display_title else body.title
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=body.artist,
|
||||
title=body.title,
|
||||
enable_cdg=body.enable_cdg,
|
||||
enable_txt=body.enable_txt,
|
||||
artist=effective_display_artist, # Display value for title screens, filenames
|
||||
title=effective_display_title, # Display value for title screens, filenames
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=effective_brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=effective_youtube_description,
|
||||
@@ -475,6 +554,7 @@ async def search_audio(
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
@@ -485,10 +565,13 @@ async def search_audio(
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
@@ -497,7 +580,28 @@ async def search_audio(
|
||||
'audio_search_title': body.title,
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if effective_theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
@@ -705,7 +809,7 @@ async def search_audio(
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
@@ -740,7 +844,7 @@ async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
+505
-159
File diff suppressed because it is too large.
Load diff
@@ -3,16 +3,126 @@ Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
@@ -99,34 +209,192 @@ async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
email_status = {
|
||||
"configured": email_service.is_configured(),
|
||||
"provider": type(email_service.provider).__name__,
|
||||
}
|
||||
|
||||
# Check Stripe service
|
||||
stripe_service = get_stripe_service()
|
||||
stripe_status = {
|
||||
"configured": stripe_service.is_configured(),
|
||||
}
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
|
||||
if not email_status["configured"]:
|
||||
logger.warning("Email service not configured - magic links will not work")
|
||||
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
"stripe": stripe_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
async def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -316,13 +316,124 @@ async def trigger_render_video_worker(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
|
||||
if getattr(job, 'made_for_you', False):
|
||||
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
|
||||
add_span_event("made_for_you_skip")
|
||||
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
|
||||
+367
-66
@@ -11,7 +11,7 @@ import asyncio
|
||||
import logging
|
||||
import httpx
|
||||
from typing import List, Optional, Dict, Any, Tuple
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
|
||||
from backend.models.job import Job, JobCreate, JobResponse, JobStatus
|
||||
from backend.models.requests import (
|
||||
@@ -25,10 +25,13 @@ from backend.models.requests import (
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.services.metrics import metrics
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -57,7 +60,7 @@ async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
async def create_job(
|
||||
request: URLSubmissionRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> JobResponse:
|
||||
"""
|
||||
Create a new karaoke generation job from a URL.
|
||||
@@ -69,20 +72,54 @@ async def create_job(
|
||||
4. When both complete, job transitions to AWAITING_REVIEW
|
||||
"""
|
||||
try:
|
||||
# Determine job owner email:
|
||||
# All authentication methods must provide a user_email for job ownership
|
||||
if auth_result.user_email:
|
||||
# Use authenticated user's email (standard case)
|
||||
user_email = auth_result.user_email
|
||||
else:
|
||||
# This should never happen - all auth methods now require user_email
|
||||
logger.error("Authentication succeeded but no user_email provided")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Authentication error: no user identity available"
|
||||
)
|
||||
|
||||
# Admins can optionally create jobs on behalf of other users
|
||||
if request.user_email and auth_result.is_admin and request.user_email != auth_result.user_email:
|
||||
user_email = request.user_email
|
||||
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
|
||||
|
||||
# Apply YouTube upload default from settings
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
settings = get_settings()
|
||||
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
|
||||
# Apply default theme - all jobs require a theme
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if not effective_theme_id:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="No default theme configured. Please contact support or specify a theme_id."
|
||||
)
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Create job with all preferences
|
||||
job_create = JobCreate(
|
||||
url=str(request.url),
|
||||
artist=request.artist,
|
||||
title=request.title,
|
||||
theme_id=effective_theme_id, # Required - all jobs must have a theme
|
||||
enable_cdg=request.enable_cdg,
|
||||
enable_txt=request.enable_txt,
|
||||
enable_youtube_upload=request.enable_youtube_upload,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=request.youtube_description,
|
||||
webhook_url=request.webhook_url,
|
||||
user_email=request.user_email
|
||||
user_email=user_email
|
||||
)
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
|
||||
# Record job creation metric
|
||||
metrics.record_job_created(job.job_id, source="url")
|
||||
|
||||
@@ -109,71 +146,130 @@ async def create_job(
|
||||
@router.get("/{job_id}", response_model=Job)
|
||||
async def get_job(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Job:
|
||||
"""Get job status and details."""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only see their own jobs, admins can see all
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
# If job is complete, include download URLs
|
||||
if job.status == JobStatus.COMPLETE:
|
||||
job.download_urls = job_manager.get_output_urls(job_id)
|
||||
|
||||
|
||||
return job
|
||||
|
||||
|
||||
def _check_job_ownership(job: Job, auth_result: AuthResult) -> bool:
|
||||
"""
|
||||
Check if the authenticated user owns the job or has admin access.
|
||||
|
||||
Returns:
|
||||
True if user can access the job, False otherwise
|
||||
"""
|
||||
# Admins can access all jobs
|
||||
if auth_result.is_admin:
|
||||
return True
|
||||
|
||||
# Check if user owns the job
|
||||
if auth_result.user_email and job.user_email:
|
||||
return auth_result.user_email.lower() == job.user_email.lower()
|
||||
|
||||
# If no user_email on auth (token auth without email), deny access to jobs with user_email
|
||||
# This prevents token-based auth from accessing user jobs
|
||||
if job.user_email:
|
||||
return False
|
||||
|
||||
# Legacy jobs without user_email - allow access for backward compatibility
|
||||
# TODO: Consider restricting this in the future
|
||||
return True
|
||||
|
||||
|
||||
@router.get("", response_model=List[Job])
|
||||
async def list_jobs(
|
||||
request: Request,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[str] = None,
|
||||
created_before: Optional[str] = None,
|
||||
exclude_test: bool = True,
|
||||
limit: int = 100,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
|
||||
Regular users only see their own jobs. Admins see all jobs.
|
||||
Users on tenant portals only see jobs from their tenant.
|
||||
|
||||
Args:
|
||||
status: Filter by job status (pending, complete, failed, etc.)
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this ISO datetime (e.g., 2024-01-01T00:00:00Z)
|
||||
created_before: Filter jobs created before this ISO datetime
|
||||
exclude_test: If True (default), exclude jobs from test users (admin only)
|
||||
limit: Maximum number of jobs to return (default 100)
|
||||
|
||||
|
||||
Returns:
|
||||
List of jobs matching filters, ordered by created_at descending
|
||||
"""
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
try:
|
||||
# Parse datetime strings if provided
|
||||
created_after_dt = None
|
||||
created_before_dt = None
|
||||
|
||||
|
||||
if created_after:
|
||||
try:
|
||||
created_after_dt = datetime.fromisoformat(created_after.replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_after format: {created_after}")
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_after format: {created_after}") from e
|
||||
|
||||
if created_before:
|
||||
try:
|
||||
created_before_dt = datetime.fromisoformat(created_before.replace('Z', '+00:00'))
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_before format: {created_before}")
|
||||
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid created_before format: {created_before}") from e
|
||||
|
||||
# Determine user_email filter based on admin status
|
||||
# Admins see all jobs, regular users only see their own
|
||||
user_email_filter = None
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email:
|
||||
user_email_filter = auth_result.user_email
|
||||
logger.debug(f"Filtering jobs for user: {user_email_filter}")
|
||||
else:
|
||||
# Token-based auth without user email - show no jobs for security
|
||||
logger.warning("Non-admin auth without user_email, returning empty job list")
|
||||
return []
|
||||
|
||||
# Get tenant_id from request for portal scoping
|
||||
# Tenant users only see jobs from their tenant
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
|
||||
jobs = job_manager.list_jobs(
|
||||
status=status,
|
||||
environment=environment,
|
||||
client_id=client_id,
|
||||
created_after=created_after_dt,
|
||||
created_before=created_before_dt,
|
||||
user_email=user_email_filter,
|
||||
tenant_id=tenant_id,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
# Filter out test user jobs if exclude_test is True (admin only)
|
||||
if exclude_test and auth_result.is_admin:
|
||||
jobs = [j for j in jobs if not is_test_email(j.user_email or "")]
|
||||
|
||||
logger.debug(f"Listed {len(jobs)} jobs for user={auth_result.user_email}, admin={auth_result.is_admin}")
|
||||
return jobs
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -186,16 +282,20 @@ async def list_jobs(
|
||||
async def delete_job(
|
||||
job_id: str,
|
||||
delete_files: bool = True,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""Delete a job and optionally its output files."""
|
||||
try:
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only delete their own jobs
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to delete this job")
|
||||
|
||||
job_manager.delete_job(job_id, delete_files=delete_files)
|
||||
|
||||
|
||||
return {"status": "success", "message": f"Job {job_id} deleted"}
|
||||
except HTTPException:
|
||||
raise
|
||||
@@ -212,7 +312,7 @@ async def bulk_delete_jobs(
|
||||
created_before: Optional[str] = None,
|
||||
delete_files: bool = True,
|
||||
confirm: bool = False,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
_auth_result: AuthResult = Depends(require_admin)
|
||||
) -> dict:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
@@ -316,18 +416,22 @@ async def bulk_delete_jobs(
|
||||
@router.get("/{job_id}/review-data")
|
||||
async def get_review_data(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get data needed for lyrics review interface.
|
||||
|
||||
|
||||
Returns corrections JSON URL and audio URL.
|
||||
Frontend loads these to render the review UI.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -376,13 +480,21 @@ async def get_review_data(
|
||||
async def start_review(
|
||||
job_id: str,
|
||||
request: StartReviewRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Mark job as IN_REVIEW (user opened review interface).
|
||||
|
||||
|
||||
This helps track that the user is actively working on the review.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
success = job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.IN_REVIEW,
|
||||
@@ -400,20 +512,24 @@ async def submit_corrections(
|
||||
job_id: str,
|
||||
submission: CorrectionsSubmission,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Save corrected lyrics during human review.
|
||||
|
||||
|
||||
This endpoint saves review progress but does NOT complete the review.
|
||||
Call POST /{job_id}/complete-review to finish and trigger video rendering.
|
||||
|
||||
|
||||
Can be called multiple times to save progress.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -459,11 +575,11 @@ async def submit_corrections(
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Complete the human review and trigger video rendering.
|
||||
|
||||
|
||||
This is the FIRST critical human-in-the-loop completion point.
|
||||
After this:
|
||||
1. Job transitions to REVIEW_COMPLETE
|
||||
@@ -474,7 +590,11 @@ async def complete_review(
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to modify this job")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -948,18 +1068,22 @@ async def select_instrumental(
|
||||
@router.get("/{job_id}/download-urls")
|
||||
async def get_download_urls(
|
||||
job_id: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Get download URLs for all job output files.
|
||||
|
||||
|
||||
Returns a dictionary mapping file types to download URLs.
|
||||
Uses the streaming download endpoint which proxies through the backend.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status != JobStatus.COMPLETE:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -990,26 +1114,42 @@ async def get_download_urls(
|
||||
}
|
||||
|
||||
|
||||
# Map file keys to human-readable suffixes (matching Dropbox naming from karaoke_finalise.py)
|
||||
DOWNLOAD_FILENAME_SUFFIXES = {
|
||||
"lossless_4k_mp4": " (Final Karaoke Lossless 4k).mp4",
|
||||
"lossless_4k_mkv": " (Final Karaoke Lossless 4k).mkv",
|
||||
"lossy_4k_mp4": " (Final Karaoke Lossy 4k).mp4",
|
||||
"lossy_720p_mp4": " (Final Karaoke Lossy 720p).mp4",
|
||||
"cdg_zip": " (Final Karaoke CDG).zip",
|
||||
"txt_zip": " (Final Karaoke TXT).zip",
|
||||
"with_vocals": " (With Vocals).mkv",
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{job_id}/download/{category}/{file_key}")
|
||||
async def download_file(
|
||||
job_id: str,
|
||||
category: str,
|
||||
file_key: str,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Stream download a specific file from a completed job.
|
||||
|
||||
|
||||
This endpoint proxies the file from GCS through the backend,
|
||||
so no client-side authentication is required.
|
||||
"""
|
||||
from fastapi.responses import StreamingResponse
|
||||
import tempfile
|
||||
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access this job")
|
||||
|
||||
if job.status != JobStatus.COMPLETE:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1050,8 +1190,18 @@ async def download_file(
|
||||
}
|
||||
content_type = content_types.get(ext, 'application/octet-stream')
|
||||
|
||||
# Get filename for Content-Disposition
|
||||
filename = gcs_path.split('/')[-1]
|
||||
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
|
||||
# Use sanitize_filename to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that cause HTTP header encoding issues (Content-Disposition uses latin-1)
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
artist_clean = sanitize_filename(job.artist) if job.artist else None
|
||||
title_clean = sanitize_filename(job.title) if job.title else None
|
||||
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
|
||||
|
||||
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
|
||||
filename = f"{base_name}{DOWNLOAD_FILENAME_SUFFIXES[file_key]}"
|
||||
else:
|
||||
filename = gcs_path.split('/')[-1] # Fallback to original
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
@@ -1086,13 +1236,21 @@ async def download_file(
|
||||
async def cancel_job(
|
||||
job_id: str,
|
||||
request: CancelJobRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Cancel a job.
|
||||
|
||||
|
||||
Jobs can be cancelled at any stage before completion.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to cancel this job")
|
||||
|
||||
success = job_manager.cancel_job(job_id, reason=request.reason)
|
||||
|
||||
if not success:
|
||||
@@ -1109,7 +1267,7 @@ async def cancel_job(
|
||||
async def retry_job(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> dict:
|
||||
"""
|
||||
Retry a failed or cancelled job from the last successful checkpoint.
|
||||
@@ -1127,6 +1285,10 @@ async def retry_job(
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Check ownership
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to retry this job")
|
||||
|
||||
if job.status not in [JobStatus.FAILED, JobStatus.CANCELLED]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
@@ -1151,12 +1313,13 @@ async def retry_job(
|
||||
|
||||
logger.info(f"Job {job_id}: Has rendered video and instrumental selection, retrying video generation")
|
||||
|
||||
# Clear error state
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'video_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to INSTRUMENTAL_SELECTED and trigger video worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1184,13 +1347,14 @@ async def retry_job(
|
||||
file_urls.get('screens', {}).get('title')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has corrections and screens, retrying from render stage")
|
||||
|
||||
# Clear error state
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'render_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to REVIEW_COMPLETE and trigger render worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1218,13 +1382,14 @@ async def retry_job(
|
||||
file_urls.get('lyrics', {}).get('corrections')):
|
||||
|
||||
logger.info(f"Job {job_id}: Has stems and corrections, retrying from screens stage")
|
||||
|
||||
# Clear error state
|
||||
|
||||
# Clear error state and reset worker progress for idempotency
|
||||
job_manager.update_job(job_id, {
|
||||
'error_message': None,
|
||||
'error_details': None,
|
||||
})
|
||||
|
||||
job_manager.update_state_data(job_id, 'screens_progress', {'stage': 'pending'})
|
||||
|
||||
# Reset to a state before screens and trigger screens worker
|
||||
if not job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
@@ -1301,19 +1466,23 @@ async def get_worker_logs(
|
||||
job_id: str,
|
||||
since_index: int = 0,
|
||||
worker: Optional[str] = None,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_auth)
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get worker logs for debugging.
|
||||
|
||||
|
||||
This endpoint returns worker logs stored in Firestore.
|
||||
Use `since_index` for efficient polling (returns only new logs).
|
||||
|
||||
|
||||
Logs are stored in a subcollection (jobs/{job_id}/logs) to avoid
|
||||
the 1MB document size limit. Older jobs may have logs in an embedded
|
||||
array (worker_logs field) - this endpoint handles both transparently.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination/polling)
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render)
|
||||
|
||||
worker: Filter by worker name (audio, lyrics, screens, video, render, distribution)
|
||||
|
||||
Returns:
|
||||
{
|
||||
"logs": [{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}],
|
||||
@@ -1324,13 +1493,145 @@ async def get_worker_logs(
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Check ownership - users can only see logs for their own jobs
|
||||
if not _check_job_ownership(job, auth_result):
|
||||
raise HTTPException(status_code=403, detail="You don't have permission to access logs for this job")
|
||||
|
||||
logs = job_manager.get_worker_logs(job_id, since_index=since_index, worker=worker)
|
||||
total = len(job.worker_logs) if job.worker_logs else 0
|
||||
|
||||
total = job_manager.get_worker_logs_count(job_id)
|
||||
|
||||
return {
|
||||
"logs": logs,
|
||||
"next_index": total,
|
||||
"next_index": since_index + len(logs),
|
||||
"total_logs": total
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{job_id}/cleanup-distribution")
|
||||
async def cleanup_distribution(
|
||||
job_id: str,
|
||||
delete_job: bool = True,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
) -> dict:
|
||||
"""
|
||||
Clean up all distributed content for a job (YouTube, Dropbox, Google Drive).
|
||||
|
||||
This admin-only endpoint is designed for E2E test cleanup. It:
|
||||
1. Deletes YouTube video (if uploaded)
|
||||
2. Deletes Dropbox folder (if uploaded)
|
||||
3. Deletes Google Drive files (if uploaded)
|
||||
4. Optionally deletes the job itself
|
||||
|
||||
Args:
|
||||
job_id: Job ID to clean up
|
||||
delete_job: If True, also delete the job after cleaning up distribution
|
||||
|
||||
Returns:
|
||||
Cleanup results for each service
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
state_data = job.state_data or {}
|
||||
results = {
|
||||
"job_id": job_id,
|
||||
"youtube": {"status": "skipped", "reason": "no youtube_url in state_data"},
|
||||
"dropbox": {"status": "skipped", "reason": "no brand_code or dropbox_path"},
|
||||
"gdrive": {"status": "skipped", "reason": "no gdrive_files in state_data"},
|
||||
"job_deleted": False
|
||||
}
|
||||
|
||||
# Clean up YouTube
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
if youtube_url:
|
||||
try:
|
||||
# Extract video ID from URL (format: https://youtu.be/VIDEO_ID or https://www.youtube.com/watch?v=VIDEO_ID)
|
||||
import re
|
||||
video_id_match = re.search(r'(?:youtu\.be/|youtube\.com/watch\?v=)([^&\s]+)', youtube_url)
|
||||
if video_id_match:
|
||||
video_id = video_id_match.group(1)
|
||||
|
||||
# Import and use karaoke_finalise for YouTube deletion
|
||||
from karaoke_gen.karaoke_finalise.karaoke_finalise import KaraokeFinalise
|
||||
from backend.services.youtube_service import get_youtube_service
|
||||
|
||||
youtube_service = get_youtube_service()
|
||||
if youtube_service.is_configured:
|
||||
# Create minimal KaraokeFinalise instance for deletion
|
||||
finalise = KaraokeFinalise(
|
||||
dry_run=False,
|
||||
non_interactive=True,
|
||||
enable_youtube=True,
|
||||
user_youtube_credentials=youtube_service.get_credentials_dict()
|
||||
)
|
||||
success = finalise.delete_youtube_video(video_id)
|
||||
results["youtube"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"video_id": video_id
|
||||
}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": "YouTube credentials not configured"}
|
||||
else:
|
||||
results["youtube"] = {"status": "failed", "reason": f"Could not extract video ID from {youtube_url}"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up YouTube for job {job_id}: {e}", exc_info=True)
|
||||
results["youtube"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Dropbox
|
||||
brand_code = state_data.get('brand_code')
|
||||
dropbox_path = getattr(job, 'dropbox_path', None)
|
||||
if brand_code and dropbox_path:
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
dropbox = get_dropbox_service()
|
||||
if dropbox.is_configured:
|
||||
base_name = f"{job.artist} - {job.title}"
|
||||
folder_name = f"{brand_code} - {base_name}"
|
||||
full_path = f"{dropbox_path}/{folder_name}"
|
||||
success = dropbox.delete_folder(full_path)
|
||||
results["dropbox"] = {
|
||||
"status": "success" if success else "failed",
|
||||
"path": full_path
|
||||
}
|
||||
else:
|
||||
results["dropbox"] = {"status": "failed", "reason": "Dropbox credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Dropbox for job {job_id}: {e}", exc_info=True)
|
||||
results["dropbox"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Clean up Google Drive
|
||||
gdrive_files = state_data.get('gdrive_files')
|
||||
if gdrive_files:
|
||||
try:
|
||||
from backend.services.gdrive_service import get_gdrive_service
|
||||
gdrive = get_gdrive_service()
|
||||
if gdrive.is_configured:
|
||||
# gdrive_files is a dict like {"mp4": "file_id", "mp4_720p": "file_id", "cdg": "file_id"}
|
||||
file_ids = list(gdrive_files.values()) if isinstance(gdrive_files, dict) else []
|
||||
delete_results = gdrive.delete_files(file_ids)
|
||||
all_success = all(delete_results.values())
|
||||
results["gdrive"] = {
|
||||
"status": "success" if all_success else "partial",
|
||||
"files": delete_results
|
||||
}
|
||||
else:
|
||||
results["gdrive"] = {"status": "failed", "reason": "Google Drive credentials not configured"}
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up Google Drive for job {job_id}: {e}", exc_info=True)
|
||||
results["gdrive"] = {"status": "error", "error": str(e)}
|
||||
|
||||
# Delete the job if requested
|
||||
if delete_job:
|
||||
try:
|
||||
job_manager.delete_job(job_id, delete_files=True)
|
||||
results["job_deleted"] = True
|
||||
logger.info(f"Deleted job {job_id} after distribution cleanup")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}", exc_info=True)
|
||||
results["job_deleted"] = False
|
||||
results["job_delete_error"] = str(e)
|
||||
|
||||
return results
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
Push Notification API routes.
|
||||
|
||||
Provides endpoints for managing Web Push notification subscriptions:
|
||||
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
|
||||
- POST /api/push/subscribe: Register a push subscription
|
||||
- POST /api/push/unsubscribe: Remove a push subscription
|
||||
- GET /api/push/subscriptions: List user's subscriptions
|
||||
- POST /api/push/test: Send a test notification (admin only)
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/push", tags=["push"])
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
|
||||
class VapidPublicKeyResponse(BaseModel):
|
||||
"""Response containing VAPID public key."""
|
||||
enabled: bool
|
||||
vapid_public_key: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeRequest(BaseModel):
|
||||
"""Request to subscribe to push notifications."""
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
"""Response after subscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnsubscribeRequest(BaseModel):
|
||||
"""Request to unsubscribe from push notifications."""
|
||||
endpoint: str
|
||||
|
||||
|
||||
class UnsubscribeResponse(BaseModel):
|
||||
"""Response after unsubscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class SubscriptionInfo(BaseModel):
|
||||
"""Information about a push subscription."""
|
||||
endpoint: str
|
||||
device_name: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_used_at: Optional[str] = None
|
||||
|
||||
|
||||
class SubscriptionsListResponse(BaseModel):
|
||||
"""Response containing user's subscriptions."""
|
||||
subscriptions: List[SubscriptionInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
"""Request to send a test notification."""
|
||||
title: Optional[str] = "Test Notification"
|
||||
body: Optional[str] = "This is a test push notification from Karaoke Generator"
|
||||
|
||||
|
||||
class TestNotificationResponse(BaseModel):
|
||||
"""Response after sending test notification."""
|
||||
status: str
|
||||
sent_count: int
|
||||
message: str
|
||||
|
||||
|
||||
# Routes
|
||||
|
||||
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
|
||||
async def get_vapid_public_key():
|
||||
"""
|
||||
Get the VAPID public key for push subscription.
|
||||
|
||||
This endpoint is public - no authentication required.
|
||||
Returns the public key needed for client-side PushManager.subscribe().
|
||||
"""
|
||||
settings = get_settings()
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
if not settings.enable_push_notifications:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
public_key = push_service.get_public_key()
|
||||
if not public_key:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
return VapidPublicKeyResponse(
|
||||
enabled=True,
|
||||
vapid_public_key=public_key
|
||||
)
|
||||
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
request: SubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Register a push notification subscription for the current user.
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in request.keys or "auth" not in request.keys:
|
||||
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint,
|
||||
keys=request.keys,
|
||||
device_name=request.device_name
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail="Failed to save subscription")
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription registered successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Remove a push notification subscription.
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
success = await push_service.remove_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint
|
||||
)
|
||||
|
||||
if not success:
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Subscription removed (or was not found)"
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription removed successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
List all push notification subscriptions for the current user.
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
|
||||
|
||||
return SubscriptionsListResponse(
|
||||
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
|
||||
count=len(subscriptions)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Send a test push notification to the current user's devices.
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
sent_count = await push_service.send_push(
|
||||
user_email=auth_result.user_email,
|
||||
title=request.title or "Test Notification",
|
||||
body=request.body or "This is a test push notification",
|
||||
url="/app/",
|
||||
tag="test"
|
||||
)
|
||||
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=f"Test notification sent to {sent_count} device(s)"
|
||||
)
|
||||
@@ -0,0 +1,428 @@
|
||||
"""
|
||||
Admin API routes for rate limit management.
|
||||
|
||||
Handles:
|
||||
- Rate limit statistics and monitoring
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- User override management (whitelist/bypass permissions)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class RateLimitStatsResponse(BaseModel):
|
||||
"""Current rate limit statistics."""
|
||||
# Configuration
|
||||
jobs_per_day_limit: int
|
||||
youtube_uploads_per_day_limit: int
|
||||
beta_ip_per_day_limit: int
|
||||
rate_limiting_enabled: bool
|
||||
|
||||
# Current usage
|
||||
youtube_uploads_today: int
|
||||
youtube_uploads_remaining: int
|
||||
|
||||
# Blocklist stats
|
||||
disposable_domains_count: int
|
||||
blocked_emails_count: int
|
||||
blocked_ips_count: int
|
||||
|
||||
# User override stats
|
||||
total_overrides: int
|
||||
|
||||
|
||||
class UserRateLimitStatusResponse(BaseModel):
|
||||
"""Rate limit status for a specific user."""
|
||||
email: str
|
||||
jobs_today: int
|
||||
jobs_limit: int
|
||||
jobs_remaining: int
|
||||
has_bypass: bool
|
||||
custom_limit: Optional[int]
|
||||
bypass_reason: Optional[str]
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data."""
|
||||
disposable_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
updated_at: Optional[datetime]
|
||||
updated_by: Optional[str]
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
"""Request to add/remove a domain."""
|
||||
domain: str
|
||||
|
||||
|
||||
class EmailRequest(BaseModel):
|
||||
"""Request to add/remove an email."""
|
||||
email: str
|
||||
|
||||
|
||||
class IPRequest(BaseModel):
|
||||
"""Request to add/remove an IP."""
|
||||
ip_address: str
|
||||
|
||||
|
||||
class UserOverride(BaseModel):
|
||||
"""User override configuration."""
|
||||
email: str
|
||||
bypass_job_limit: bool
|
||||
custom_daily_job_limit: Optional[int]
|
||||
reason: str
|
||||
created_by: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class UserOverrideRequest(BaseModel):
|
||||
"""Request to set/update a user override."""
|
||||
bypass_job_limit: bool = False
|
||||
custom_daily_job_limit: Optional[int] = None
|
||||
reason: str
|
||||
|
||||
|
||||
class UserOverridesListResponse(BaseModel):
|
||||
"""List of all user overrides."""
|
||||
overrides: List[UserOverride]
|
||||
total: int
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Statistics Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats", response_model=RateLimitStatsResponse)
|
||||
async def get_rate_limit_stats(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns configuration values, current usage counts, and blocklist stats.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Get YouTube uploads today
|
||||
youtube_today = rate_limit_service.get_youtube_uploads_today()
|
||||
youtube_limit = settings.rate_limit_youtube_uploads_per_day
|
||||
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
|
||||
|
||||
# Get blocklist stats
|
||||
blocklist_stats = email_validation.get_blocklist_stats()
|
||||
|
||||
# Get override count
|
||||
overrides = rate_limit_service.get_all_overrides()
|
||||
|
||||
return RateLimitStatsResponse(
|
||||
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
|
||||
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
|
||||
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
|
||||
rate_limiting_enabled=settings.enable_rate_limiting,
|
||||
youtube_uploads_today=youtube_today,
|
||||
youtube_uploads_remaining=youtube_remaining,
|
||||
disposable_domains_count=blocklist_stats["disposable_domains_count"],
|
||||
blocked_emails_count=blocklist_stats["blocked_emails_count"],
|
||||
blocked_ips_count=blocklist_stats["blocked_ips_count"],
|
||||
total_overrides=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
|
||||
async def get_user_rate_limit_status(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get rate limit status for a specific user.
|
||||
|
||||
Returns their current usage and any override settings.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
jobs_today = rate_limit_service.get_user_job_count_today(email)
|
||||
override = rate_limit_service.get_user_override(email)
|
||||
|
||||
if override:
|
||||
has_bypass = override.get("bypass_job_limit", False)
|
||||
custom_limit = override.get("custom_daily_job_limit")
|
||||
bypass_reason = override.get("reason")
|
||||
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
|
||||
else:
|
||||
has_bypass = False
|
||||
custom_limit = None
|
||||
bypass_reason = None
|
||||
jobs_limit = settings.rate_limit_jobs_per_day
|
||||
|
||||
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
|
||||
|
||||
return UserRateLimitStatusResponse(
|
||||
email=email,
|
||||
jobs_today=jobs_today,
|
||||
jobs_limit=jobs_limit,
|
||||
jobs_remaining=jobs_remaining,
|
||||
has_bypass=has_bypass,
|
||||
custom_limit=custom_limit,
|
||||
bypass_reason=bypass_reason,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/blocklists", response_model=BlocklistsResponse)
|
||||
async def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get all blocklist data.
|
||||
|
||||
Returns disposable domains, blocked emails, and blocked IPs.
|
||||
"""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Force refresh to get latest data
|
||||
config = email_validation.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Get the raw document for metadata
|
||||
from google.cloud import firestore
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
|
||||
|
||||
db = get_firestore_client()
|
||||
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
|
||||
|
||||
updated_at = None
|
||||
updated_by = None
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
updated_at = data.get("updated_at")
|
||||
updated_by = data.get("updated_by")
|
||||
|
||||
return BlocklistsResponse(
|
||||
disposable_domains=sorted(list(config["disposable_domains"])),
|
||||
blocked_emails=sorted(list(config["blocked_emails"])),
|
||||
blocked_ips=sorted(list(config["blocked_ips"])),
|
||||
updated_at=updated_at,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
async def add_disposable_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail="Invalid domain format")
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' added to disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_disposable_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' removed from disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
|
||||
async def add_blocked_email(
|
||||
request: EmailRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an email to the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' added to blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
|
||||
async def remove_blocked_email(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Email not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' removed from blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
|
||||
async def add_blocked_ip(
|
||||
request: IPRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail="Invalid IP address")
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' added to blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
|
||||
async def remove_blocked_ip(
|
||||
ip_address: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="IP not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' removed from blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# User Override Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/overrides", response_model=UserOverridesListResponse)
|
||||
async def get_all_overrides(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all user overrides."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
overrides_data = rate_limit_service.get_all_overrides()
|
||||
|
||||
overrides = [
|
||||
UserOverride(
|
||||
email=email,
|
||||
bypass_job_limit=data.get("bypass_job_limit", False),
|
||||
custom_daily_job_limit=data.get("custom_daily_job_limit"),
|
||||
reason=data.get("reason", ""),
|
||||
created_by=data.get("created_by", "unknown"),
|
||||
created_at=data.get("created_at", datetime.now(timezone.utc)),
|
||||
)
|
||||
for email, data in overrides_data.items()
|
||||
]
|
||||
|
||||
return UserOverridesListResponse(
|
||||
overrides=overrides,
|
||||
total=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def set_user_override(
|
||||
email: str,
|
||||
request: UserOverrideRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Set or update a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
if not request.reason or len(request.reason.strip()) < 3:
|
||||
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
|
||||
|
||||
rate_limit_service.set_user_override(
|
||||
user_email=email,
|
||||
bypass_job_limit=request.bypass_job_limit,
|
||||
custom_daily_job_limit=request.custom_daily_job_limit,
|
||||
reason=request.reason,
|
||||
admin_email=auth_result.user_email,
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override set for user '{email}'"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def remove_user_override(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Override not found for user")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override removed for user '{email}'"
|
||||
)
|
||||
+129
-40
@@ -27,8 +27,10 @@ from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from lyrics_transcriber.types import CorrectionResult
|
||||
@@ -385,28 +387,38 @@ async def generate_preview_video(
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Generating preview video")
|
||||
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
|
||||
# Check if user wants theme background image (slower) or black background (faster, default)
|
||||
use_background_image = updated_data.get("use_background_image", False)
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE: {use_gce_preview}, background image: {use_background_image})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id}) as span:
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
@@ -416,19 +428,19 @@ async def generate_preview_video(
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
@@ -439,54 +451,131 @@ async def generate_preview_video(
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview video using CorrectionOperations (heavy operation)
|
||||
with create_span("render-preview-video") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("render_complete")
|
||||
|
||||
# 7. Upload preview video to GCS for serving
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
# Only use background image if user explicitly requested it
|
||||
# Default is black background for faster preview generation (~10s vs ~30-60s)
|
||||
background_image_gcs_path = None
|
||||
if use_background_image:
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
gce_span.set_attribute("use_background_image", use_background_image)
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
"""
|
||||
Tenant API routes for white-label portal configuration.
|
||||
|
||||
These endpoints are public (no auth required) since the frontend needs
|
||||
to fetch tenant branding before the user logs in.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
|
||||
|
||||
|
||||
@router.get("/config", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config(
|
||||
request: Request,
|
||||
tenant: Optional[str] = Query(
|
||||
None,
|
||||
description="Tenant ID override (for development). If not provided, detected from subdomain.",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Get tenant configuration for the current portal.
|
||||
|
||||
This endpoint detects the tenant from:
|
||||
1. Query parameter `tenant` (for development/testing)
|
||||
2. X-Tenant-ID header (set by frontend)
|
||||
3. Host header subdomain
|
||||
|
||||
Returns the public tenant configuration (branding, features, defaults)
|
||||
or indicates this is the default Nomad Karaoke portal.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
|
||||
# Priority 1: Query parameter (for dev/testing)
|
||||
tenant_id = tenant
|
||||
|
||||
# Priority 2: X-Tenant-ID header
|
||||
if not tenant_id:
|
||||
tenant_id = request.headers.get("X-Tenant-ID")
|
||||
|
||||
# Priority 3: Detect from Host header
|
||||
if not tenant_id:
|
||||
host = request.headers.get("Host", "")
|
||||
# Check if this is a tenant subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
if host and "nomadkaraoke.com" in host.lower():
|
||||
parts = host.lower().split(".")
|
||||
# Skip known non-tenant subdomains
|
||||
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
|
||||
potential_tenant = parts[0]
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
tenant_id = potential_tenant
|
||||
|
||||
# If no tenant detected, return default config
|
||||
if not tenant_id:
|
||||
logger.debug("No tenant detected, returning default config")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Load tenant config
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
logger.warning(f"Tenant not found: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
logger.info(f"Returning config for tenant: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config_by_id(tenant_id: str):
|
||||
"""
|
||||
Get tenant configuration by explicit tenant ID.
|
||||
|
||||
This is useful for admin tools or debugging.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Check if tenant is active
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/asset/{tenant_id}/{asset_name}")
|
||||
async def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
This redirects to the signed GCS URL for the asset.
|
||||
"""
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
tenant_service = get_tenant_service()
|
||||
url = tenant_service.get_asset_url(tenant_id, asset_name)
|
||||
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail="Asset not found")
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
@@ -0,0 +1,162 @@
|
||||
"""
|
||||
Theme API routes for listing and retrieving theme information.
|
||||
|
||||
Themes are pre-created style configurations that users can select when creating jobs.
|
||||
These endpoints are public (no authentication required) to allow theme selection
|
||||
before job creation.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
Returns a list of themes with their metadata and preview image URLs.
|
||||
Preview URLs are signed and valid for 60 minutes.
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
return ThemesListResponse(themes=themes)
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
Returns the full theme details including style parameters.
|
||||
This can be used to preview theme settings or inspect configuration.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
ThemeDetailResponse with full theme details
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
This endpoint is useful when you need just the preview URL
|
||||
without loading the full theme details.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with preview_url field
|
||||
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Some themes include a default YouTube video description template
|
||||
that can be used when uploading videos.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with description field containing the template text,
|
||||
or null if no template is available
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
return {"description": description}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
File diff suppressed because it is too large.
Load diff
+54
-2
@@ -40,7 +40,15 @@ class Settings(BaseSettings):
|
||||
# Processing
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
# Timeout for agentic correction in seconds. If correction takes longer, abort and
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
@@ -52,9 +60,29 @@ class Settings(BaseSettings):
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
@@ -73,13 +101,37 @@ class Settings(BaseSettings):
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
|
||||
# Rate Limiting Configuration
|
||||
# Enable/disable rate limiting system-wide (useful for development)
|
||||
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
|
||||
# Maximum jobs a user can create per day (0 = unlimited)
|
||||
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
|
||||
# Maximum YouTube uploads system-wide per day (0 = unlimited)
|
||||
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
|
||||
# Maximum beta enrollments from same IP per day (0 = unlimited)
|
||||
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
|
||||
# Default CDG/TXT generation settings
|
||||
# When True, CDG and TXT packages are generated by default (when a theme is set)
|
||||
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
|
||||
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
|
||||
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Push Notifications Configuration
|
||||
# When enabled, users can subscribe to push notifications for job status updates
|
||||
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
|
||||
# Maximum number of push subscriptions per user (oldest removed when exceeded)
|
||||
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
|
||||
# VAPID subject (email or URL for push service to contact)
|
||||
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""
|
||||
Custom exceptions for the karaoke generation backend.
|
||||
|
||||
These exceptions are used for structured error handling across the application,
|
||||
particularly for rate limiting and validation errors.
|
||||
"""
|
||||
|
||||
|
||||
class RateLimitExceededError(Exception):
|
||||
"""
|
||||
Raised when a rate limit is exceeded.
|
||||
|
||||
Includes information about when the limit will reset for client retry logic.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
|
||||
remaining_seconds: Seconds until the limit resets (for Retry-After header)
|
||||
current_count: Current usage count
|
||||
limit_value: The limit that was exceeded
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
limit_type: str = "unknown",
|
||||
remaining_seconds: int = 0,
|
||||
current_count: int = 0,
|
||||
limit_value: int = 0
|
||||
):
|
||||
self.message = message
|
||||
self.limit_type = limit_type
|
||||
self.remaining_seconds = remaining_seconds
|
||||
self.current_count = current_count
|
||||
self.limit_value = limit_value
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class EmailValidationError(Exception):
|
||||
"""
|
||||
Raised when email validation fails.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, reason: str = "invalid"):
|
||||
self.message = message
|
||||
self.reason = reason
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class IPBlockedError(Exception):
|
||||
"""
|
||||
Raised when a request comes from a blocked IP address.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
ip_address: The blocked IP address (may be partially masked)
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, ip_address: str = ""):
|
||||
self.message = message
|
||||
self.ip_address = ip_address
|
||||
super().__init__(message)
|
||||
+63
-2
@@ -7,9 +7,14 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
from backend.services.nltk_preloader import preload_all_nltk_resources
|
||||
from backend.services.langfuse_preloader import preload_langfuse_handler
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
@@ -66,7 +71,28 @@ async def lifespan(app: FastAPI):
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
|
||||
# Preload NLP models and resources to avoid cold start delays
|
||||
# See docs/archive/2026-01-08-performance-investigation.md for background
|
||||
|
||||
# 1. SpaCy model (60+ second delay without preload)
|
||||
try:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
except Exception as e:
|
||||
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
||||
|
||||
# 2. NLTK cmudict (50-100+ second delay without preload)
|
||||
try:
|
||||
preload_all_nltk_resources()
|
||||
except Exception as e:
|
||||
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
||||
|
||||
# 3. Langfuse callback handler (200+ second delay without preload)
|
||||
try:
|
||||
preload_langfuse_handler()
|
||||
except Exception as e:
|
||||
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
@@ -100,6 +126,12 @@ app.add_middleware(
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Add tenant detection middleware (extracts tenant from subdomain/headers)
|
||||
app.add_middleware(TenantMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
@@ -108,6 +140,35 @@ app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
||||
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
|
||||
|
||||
# Exception handler for rate limiting
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
|
||||
|
||||
@app.exception_handler(RateLimitExceededError)
|
||||
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
|
||||
"""Handle rate limit exceeded errors with 429 status."""
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"detail": exc.message,
|
||||
"limit_type": exc.limit_type,
|
||||
"current_count": exc.current_count,
|
||||
"limit_value": exc.limit_value,
|
||||
},
|
||||
headers={
|
||||
"Retry-After": str(exc.remaining_seconds),
|
||||
} if exc.remaining_seconds > 0 else None,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
|
||||
|
||||
__all__ = [
|
||||
"AuditLoggingMiddleware",
|
||||
"TenantMiddleware",
|
||||
"get_tenant_from_request",
|
||||
"get_tenant_config_from_request",
|
||||
]
|
||||
@@ -0,0 +1,124 @@
|
||||
"""
|
||||
Audit logging middleware for tracking all HTTP requests.
|
||||
|
||||
This middleware captures request metadata and logs it to Cloud Logging
|
||||
for audit trail and user activity investigation purposes.
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
logger = logging.getLogger("audit")
|
||||
|
||||
# Endpoints to exclude from audit logging (high-frequency health checks)
|
||||
EXCLUDED_PATHS = {
|
||||
"/",
|
||||
"/api/health",
|
||||
"/api/health/detailed",
|
||||
"/api/readiness",
|
||||
"/healthz",
|
||||
"/ready",
|
||||
}
|
||||
|
||||
|
||||
class AuditLoggingMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that logs all HTTP requests for audit purposes.
|
||||
|
||||
Captures:
|
||||
- request_id: UUID for correlating with auth logs
|
||||
- method: HTTP method
|
||||
- path: Request path
|
||||
- status_code: Response status
|
||||
- latency_ms: Request duration
|
||||
- client_ip: Client IP (from X-Forwarded-For for proxied requests)
|
||||
- user_agent: Browser/client identifier
|
||||
|
||||
The request_id is stored in request.state and added to response headers,
|
||||
allowing correlation with auth logs that capture user_email.
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Skip excluded paths (health checks)
|
||||
if request.url.path in EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
# Generate unique request ID for correlation
|
||||
request_id = str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
# Capture start time
|
||||
start_time = time.time()
|
||||
|
||||
# Extract client info
|
||||
client_ip = self._get_client_ip(request)
|
||||
user_agent = request.headers.get("user-agent", "")
|
||||
|
||||
# Process request
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
# Log failed requests too (exception() auto-includes stack trace)
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
logger.exception(
|
||||
"request_audit_error",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate latency
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Log audit entry (INFO level for successful requests)
|
||||
log_level = logging.WARNING if response.status_code >= 400 else logging.INFO
|
||||
logger.log(
|
||||
log_level,
|
||||
"request_audit",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"status_code": response.status_code,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
|
||||
# Add request_id to response headers for debugging/correlation
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
return response
|
||||
|
||||
def _get_client_ip(self, request: Request) -> str:
|
||||
"""
|
||||
Extract client IP address, handling proxy scenarios.
|
||||
|
||||
Cloud Run and other proxies set X-Forwarded-For header.
|
||||
"""
|
||||
# Check X-Forwarded-For for proxy scenarios (Cloud Run, load balancers)
|
||||
forwarded = request.headers.get("x-forwarded-for", "")
|
||||
if forwarded:
|
||||
# First IP is the original client
|
||||
return forwarded.split(",")[0].strip()
|
||||
|
||||
# Fall back to direct connection
|
||||
if request.client:
|
||||
return request.client.host
|
||||
|
||||
return "unknown"
|
||||
@@ -0,0 +1,192 @@
|
||||
"""
|
||||
Tenant detection middleware for white-label B2B portals.
|
||||
|
||||
This middleware extracts the tenant ID from incoming requests and attaches
|
||||
the tenant configuration to the request state for use by downstream handlers.
|
||||
|
||||
Tenant detection priority:
|
||||
1. X-Tenant-ID header (explicitly set by frontend)
|
||||
2. `tenant` query parameter (for development/testing)
|
||||
3. Host header subdomain detection (production flow)
|
||||
|
||||
The middleware is non-blocking - if no tenant is detected, the request
|
||||
proceeds as a default Nomad Karaoke request (tenant_id = None).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.models.tenant import TenantConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Only allow query param tenant override in non-production environments
|
||||
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
|
||||
os.environ.get("ENVIRONMENT", "").lower() == "production"
|
||||
|
||||
|
||||
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
|
||||
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
|
||||
|
||||
|
||||
class TenantMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that detects tenant from request and attaches config to state.
|
||||
|
||||
After this middleware runs, routes can access:
|
||||
- request.state.tenant_id: str or None
|
||||
- request.state.tenant_config: TenantConfig or None
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = getattr(request.state, "tenant_id", None)
|
||||
tenant_config = getattr(request.state, "tenant_config", None)
|
||||
```
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Extract tenant ID from request
|
||||
tenant_id = self._extract_tenant_id(request)
|
||||
|
||||
# Load tenant config if tenant detected
|
||||
tenant_config: Optional[TenantConfig] = None
|
||||
if tenant_id:
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(tenant_id)
|
||||
|
||||
if tenant_config and not tenant_config.is_active:
|
||||
# Tenant exists but is inactive - treat as default
|
||||
logger.warning(f"Inactive tenant requested: {tenant_id}")
|
||||
tenant_id = None
|
||||
tenant_config = None
|
||||
|
||||
# Attach to request state
|
||||
request.state.tenant_id = tenant_id
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log tenant detection for debugging
|
||||
if tenant_id:
|
||||
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
|
||||
|
||||
# Process request
|
||||
response = await call_next(request)
|
||||
|
||||
# Add tenant ID to response headers for debugging
|
||||
if tenant_id:
|
||||
response.headers["X-Tenant-ID"] = tenant_id
|
||||
|
||||
return response
|
||||
|
||||
def _extract_tenant_id(self, request: Request) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from request using priority-based detection.
|
||||
|
||||
Returns:
|
||||
Tenant ID if detected, None for default Nomad Karaoke
|
||||
"""
|
||||
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
|
||||
header_tenant = request.headers.get("X-Tenant-ID")
|
||||
if header_tenant:
|
||||
return header_tenant.lower().strip()
|
||||
|
||||
# Priority 2: Query parameter (for development/testing only)
|
||||
# Disabled in production to prevent tenant spoofing
|
||||
if not IS_PRODUCTION:
|
||||
query_tenant = request.query_params.get("tenant")
|
||||
if query_tenant:
|
||||
return query_tenant.lower().strip()
|
||||
|
||||
# Priority 3: Host header subdomain detection
|
||||
host = request.headers.get("Host", "")
|
||||
return self._extract_tenant_from_host(host)
|
||||
|
||||
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from Host header subdomain.
|
||||
|
||||
Patterns handled:
|
||||
- vocalstar.nomadkaraoke.com -> vocalstar
|
||||
- vocalstar.gen.nomadkaraoke.com -> vocalstar
|
||||
- localhost:3000 -> None (local dev)
|
||||
- gen.nomadkaraoke.com -> None (main app)
|
||||
- api.nomadkaraoke.com -> None (API)
|
||||
|
||||
Returns:
|
||||
Tenant ID if subdomain matches tenant, None otherwise
|
||||
"""
|
||||
if not host:
|
||||
return None
|
||||
|
||||
# Normalize
|
||||
host_lower = host.lower()
|
||||
|
||||
# Remove port if present
|
||||
if ":" in host_lower:
|
||||
host_lower = host_lower.split(":")[0]
|
||||
|
||||
# Check if this is a nomadkaraoke.com domain
|
||||
if "nomadkaraoke.com" not in host_lower:
|
||||
return None
|
||||
|
||||
# Split into parts
|
||||
parts = host_lower.split(".")
|
||||
|
||||
# Need at least 3 parts: subdomain.nomadkaraoke.com
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
|
||||
# First part is the potential tenant ID
|
||||
potential_tenant = parts[0]
|
||||
|
||||
# Skip known non-tenant subdomains
|
||||
if potential_tenant in NON_TENANT_SUBDOMAINS:
|
||||
return None
|
||||
|
||||
# Verify tenant exists in GCS
|
||||
tenant_service = get_tenant_service()
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
return potential_tenant
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_tenant_from_request(request: Request) -> Optional[str]:
|
||||
"""
|
||||
Helper function to get tenant ID from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_id", None)
|
||||
|
||||
|
||||
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Helper function to get tenant config from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
config = get_tenant_config_from_request(request)
|
||||
if config and not config.features.audio_search:
|
||||
raise HTTPException(403, "Audio search not available")
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_config", None)
|
||||
+70
-8
@@ -9,6 +9,8 @@ from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from karaoke_gen.utils import normalize_text
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
@@ -204,8 +206,23 @@ class Job(BaseModel):
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
|
||||
|
||||
# Style configuration (uploaded files)
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (uploaded files - used when theme_id is not set)
|
||||
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -262,7 +279,16 @@ class Job(BaseModel):
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Output deletion tracking (for admin cleanup without deleting job)
|
||||
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
|
||||
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
@@ -408,8 +434,20 @@ class JobCreate(BaseModel):
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Style configuration (will be populated after file upload)
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (will be populated after file upload, or from theme)
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
@@ -448,7 +486,12 @@ class JobCreate(BaseModel):
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -460,14 +503,33 @@ class JobCreate(BaseModel):
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
|
||||
# Tenant scoping for white-label portals
|
||||
tenant_id: Optional[str] = None # Tenant ID for job scoping
|
||||
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
"""Validate URL is not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_artist_title(cls, v):
|
||||
"""Normalize artist/title text to standardize Unicode characters.
|
||||
|
||||
This ensures consistent data storage by converting:
|
||||
- Curly quotes -> straight quotes
|
||||
- Various dashes -> hyphen
|
||||
- Unusual whitespace -> regular space
|
||||
"""
|
||||
if v is not None and isinstance(v, str):
|
||||
if not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
# normalize_text handles stripping and Unicode normalization
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
|
||||
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
@@ -0,0 +1,208 @@
|
||||
"""
|
||||
Tenant data models for white-label B2B portals.
|
||||
|
||||
Tenants are B2B customers who get their own branded portal with custom
|
||||
configuration. Each tenant has:
|
||||
- Custom branding (logo, colors, site title)
|
||||
- Feature flags (enable/disable audio search, distribution, etc.)
|
||||
- Default settings (theme, distribution mode)
|
||||
- Auth configuration (allowed email domains, fixed tokens)
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class TenantBranding(BaseModel):
|
||||
"""Branding configuration for a tenant's portal."""
|
||||
|
||||
logo_url: Optional[str] = Field(
|
||||
None, description="URL to tenant's logo image (PNG with transparency preferred)"
|
||||
)
|
||||
logo_height: int = Field(40, description="Logo height in pixels for header display")
|
||||
primary_color: str = Field(
|
||||
"#ff5bb8", description="Primary brand color (hex format)"
|
||||
)
|
||||
secondary_color: str = Field(
|
||||
"#8b5cf6", description="Secondary brand color (hex format)"
|
||||
)
|
||||
accent_color: Optional[str] = Field(
|
||||
None, description="Accent color for highlights (hex format)"
|
||||
)
|
||||
background_color: Optional[str] = Field(
|
||||
None, description="Custom background color (defaults to dark theme)"
|
||||
)
|
||||
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
|
||||
site_title: str = Field(
|
||||
"Karaoke Generator", description="Browser tab title"
|
||||
)
|
||||
tagline: Optional[str] = Field(
|
||||
None, description="Optional tagline shown below logo"
|
||||
)
|
||||
|
||||
|
||||
class TenantFeatures(BaseModel):
|
||||
"""Feature flags controlling what's available in the tenant's portal."""
|
||||
|
||||
# Audio input methods
|
||||
audio_search: bool = Field(
|
||||
True, description="Enable audio search (flacfetch integration)"
|
||||
)
|
||||
file_upload: bool = Field(True, description="Enable direct file upload")
|
||||
youtube_url: bool = Field(True, description="Enable YouTube URL input")
|
||||
|
||||
# Distribution options
|
||||
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
|
||||
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
|
||||
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
|
||||
|
||||
# Customization options
|
||||
theme_selection: bool = Field(
|
||||
True, description="Allow user to select theme (false = use default)"
|
||||
)
|
||||
color_overrides: bool = Field(
|
||||
True, description="Allow user to customize colors"
|
||||
)
|
||||
|
||||
# Output formats
|
||||
enable_cdg: bool = Field(True, description="Generate CDG format output")
|
||||
enable_4k: bool = Field(True, description="Generate 4K video output")
|
||||
|
||||
# Advanced features
|
||||
admin_access: bool = Field(
|
||||
False, description="Allow access to admin dashboard"
|
||||
)
|
||||
|
||||
|
||||
class TenantDefaults(BaseModel):
|
||||
"""Default settings applied to all jobs for this tenant."""
|
||||
|
||||
theme_id: Optional[str] = Field(
|
||||
None, description="Default theme ID (required if theme_selection=false)"
|
||||
)
|
||||
locked_theme: Optional[str] = Field(
|
||||
None,
|
||||
description="If set, users cannot change theme - always uses this theme ID"
|
||||
)
|
||||
distribution_mode: str = Field(
|
||||
"all",
|
||||
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
|
||||
)
|
||||
brand_prefix: Optional[str] = Field(
|
||||
None, description="Prefix for output filenames (e.g., 'VSTAR')"
|
||||
)
|
||||
youtube_description_template: Optional[str] = Field(
|
||||
None, description="Default YouTube description template"
|
||||
)
|
||||
|
||||
|
||||
class TenantAuth(BaseModel):
|
||||
"""Authentication configuration for a tenant."""
|
||||
|
||||
allowed_email_domains: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
|
||||
)
|
||||
require_email_domain: bool = Field(
|
||||
True,
|
||||
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
|
||||
)
|
||||
fixed_token_ids: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
|
||||
)
|
||||
sender_email: Optional[str] = Field(
|
||||
None,
|
||||
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
|
||||
)
|
||||
|
||||
|
||||
class TenantConfig(BaseModel):
|
||||
"""Complete configuration for a white-label tenant."""
|
||||
|
||||
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
|
||||
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
|
||||
subdomain: str = Field(
|
||||
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
|
||||
)
|
||||
is_active: bool = Field(True, description="Whether tenant portal is active")
|
||||
|
||||
branding: TenantBranding = Field(default_factory=TenantBranding)
|
||||
features: TenantFeatures = Field(default_factory=TenantFeatures)
|
||||
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
|
||||
auth: TenantAuth = Field(default_factory=TenantAuth)
|
||||
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
def get_sender_email(self) -> str:
|
||||
"""Get the email sender address for this tenant."""
|
||||
if self.auth.sender_email:
|
||||
return self.auth.sender_email
|
||||
# Default pattern: {tenant_id}@nomadkaraoke.com
|
||||
return f"{self.id}@nomadkaraoke.com"
|
||||
|
||||
def is_email_allowed(self, email: str) -> bool:
|
||||
"""Check if an email address is allowed for this tenant."""
|
||||
if not self.auth.allowed_email_domains:
|
||||
# No domain restrictions
|
||||
return True
|
||||
|
||||
email_lower = email.lower()
|
||||
for domain in self.auth.allowed_email_domains:
|
||||
if email_lower.endswith(f"@{domain.lower()}"):
|
||||
return True
|
||||
|
||||
return not self.auth.require_email_domain
|
||||
|
||||
|
||||
class TenantPublicConfig(BaseModel):
|
||||
"""
|
||||
Public tenant configuration returned to frontend.
|
||||
|
||||
This excludes sensitive auth details like token IDs.
|
||||
"""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
subdomain: str
|
||||
is_active: bool
|
||||
branding: TenantBranding
|
||||
features: TenantFeatures
|
||||
defaults: TenantDefaults
|
||||
# Auth info limited to what frontend needs
|
||||
allowed_email_domains: List[str] = Field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
|
||||
"""Create public config from full config."""
|
||||
return cls(
|
||||
id=config.id,
|
||||
name=config.name,
|
||||
subdomain=config.subdomain,
|
||||
is_active=config.is_active,
|
||||
branding=config.branding,
|
||||
features=config.features,
|
||||
defaults=TenantDefaults(
|
||||
theme_id=config.defaults.theme_id,
|
||||
locked_theme=config.defaults.locked_theme,
|
||||
distribution_mode=config.defaults.distribution_mode,
|
||||
# Don't expose brand_prefix or youtube_description_template
|
||||
),
|
||||
allowed_email_domains=config.auth.allowed_email_domains,
|
||||
)
|
||||
|
||||
|
||||
class TenantConfigResponse(BaseModel):
|
||||
"""Response from GET /api/tenant/config endpoint."""
|
||||
|
||||
tenant: Optional[TenantPublicConfig] = Field(
|
||||
None, description="Tenant config if found, null for default Nomad Karaoke"
|
||||
)
|
||||
is_default: bool = Field(
|
||||
True, description="True if using default Nomad Karaoke config"
|
||||
)
|
||||
@@ -0,0 +1,153 @@
|
||||
"""
|
||||
Theme data models for karaoke-gen.
|
||||
|
||||
Themes are pre-created style configurations stored in GCS that users can select
|
||||
when creating karaoke jobs. Each theme includes:
|
||||
- Style parameters (fonts, colors, backgrounds, layouts)
|
||||
- Asset files (images, fonts)
|
||||
- Preview images
|
||||
- Optional YouTube description template
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ThemeSummary(BaseModel):
|
||||
"""Summary of a theme for listing purposes."""
|
||||
|
||||
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
|
||||
name: str = Field(..., description="Human-readable theme name")
|
||||
description: str = Field(..., description="Brief description of the theme style")
|
||||
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
|
||||
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
|
||||
is_default: bool = Field(False, description="Whether this is the default theme")
|
||||
|
||||
|
||||
class ThemeMetadata(BaseModel):
|
||||
"""Metadata for a single theme in the registry."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
is_default: bool = False
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ThemeRegistry(BaseModel):
|
||||
"""Registry of all available themes, stored as _metadata.json in GCS."""
|
||||
|
||||
version: int = 1
|
||||
themes: List[ThemeMetadata] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ThemeDetail(BaseModel):
|
||||
"""Full theme details including style parameters."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
preview_url: Optional[str] = None
|
||||
is_default: bool = False
|
||||
style_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
has_youtube_description: bool = False
|
||||
|
||||
|
||||
class ColorOverrides(BaseModel):
|
||||
"""
|
||||
User-customizable color overrides that can be applied on top of a theme.
|
||||
|
||||
All colors are hex format (#RRGGBB). When applied:
|
||||
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
|
||||
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
|
||||
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
|
||||
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
|
||||
"""
|
||||
|
||||
artist_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for artist name on intro/end screens",
|
||||
)
|
||||
title_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for song title on intro/end screens",
|
||||
)
|
||||
sung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics being sung (highlighted)",
|
||||
)
|
||||
unsung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics not yet sung",
|
||||
)
|
||||
|
||||
def has_overrides(self) -> bool:
|
||||
"""Check if any color overrides are set."""
|
||||
return any(
|
||||
[
|
||||
self.artist_color,
|
||||
self.title_color,
|
||||
self.sung_lyrics_color,
|
||||
self.unsung_lyrics_color,
|
||||
]
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
class ThemesListResponse(BaseModel):
|
||||
"""Response from GET /api/themes endpoint."""
|
||||
|
||||
themes: List[ThemeSummary]
|
||||
|
||||
|
||||
class ThemeDetailResponse(BaseModel):
|
||||
"""Response from GET /api/themes/{theme_id} endpoint."""
|
||||
|
||||
theme: ThemeDetail
|
||||
|
||||
|
||||
# Utility functions for color conversion
|
||||
|
||||
|
||||
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
|
||||
"""
|
||||
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
|
||||
|
||||
ASS subtitle format uses "R, G, B, A" string format.
|
||||
|
||||
Args:
|
||||
hex_color: Hex color string like "#7070F7"
|
||||
alpha: Alpha value 0-255 (default 255 = opaque)
|
||||
|
||||
Returns:
|
||||
RGBA string like "112, 112, 247, 255"
|
||||
"""
|
||||
hex_color = hex_color.lstrip("#")
|
||||
r = int(hex_color[0:2], 16)
|
||||
g = int(hex_color[2:4], 16)
|
||||
b = int(hex_color[4:6], 16)
|
||||
return f"{r}, {g}, {b}, {alpha}"
|
||||
|
||||
|
||||
def rgba_to_hex(rgba_str: str) -> str:
|
||||
"""
|
||||
Convert RGBA format to hex color.
|
||||
|
||||
Args:
|
||||
rgba_str: RGBA string like "112, 112, 247, 255"
|
||||
|
||||
Returns:
|
||||
Hex color string like "#7070F7"
|
||||
"""
|
||||
parts = [int(x.strip()) for x in rgba_str.split(",")]
|
||||
r, g, b = parts[0], parts[1], parts[2]
|
||||
return f"#{r:02x}{g:02x}{b:02x}"
|
||||
@@ -0,0 +1,290 @@
|
||||
"""
|
||||
User model for authentication and credits.
|
||||
|
||||
Supports:
|
||||
- Magic link authentication (email-based)
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Optional, List, Dict
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class UserRole(str, Enum):
|
||||
"""User roles for access control."""
|
||||
USER = "user"
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
amount: int # Positive = add, negative = deduct
|
||||
reason: str # e.g., "purchase", "refund", "admin_grant", "job_creation"
|
||||
job_id: Optional[str] = None
|
||||
stripe_session_id: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class PushSubscription(BaseModel):
|
||||
"""
|
||||
Web Push subscription for a user's device.
|
||||
|
||||
Stores the push subscription endpoint and encryption keys needed
|
||||
to send push notifications to the user's browser/device.
|
||||
"""
|
||||
endpoint: str # Push service endpoint URL
|
||||
keys: Dict[str, str] # p256dh and auth keys for encryption
|
||||
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
last_used_at: Optional[datetime] = None # Last time a notification was sent
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
|
||||
For multi-tenant white-label portals, users are scoped to a tenant.
|
||||
Users with tenant_id=None are default Nomad Karaoke users.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
# Account state
|
||||
is_active: bool = True
|
||||
email_verified: bool = False
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
last_login_at: Optional[datetime] = None
|
||||
|
||||
# Credit history (last 100 transactions)
|
||||
credit_transactions: List[CreditTransaction] = Field(default_factory=list)
|
||||
|
||||
# Job tracking
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
# Push notification subscriptions (Web Push API)
|
||||
# Users can subscribe from multiple devices/browsers
|
||||
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
Magic link token for passwordless authentication.
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
|
||||
For multi-tenant portals, the tenant_id determines which portal
|
||||
the user will be redirected to after verification.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
used: bool = False
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
User session for authenticated requests.
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
|
||||
For multi-tenant portals, sessions are scoped to a tenant.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
"""Response after sending magic link."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class VerifyMagicLinkRequest(BaseModel):
|
||||
"""Request to verify a magic link token."""
|
||||
token: str
|
||||
|
||||
|
||||
class VerifyMagicLinkResponse(BaseModel):
|
||||
"""Response after verifying magic link."""
|
||||
status: str
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
"""Public user information (safe to expose to frontend)."""
|
||||
email: str
|
||||
role: UserRole
|
||||
credits: int
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
"""Admin request to add credits to a user."""
|
||||
email: str
|
||||
amount: int
|
||||
reason: str = "admin_grant"
|
||||
|
||||
|
||||
class AddCreditsResponse(BaseModel):
|
||||
"""Response after adding credits."""
|
||||
status: str
|
||||
email: str
|
||||
credits_added: int
|
||||
new_balance: int
|
||||
message: str
|
||||
|
||||
|
||||
class UserListResponse(BaseModel):
|
||||
"""Response for listing users (admin only)."""
|
||||
users: List[UserPublic]
|
||||
total: int
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Beta Tester Program Models
|
||||
# ============================================================================
|
||||
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from a beta tester after using the service.
|
||||
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
"""
|
||||
id: str # Unique feedback ID
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
|
||||
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
status: str
|
||||
message: str
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
@@ -0,0 +1,164 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -276,20 +276,35 @@ class AudioSearchService:
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
if result_index < 0 or result_index >= len(self._cached_results):
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if we have local cached results
|
||||
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
|
||||
|
||||
# If no local cache but we have remote search_id, use remote download
|
||||
# This handles multi-instance deployments where the cache doesn't persist
|
||||
if not has_local_cache:
|
||||
if effective_search_id and self._remote_client:
|
||||
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
else:
|
||||
# Provide clear error message based on whether cache is empty or index out of bounds
|
||||
if len(self._cached_results) == 0:
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
"No cached results available. Run search() first."
|
||||
)
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
@@ -481,8 +496,12 @@ class AudioSearchService:
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
# Try to get local result for metadata, but it's optional
|
||||
# The remote service maintains its own cache by search_id
|
||||
result = None
|
||||
if result_index >= 0 and result_index < len(self._cached_results):
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
@@ -529,13 +548,15 @@ class AudioSearchService:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
|
||||
# Use local result metadata if available, otherwise use empty strings
|
||||
# (the actual download is handled by the remote service using search_id)
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist,
|
||||
title=result.title,
|
||||
provider=result.provider,
|
||||
quality=result.quality,
|
||||
artist=result.artist if result else "",
|
||||
title=result.title if result else "",
|
||||
provider=result.provider if result else "remote",
|
||||
quality=result.quality if result else "",
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
|
||||
@@ -3,50 +3,81 @@ Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- User tokens (stored in Firestore)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
- API keys for business users
|
||||
- Usage tracking and limits
|
||||
- Token management API
|
||||
|
||||
This is designed to be extensible for future Stripe integration
|
||||
and admin dashboard functionality.
|
||||
"""
|
||||
import logging
|
||||
import hashlib
|
||||
import time
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Admin email domains - users with these domains are automatically admin
|
||||
ADMIN_EMAIL_DOMAINS = ["nomadkaraoke.com"]
|
||||
|
||||
|
||||
class UserType(str, Enum):
|
||||
"""User access levels."""
|
||||
ADMIN = "admin" # Unlimited access, can manage tokens
|
||||
UNLIMITED = "unlimited" # Unlimited karaoke generation
|
||||
LIMITED = "limited" # Limited number of uses
|
||||
STRIPE = "stripe" # Paid access via Stripe (future)
|
||||
STRIPE = "stripe" # Paid access via Stripe
|
||||
API_KEY = "api_key" # Business API key access
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthResult:
|
||||
"""Result of authentication validation."""
|
||||
is_valid: bool
|
||||
user_type: UserType
|
||||
remaining_uses: int # -1 = unlimited, 0 = exhausted, >0 = remaining
|
||||
message: str
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
return iter((self.is_valid, self.user_type, self.remaining_uses, self.message))
|
||||
|
||||
|
||||
def is_admin_email(email: str) -> bool:
|
||||
"""Check if an email belongs to an admin domain."""
|
||||
if not email:
|
||||
return False
|
||||
email_lower = email.lower()
|
||||
return any(email_lower.endswith(f"@{domain}") for domain in ADMIN_EMAIL_DOMAINS)
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Service for authentication and token management."""
|
||||
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize auth service."""
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
# Get admin tokens from secret manager or env vars
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
|
||||
|
||||
if self.admin_tokens:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
@@ -78,10 +109,30 @@ class AuthService:
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
# Return their credits as remaining uses
|
||||
credits = user.credits
|
||||
logger.info(f"Session token validated for user {user.email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
# User has no credits but is authenticated
|
||||
return True, UserType.STRIPE, 0, f"Authenticated but no credits remaining"
|
||||
|
||||
return True, UserType.STRIPE, credits, f"Session valid: {credits} credits"
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return False, UserType.LIMITED, 0, "Invalid token"
|
||||
|
||||
# Check if token is active
|
||||
@@ -129,35 +180,319 @@ class AuthService:
|
||||
return True, token_type, -1, "Stripe access granted"
|
||||
|
||||
return False, UserType.LIMITED, 0, "Unknown token type"
|
||||
|
||||
|
||||
def validate_token_full(self, token: str) -> AuthResult:
|
||||
"""
|
||||
Validate an access token and return full authentication details.
|
||||
|
||||
This is the preferred method for new code as it returns complete
|
||||
authentication context including user email and admin status.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication details
|
||||
"""
|
||||
if not token:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="No token provided"
|
||||
)
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
# Admin tokens from env var are associated with the system admin email
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, _message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
credits = user.credits
|
||||
user_email = user.email
|
||||
|
||||
# Check if user is admin by email domain or role
|
||||
user_is_admin = is_admin_email(user_email) or user.role == UserRole.ADMIN
|
||||
|
||||
if user_is_admin:
|
||||
logger.info(f"Admin session validated for {user_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted (domain)",
|
||||
user_email=user_email,
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
logger.info(f"Session token validated for user {user_email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=0,
|
||||
message="Authenticated but no credits remaining",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=credits,
|
||||
message=f"Session valid: {credits} credits",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Invalid token"
|
||||
)
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType(token_data["type"]),
|
||||
remaining_uses=0,
|
||||
message="Token has been revoked"
|
||||
)
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
logger.warning("Auth token missing required user_email field")
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token configuration error: missing user_email. Please contact support."
|
||||
)
|
||||
|
||||
# Check if token's user is an admin (by email domain)
|
||||
token_is_admin = is_admin_email(token_user_email)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at and time.time() > expires_at:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
if token_type == UserType.API_KEY:
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Unknown token type"
|
||||
)
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
job_id: The job ID being created
|
||||
|
||||
|
||||
Returns:
|
||||
True if usage was tracked, False otherwise
|
||||
"""
|
||||
# Validate token first
|
||||
is_valid, user_type, remaining_uses, message = self.validate_token(token)
|
||||
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Cannot increment usage for invalid token: {message}")
|
||||
return False
|
||||
|
||||
|
||||
# Don't track usage for admin or unlimited tokens
|
||||
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
|
||||
logger.debug(f"Skipping usage tracking for {user_type} token")
|
||||
return True
|
||||
|
||||
|
||||
# For admin tokens (not in Firestore), no tracking needed
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Increment usage in Firestore
|
||||
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
self.firestore.increment_token_usage(token, job_id)
|
||||
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
"""
|
||||
Discord Notification Service.
|
||||
|
||||
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Posting messages to Discord webhooks
|
||||
- Video upload notifications
|
||||
- Validation of webhook URLs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DiscordNotificationService:
|
||||
"""
|
||||
Service for sending Discord webhook notifications.
|
||||
|
||||
Supports posting messages to Discord channels via webhooks,
|
||||
commonly used to notify when new karaoke videos are uploaded.
|
||||
"""
|
||||
|
||||
# Discord webhook URL pattern
|
||||
WEBHOOK_URL_PATTERN = re.compile(
|
||||
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
webhook_url: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the Discord notification service.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for posting notifications
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.webhook_url = webhook_url
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Validate webhook URL if provided
|
||||
if webhook_url:
|
||||
self._validate_webhook_url(webhook_url)
|
||||
|
||||
def _validate_webhook_url(self, url: str) -> None:
|
||||
"""
|
||||
Validate that the webhook URL is a valid Discord webhook URL.
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is not a valid Discord webhook URL
|
||||
"""
|
||||
url = url.strip()
|
||||
if not url.startswith("https://discord.com/api/webhooks/"):
|
||||
raise ValueError(
|
||||
f"Invalid Discord webhook URL: {url}. "
|
||||
"URL must start with 'https://discord.com/api/webhooks/'"
|
||||
)
|
||||
|
||||
def post_message(
|
||||
self,
|
||||
message: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a message to a Discord channel via webhook.
|
||||
|
||||
Args:
|
||||
message: The message content to post
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if message was posted successfully, False otherwise
|
||||
|
||||
Raises:
|
||||
ValueError: If no webhook URL is available
|
||||
requests.HTTPError: If the webhook request fails
|
||||
"""
|
||||
url = webhook_url or self.webhook_url
|
||||
if not url:
|
||||
raise ValueError("No Discord webhook URL provided")
|
||||
|
||||
url = url.strip()
|
||||
self._validate_webhook_url(url)
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would post Discord message: '{message}' to {url}"
|
||||
)
|
||||
return True
|
||||
|
||||
self.logger.info(f"Posting message to Discord webhook...")
|
||||
data = {"content": message}
|
||||
response = requests.post(url, json=data, timeout=30)
|
||||
response.raise_for_status()
|
||||
self.logger.info("Message posted to Discord successfully")
|
||||
return True
|
||||
|
||||
def post_video_notification(
|
||||
self,
|
||||
youtube_url: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a notification about a new video upload.
|
||||
|
||||
Args:
|
||||
youtube_url: The YouTube URL of the uploaded video
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if notification was posted successfully, False otherwise
|
||||
"""
|
||||
if not youtube_url:
|
||||
self.logger.info("Skipping Discord notification - no YouTube URL available")
|
||||
return False
|
||||
|
||||
message = f"New upload: {youtube_url}"
|
||||
return self.post_message(message, webhook_url)
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""
|
||||
Check if Discord notifications are enabled.
|
||||
|
||||
Returns:
|
||||
True if a valid webhook URL is configured
|
||||
"""
|
||||
return bool(self.webhook_url)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_discord_notification_service: Optional[DiscordNotificationService] = None
|
||||
|
||||
|
||||
def get_discord_notification_service(
|
||||
webhook_url: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> DiscordNotificationService:
|
||||
"""
|
||||
Get a Discord notification service instance.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for notifications
|
||||
**kwargs: Additional arguments passed to DiscordNotificationService
|
||||
|
||||
Returns:
|
||||
DiscordNotificationService instance
|
||||
"""
|
||||
global _discord_notification_service
|
||||
|
||||
# Create new instance if webhook URL changed or not yet created
|
||||
if _discord_notification_service is None or webhook_url:
|
||||
_discord_notification_service = DiscordNotificationService(
|
||||
webhook_url=webhook_url,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _discord_notification_service
|
||||
@@ -199,7 +199,7 @@ class DropboxService:
|
||||
|
||||
def upload_folder(self, local_dir: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload all files in a directory to Dropbox folder.
|
||||
Recursively upload all files and subdirectories to Dropbox folder.
|
||||
|
||||
Args:
|
||||
local_dir: Local directory to upload
|
||||
@@ -212,10 +212,17 @@ class DropboxService:
|
||||
logger.info(f"Uploading folder {local_dir} to {remote_path}")
|
||||
|
||||
uploaded_count = 0
|
||||
for filename in os.listdir(local_dir):
|
||||
local_file = os.path.join(local_dir, filename)
|
||||
if os.path.isfile(local_file):
|
||||
remote_file = f"{remote_path}/{filename}"
|
||||
for root, _dirs, files in os.walk(local_dir):
|
||||
# Calculate the relative path from local_dir to current root
|
||||
rel_root = os.path.relpath(root, local_dir)
|
||||
if rel_root == ".":
|
||||
current_remote = remote_path
|
||||
else:
|
||||
current_remote = f"{remote_path}/{rel_root}"
|
||||
|
||||
for filename in files:
|
||||
local_file = os.path.join(root, filename)
|
||||
remote_file = f"{current_remote}/{filename}"
|
||||
self.upload_file(local_file, remote_file)
|
||||
uploaded_count += 1
|
||||
|
||||
@@ -256,6 +263,38 @@ class DropboxService:
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
def delete_folder(self, path: str) -> bool:
|
||||
"""
|
||||
Delete a folder and all its contents from Dropbox.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Deleting Dropbox folder: {path}")
|
||||
|
||||
try:
|
||||
self.client.files_delete_v2(path)
|
||||
logger.info(f"Successfully deleted: {path}")
|
||||
return True
|
||||
except ApiError as e:
|
||||
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
|
||||
logger.warning(f"Folder not found (already deleted?): {path}")
|
||||
return True # Consider it success if already gone
|
||||
logger.error(f"Failed to delete Dropbox folder: {e}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting Dropbox folder: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,646 @@
|
||||
"""
|
||||
Email Validation Service.
|
||||
|
||||
Provides email validation, normalization, and blocklist checking
|
||||
to prevent abuse during beta enrollment and other flows.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Tuple, Set
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask an email address for privacy-safe logging."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 2:
|
||||
masked_local = "*" * len(local)
|
||||
else:
|
||||
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
|
||||
return f"{masked_local}@{domain}"
|
||||
|
||||
|
||||
# Firestore collection for blocklist data
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
BLOCKLIST_CONFIG_DOC = "config"
|
||||
|
||||
# Default disposable email domains (can be extended via admin UI)
|
||||
DEFAULT_DISPOSABLE_DOMAINS = {
|
||||
# Popular temporary email services
|
||||
"10minutemail.com",
|
||||
"10minutemail.net",
|
||||
"guerrillamail.com",
|
||||
"guerrillamail.org",
|
||||
"guerrillamail.net",
|
||||
"guerrillamail.biz",
|
||||
"guerrillamailblock.com",
|
||||
"tempmail.com",
|
||||
"tempmail.net",
|
||||
"temp-mail.org",
|
||||
"temp-mail.io",
|
||||
"throwawaymail.com",
|
||||
"mailinator.com",
|
||||
"mailinator.net",
|
||||
"mailinator.org",
|
||||
"mailinator.info",
|
||||
"maildrop.cc",
|
||||
"yopmail.com",
|
||||
"yopmail.fr",
|
||||
"yopmail.net",
|
||||
"fakeinbox.com",
|
||||
"fakemailgenerator.com",
|
||||
"dispostable.com",
|
||||
"getairmail.com",
|
||||
"getnada.com",
|
||||
"mohmal.com",
|
||||
"trashmail.com",
|
||||
"trashmail.net",
|
||||
"trashmail.org",
|
||||
"sharklasers.com",
|
||||
"grr.la",
|
||||
"guerrillamail.de",
|
||||
"pokemail.net",
|
||||
"spam4.me",
|
||||
"spamgourmet.com",
|
||||
"mytrashmail.com",
|
||||
"mailnesia.com",
|
||||
"mailcatch.com",
|
||||
"mintemail.com",
|
||||
"tempr.email",
|
||||
"tempail.com",
|
||||
"emailondeck.com",
|
||||
"incognitomail.com",
|
||||
"inboxalias.com",
|
||||
"33mail.com",
|
||||
"spamex.com",
|
||||
"spamfree24.org",
|
||||
"spamspot.com",
|
||||
"mailnull.com",
|
||||
"mailsac.com",
|
||||
"emailfake.com",
|
||||
"fakemail.fr",
|
||||
"tempinbox.com",
|
||||
"throwaway.email",
|
||||
"burnermail.io",
|
||||
"anonbox.net",
|
||||
"anonymbox.com",
|
||||
"discard.email",
|
||||
"discardmail.com",
|
||||
"discardmail.de",
|
||||
"mailexpire.com",
|
||||
"mailforspam.com",
|
||||
"meltmail.com",
|
||||
"mt2009.com",
|
||||
"mt2014.com",
|
||||
"nospam.ze.tc",
|
||||
"nospamfor.us",
|
||||
"nowmymail.com",
|
||||
"receiveee.com",
|
||||
"safe-mail.net",
|
||||
"spamavert.com",
|
||||
"spambob.com",
|
||||
"spambog.com",
|
||||
"spambox.us",
|
||||
"spamcannon.com",
|
||||
"spamcannon.net",
|
||||
"spamcon.org",
|
||||
"spamcorptastic.com",
|
||||
"spamday.com",
|
||||
"spamfree.eu",
|
||||
"spamherelots.com",
|
||||
"spamhereplease.com",
|
||||
"spamhole.com",
|
||||
"spamify.com",
|
||||
"spaminator.de",
|
||||
"spamkill.info",
|
||||
"spaml.com",
|
||||
"spaml.de",
|
||||
"spamoff.de",
|
||||
"spamobox.com",
|
||||
"spamslicer.com",
|
||||
"spamstack.net",
|
||||
"spamthis.co.uk",
|
||||
"spamthisplease.com",
|
||||
"supergreatmail.com",
|
||||
"suremail.info",
|
||||
"teleworm.us",
|
||||
"tempemail.co.za",
|
||||
"tempemail.net",
|
||||
"tempmailaddress.com",
|
||||
"tempmailo.com",
|
||||
"thankyou2010.com",
|
||||
"thisisnotmyrealemail.com",
|
||||
"tm.slsrs.ru",
|
||||
"tmpeml.info",
|
||||
"trash-mail.at",
|
||||
"trash-mail.de",
|
||||
"trash2009.com",
|
||||
"trashemail.de",
|
||||
"trashmail.at",
|
||||
"trashmailer.com",
|
||||
"wegwerfmail.de",
|
||||
"wegwerfmail.net",
|
||||
"wegwerfmail.org",
|
||||
"wh4f.org",
|
||||
"willhackforfood.biz",
|
||||
"willselfdestruct.com",
|
||||
"xmaily.com",
|
||||
"xyzfree.net",
|
||||
"yep.it",
|
||||
"yogamaven.com",
|
||||
"yuurok.com",
|
||||
"zehnminutenmail.de",
|
||||
"zippymail.info",
|
||||
# Additional common ones
|
||||
"mailnator.com",
|
||||
"bugmenot.com",
|
||||
"dodgeit.com",
|
||||
"dodgit.com",
|
||||
"e4ward.com",
|
||||
"emailsensei.com",
|
||||
"hushmail.com",
|
||||
"jetable.org",
|
||||
"kasmail.com",
|
||||
"mailblock.net",
|
||||
"mailcatch.com",
|
||||
"mymailoasis.com",
|
||||
"nervmich.net",
|
||||
"nervtmansen.de",
|
||||
"oneoffemail.com",
|
||||
"pookmail.com",
|
||||
"shortmail.net",
|
||||
"sneakemail.com",
|
||||
"sogetthis.com",
|
||||
"tempemailaddress.com",
|
||||
"tempomail.fr",
|
||||
"temporaryemail.net",
|
||||
"temporaryforwarding.com",
|
||||
"temporaryinbox.com",
|
||||
"thankyou2010.com",
|
||||
"tyldd.com",
|
||||
"uggsrock.com",
|
||||
"veryrealemail.com",
|
||||
"yourewronghereswhy.com",
|
||||
}
|
||||
|
||||
# Gmail-like domains that support alias normalization
|
||||
GMAIL_LIKE_DOMAINS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
}
|
||||
|
||||
|
||||
class EmailValidationService:
|
||||
"""
|
||||
Service for email validation and abuse prevention.
|
||||
|
||||
Features:
|
||||
- Email normalization (Gmail alias stripping)
|
||||
- Disposable domain detection
|
||||
- Blocklist checking (emails, IPs)
|
||||
- IP-based enrollment rate limiting
|
||||
"""
|
||||
|
||||
_instance: Optional["EmailValidationService"] = None
|
||||
_blocklist_cache: Optional[dict] = None
|
||||
_blocklist_cache_time: Optional[datetime] = None
|
||||
_cache_ttl_seconds: int = 300 # 5 minutes
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize the email validation service."""
|
||||
if db is None:
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
db = get_firestore_client()
|
||||
self.db = db
|
||||
|
||||
@classmethod
|
||||
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
|
||||
"""Get singleton instance of the service."""
|
||||
if cls._instance is None:
|
||||
cls._instance = cls(db)
|
||||
return cls._instance
|
||||
|
||||
def normalize_email(self, email: str) -> str:
|
||||
"""
|
||||
Normalize an email address.
|
||||
|
||||
For Gmail-like domains:
|
||||
- Removes dots from local part
|
||||
- Removes everything after + in local part
|
||||
- Converts to lowercase
|
||||
|
||||
For other domains:
|
||||
- Only converts to lowercase
|
||||
|
||||
Args:
|
||||
email: Email address to normalize
|
||||
|
||||
Returns:
|
||||
Normalized email address
|
||||
|
||||
Examples:
|
||||
j.o.h.n+spam@gmail.com -> john@gmail.com
|
||||
John.Doe@example.com -> john.doe@example.com
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return email.lower().strip() if email else ""
|
||||
|
||||
email = email.lower().strip()
|
||||
local, domain = email.rsplit("@", 1)
|
||||
|
||||
if domain in GMAIL_LIKE_DOMAINS:
|
||||
# Remove dots from local part
|
||||
local = local.replace(".", "")
|
||||
# Remove everything after +
|
||||
if "+" in local:
|
||||
local = local.split("+")[0]
|
||||
|
||||
return f"{local}@{domain}"
|
||||
|
||||
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
|
||||
"""
|
||||
Get blocklist configuration from Firestore.
|
||||
|
||||
Uses caching to reduce Firestore reads.
|
||||
|
||||
Returns:
|
||||
Dict with disposable_domains, blocked_emails, blocked_ips sets
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache validity
|
||||
if (
|
||||
not force_refresh
|
||||
and self._blocklist_cache is not None
|
||||
and self._blocklist_cache_time is not None
|
||||
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
|
||||
):
|
||||
return self._blocklist_cache
|
||||
|
||||
# Fetch from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
config = {
|
||||
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
|
||||
"blocked_emails": set(data.get("blocked_emails", [])),
|
||||
"blocked_ips": set(data.get("blocked_ips", [])),
|
||||
}
|
||||
else:
|
||||
# Use defaults if no config exists
|
||||
config = {
|
||||
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
|
||||
"blocked_emails": set(),
|
||||
"blocked_ips": set(),
|
||||
}
|
||||
|
||||
# Update cache
|
||||
EmailValidationService._blocklist_cache = config
|
||||
EmailValidationService._blocklist_cache_time = now
|
||||
|
||||
return config
|
||||
|
||||
def is_disposable_domain(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email uses a disposable domain.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the domain is known to be disposable
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False
|
||||
|
||||
domain = email.lower().split("@")[-1]
|
||||
config = self.get_blocklist_config()
|
||||
return domain in config["disposable_domains"]
|
||||
|
||||
def is_email_blocked(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is explicitly blocked.
|
||||
|
||||
Checks both the raw email and normalized version.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the email is blocked
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
email_lower = email.lower().strip()
|
||||
email_normalized = self.normalize_email(email)
|
||||
|
||||
return (
|
||||
email_lower in config["blocked_emails"]
|
||||
or email_normalized in config["blocked_emails"]
|
||||
)
|
||||
|
||||
def is_ip_blocked(self, ip_address: str) -> bool:
|
||||
"""
|
||||
Check if an IP address is blocked.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to check
|
||||
|
||||
Returns:
|
||||
True if the IP is blocked
|
||||
"""
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
return ip_address in config["blocked_ips"]
|
||||
|
||||
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
|
||||
"""
|
||||
Validate an email for beta enrollment.
|
||||
|
||||
Performs all validation checks:
|
||||
1. Basic format validation
|
||||
2. Disposable domain check
|
||||
3. Blocked email check
|
||||
|
||||
Args:
|
||||
email: Email address to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_message)
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False, "Invalid email format"
|
||||
|
||||
# Check disposable domain
|
||||
if self.is_disposable_domain(email):
|
||||
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
|
||||
return False, "Disposable email addresses are not allowed"
|
||||
|
||||
# Check blocked email
|
||||
if self.is_email_blocked(email):
|
||||
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
|
||||
return False, "This email address is not allowed"
|
||||
|
||||
return True, ""
|
||||
|
||||
def hash_ip(self, ip_address: str) -> str:
|
||||
"""
|
||||
Hash an IP address for privacy-preserving storage.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to hash
|
||||
|
||||
Returns:
|
||||
SHA-256 hash of the IP
|
||||
"""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management (Admin)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Add an email to the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
if not email:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
else:
|
||||
data = {}
|
||||
emails = set()
|
||||
|
||||
emails.add(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
|
||||
if email not in emails:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
emails.discard(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
else:
|
||||
data = {}
|
||||
ips = set()
|
||||
|
||||
ips.add(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
ips.discard(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def get_blocklist_stats(self) -> dict:
|
||||
"""Get statistics about current blocklists."""
|
||||
config = self.get_blocklist_config(force_refresh=True)
|
||||
return {
|
||||
"disposable_domains_count": len(config["disposable_domains"]),
|
||||
"blocked_emails_count": len(config["blocked_emails"]),
|
||||
"blocked_ips_count": len(config["blocked_ips"]),
|
||||
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
|
||||
}
|
||||
|
||||
|
||||
def get_email_validation_service(
|
||||
db: Optional[firestore.Client] = None,
|
||||
) -> EmailValidationService:
|
||||
"""Get the singleton EmailValidationService instance."""
|
||||
return EmailValidationService.get_instance(db)
|
||||
@@ -0,0 +1,454 @@
|
||||
"""
|
||||
Encoding Interface.
|
||||
|
||||
Defines the abstract interface for video encoding backends, allowing
|
||||
the video worker orchestrator to use either GCE or local encoding
|
||||
interchangeably.
|
||||
|
||||
This follows the Strategy pattern - different encoding implementations
|
||||
can be swapped without changing the orchestration logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingInput:
|
||||
"""
|
||||
Input configuration for video encoding.
|
||||
|
||||
Contains all the paths and metadata needed to encode a karaoke video.
|
||||
"""
|
||||
# Required input files
|
||||
title_video_path: str # Title card video (MOV)
|
||||
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
|
||||
instrumental_audio_path: str # Instrumental audio track (FLAC)
|
||||
|
||||
# Optional input files
|
||||
end_video_path: Optional[str] = None # End credits video (MOV)
|
||||
|
||||
# Metadata for output naming
|
||||
artist: str = ""
|
||||
title: str = ""
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingOutput:
|
||||
"""
|
||||
Output from video encoding.
|
||||
|
||||
Contains paths to all generated video files and status information.
|
||||
"""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Output file paths (relative to output_dir or absolute)
|
||||
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
|
||||
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
|
||||
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
|
||||
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
|
||||
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
|
||||
lossy_720p_mp4_path: Optional[str] = None # 720p web version
|
||||
|
||||
# All output files as a dict for convenience
|
||||
output_files: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
# Encoding metadata
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
encoding_backend: Optional[str] = None # "gce" or "local"
|
||||
|
||||
|
||||
class EncodingBackend(ABC):
|
||||
"""
|
||||
Abstract base class for video encoding backends.
|
||||
|
||||
Implementations:
|
||||
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
|
||||
- LocalEncodingBackend: Local FFmpeg-based encoding
|
||||
"""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Return the name of this encoding backend."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video files according to the input configuration.
|
||||
|
||||
This method should:
|
||||
1. Create karaoke video with instrumental audio
|
||||
2. Convert with-vocals video to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
|
||||
4. Encode lossy 4K MP4 with AAC audio
|
||||
5. Create MKV with FLAC audio for YouTube
|
||||
6. Encode 720p version for web
|
||||
|
||||
Args:
|
||||
input_config: EncodingInput with all required paths and options
|
||||
|
||||
Returns:
|
||||
EncodingOutput with paths to generated files and status
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def is_available(self) -> bool:
|
||||
"""
|
||||
Check if this encoding backend is available and configured.
|
||||
|
||||
Returns:
|
||||
True if the backend can be used, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of the encoding backend.
|
||||
|
||||
Returns:
|
||||
Dict with status information (availability, queue length, etc.)
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class LocalEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
Local FFmpeg-based encoding backend.
|
||||
|
||||
Wraps the LocalEncodingService to implement the EncodingBackend interface.
|
||||
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log operations without executing
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the local encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.local_encoding_service import LocalEncodingService
|
||||
self._service = LocalEncodingService(
|
||||
dry_run=self.dry_run,
|
||||
logger=self.logger
|
||||
)
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using local FFmpeg.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from backend.services.local_encoding_service import EncodingConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Build output file paths
|
||||
base_name = f"{input_config.artist} - {input_config.title}"
|
||||
output_dir = input_config.output_dir or "."
|
||||
|
||||
import os
|
||||
config = EncodingConfig(
|
||||
title_video=input_config.title_video_path,
|
||||
karaoke_video=input_config.karaoke_video_path,
|
||||
instrumental_audio=input_config.instrumental_audio_path,
|
||||
end_video=input_config.end_video_path,
|
||||
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
service = self._get_service()
|
||||
result = await asyncio.to_thread(service.encode_all_formats, config)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if result.success:
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
karaoke_mp4_path=config.output_karaoke_mp4,
|
||||
with_vocals_mp4_path=config.output_with_vocals_mp4,
|
||||
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
|
||||
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
|
||||
lossless_mkv_path=config.output_lossless_mkv,
|
||||
lossy_720p_mp4_path=config.output_720p_mp4,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
else:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=result.error,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if local encoding is available (FFmpeg installed)."""
|
||||
import subprocess
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(
|
||||
subprocess.run,
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
timeout=10
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get local encoding status."""
|
||||
available = await self.is_available()
|
||||
service = self._get_service()
|
||||
|
||||
return {
|
||||
"backend": self.name,
|
||||
"available": available,
|
||||
"hwaccel_available": service.hwaccel_available if available else False,
|
||||
"video_encoder": service.video_encoder if available else None,
|
||||
}
|
||||
|
||||
|
||||
class GCEEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
GCE-based encoding backend.
|
||||
|
||||
Wraps the existing EncodingService to implement the EncodingBackend interface.
|
||||
Submits jobs to a remote GCE worker for high-performance encoding.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the GCE encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run # Stored but not used (GCE is remote)
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "gce"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the GCE encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
self._service = get_encoding_service()
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using GCE worker.
|
||||
|
||||
Note: GCE encoding requires files to be in GCS, so this method
|
||||
expects input_config.options to contain 'input_gcs_path' and
|
||||
'output_gcs_path'.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
service = self._get_service()
|
||||
|
||||
# GCE requires GCS paths
|
||||
input_gcs_path = input_config.options.get("input_gcs_path")
|
||||
output_gcs_path = input_config.options.get("output_gcs_path")
|
||||
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
|
||||
|
||||
if not input_gcs_path or not output_gcs_path:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
try:
|
||||
# Build encoding config
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
}
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
# Extract output file paths from result
|
||||
# Handle case where GCE worker returns a list or unexpected format
|
||||
if isinstance(result, list):
|
||||
# If result is a list, try to find the output_files in the first dict
|
||||
self.logger.warning(f"GCE returned list instead of dict: {result}")
|
||||
result = result[0] if result and isinstance(result[0], dict) else {}
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
for path in raw_output_files:
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
encoding_time = time.time() - start_time
|
||||
self.logger.error(f"GCE encoding failed: {e}")
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=str(e),
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if GCE encoding is available and configured."""
|
||||
service = self._get_service()
|
||||
return service.is_enabled
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get GCE encoding status."""
|
||||
service = self._get_service()
|
||||
|
||||
status = {
|
||||
"backend": self.name,
|
||||
"available": service.is_enabled,
|
||||
"configured": service.is_configured,
|
||||
}
|
||||
|
||||
if service.is_configured:
|
||||
try:
|
||||
health = await service.health_check()
|
||||
status["health"] = health
|
||||
except Exception as e:
|
||||
status["health_error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
|
||||
# Factory function to get an encoding backend
|
||||
def get_encoding_backend(
|
||||
backend_type: str = "auto",
|
||||
**kwargs
|
||||
) -> EncodingBackend:
|
||||
"""
|
||||
Get an encoding backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Type of backend - "local", "gce", or "auto"
|
||||
**kwargs: Additional arguments for the backend
|
||||
|
||||
Returns:
|
||||
EncodingBackend instance
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is unknown
|
||||
"""
|
||||
if backend_type == "local":
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
elif backend_type == "gce":
|
||||
return GCEEncodingBackend(**kwargs)
|
||||
elif backend_type == "auto":
|
||||
# Check if GCE is available, otherwise use local
|
||||
gce_backend = GCEEncodingBackend(**kwargs)
|
||||
if gce_backend._get_service().is_enabled:
|
||||
return gce_backend
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown encoding backend type: {backend_type}")
|
||||
@@ -0,0 +1,502 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This service dispatches video encoding jobs to a dedicated high-performance
|
||||
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
|
||||
|
||||
The GCE worker provides:
|
||||
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
|
||||
- Dedicated vCPUs (no contention)
|
||||
- 2-3x faster FFmpeg libx264 encoding
|
||||
|
||||
Usage:
|
||||
encoding_service = get_encoding_service()
|
||||
if encoding_service.is_configured:
|
||||
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts)
|
||||
MAX_RETRIES = 3
|
||||
INITIAL_BACKOFF_SECONDS = 2.0
|
||||
MAX_BACKOFF_SECONDS = 10.0
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Try environment variables first, then Secret Manager
|
||||
self._url = self.settings.encoding_worker_url
|
||||
self._api_key = self.settings.encoding_worker_api_key
|
||||
|
||||
# Fall back to Secret Manager
|
||||
if not self._url:
|
||||
self._url = self.settings.get_secret("encoding-worker-url")
|
||||
if not self._api_key:
|
||||
self._api_key = self.settings.get_secret("encoding-worker-api-key")
|
||||
|
||||
self._initialized = True
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if encoding service is configured with URL and API key."""
|
||||
self._load_credentials()
|
||||
return bool(self._url and self._api_key)
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
headers: Dict[str, str],
|
||||
json_payload: Optional[Dict[str, Any]] = None,
|
||||
timeout: float = 30.0,
|
||||
job_id: str = "unknown",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make an HTTP request with retry logic for transient failures.
|
||||
|
||||
This handles connection errors that occur when the GCE worker is
|
||||
restarting (e.g., during deployments) by retrying with exponential backoff.
|
||||
|
||||
Args:
|
||||
method: HTTP method (GET, POST)
|
||||
url: Request URL
|
||||
headers: Request headers
|
||||
json_payload: JSON body for POST requests
|
||||
timeout: Request timeout in seconds
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- status (int): HTTP status code
|
||||
- json (Any): Parsed JSON response body (if status 200, else None)
|
||||
- text (str): Raw response text (if status != 200, else None)
|
||||
|
||||
Raises:
|
||||
aiohttp.ClientConnectorError: If all retries fail due to connection errors
|
||||
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
|
||||
asyncio.TimeoutError: If all retries fail due to timeout
|
||||
"""
|
||||
last_exception = None
|
||||
backoff = INITIAL_BACKOFF_SECONDS
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
if method.upper() == "POST":
|
||||
async with session.post(
|
||||
url, json=json_payload, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
# Return a copy of the response data since we exit the context
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
else: # GET
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
f"Retrying in {backoff:.1f}s..."
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
|
||||
)
|
||||
|
||||
raise last_exception
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit an encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files (gs://bucket/path/)
|
||||
output_gcs_path: GCS path for output files (gs://bucket/path/)
|
||||
encoding_config: Configuration for encoding (formats, quality, etc.)
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"encoding_config": encoding_config,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the status of an encoding job.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
|
||||
Returns:
|
||||
Job status including: status, progress, error, output_files
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=headers,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
job_id: str,
|
||||
poll_interval: float = 10.0,
|
||||
timeout: float = 3600.0,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
poll_interval: Seconds between status checks
|
||||
timeout: Maximum time to wait (default 1 hour)
|
||||
progress_callback: Optional callback(progress: int) for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
|
||||
status = {}
|
||||
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
# Report progress
|
||||
if progress != last_progress:
|
||||
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
|
||||
last_progress = progress
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(progress)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback failed: {e}")
|
||||
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
|
||||
return status
|
||||
|
||||
if job_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def encode_videos(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Optional[Dict[str, Any]] = None,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files
|
||||
output_gcs_path: GCS path for output files
|
||||
encoding_config: Optional encoding configuration
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
Returns:
|
||||
Health status including active jobs and FFmpeg version
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.json()
|
||||
return {"status": "error", "code": resp.status}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_encoding_service: Optional[EncodingService] = None
|
||||
|
||||
|
||||
def get_encoding_service() -> EncodingService:
|
||||
"""Get the singleton encoding service instance."""
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
return _encoding_service
|
||||
@@ -9,6 +9,7 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -114,47 +115,59 @@ class FirestoreService:
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
|
||||
# Filter by request_metadata fields using dot notation
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
|
||||
# Filter by user_email (job owner)
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Filter by tenant_id (white-label portal scoping)
|
||||
if tenant_id:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
|
||||
|
||||
docs = query.stream()
|
||||
jobs = [Job(**doc.to_dict()) for doc in docs]
|
||||
|
||||
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
@@ -239,10 +252,13 @@ class FirestoreService:
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
@@ -257,6 +273,165 @@ class FirestoreService:
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
@@ -341,3 +516,24 @@ class FirestoreService:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_firestore_client: Optional[firestore.Client] = None
|
||||
|
||||
|
||||
def get_firestore_client() -> firestore.Client:
|
||||
"""
|
||||
Get a shared Firestore client instance.
|
||||
|
||||
This returns a raw Firestore client (not the FirestoreService) for use
|
||||
in services that need direct Firestore access without the job-specific
|
||||
abstractions provided by FirestoreService.
|
||||
|
||||
Returns:
|
||||
Firestore client instance
|
||||
"""
|
||||
global _firestore_client
|
||||
if _firestore_client is None:
|
||||
_firestore_client = firestore.Client(project=settings.google_cloud_project)
|
||||
return _firestore_client
|
||||
|
||||
@@ -408,11 +408,11 @@ class FlacfetchClient:
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
@@ -429,12 +429,112 @@ class FlacfetchClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def clear_search_cache(self, artist: str, title: str) -> bool:
|
||||
"""
|
||||
Clear a specific cached search result by artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
True if a cache entry was deleted, False if no entry existed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.request(
|
||||
"DELETE",
|
||||
f"{self.base_url}/cache/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted = data.get("deleted", False)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if deleted else 'no entry found'}"
|
||||
)
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def clear_all_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached search results.
|
||||
|
||||
Returns:
|
||||
Number of cache entries deleted
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/cache",
|
||||
headers=self._headers(),
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted_count = data.get("deleted_count", 0)
|
||||
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
|
||||
Returns:
|
||||
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/cache/stats",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# GCE Encoding Worker
|
||||
|
||||
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/encode` | POST | Submit a full encoding job |
|
||||
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
|
||||
| `/status/{job_id}` | GET | Get job status and progress |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
|
||||
|
||||
## Encoding Process
|
||||
|
||||
### Full Encoding (`/encode`)
|
||||
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
|
||||
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
|
||||
- Lossless 4K MP4
|
||||
- Lossy 4K MP4
|
||||
- Lossless MKV
|
||||
- 720p MP4
|
||||
3. Uploads results to GCS
|
||||
4. Reports progress via status endpoint
|
||||
|
||||
### Preview Encoding (`/encode-preview`)
|
||||
1. Downloads ASS subtitle and audio files from GCS
|
||||
2. Runs FFmpeg to create a quick 480x270 preview video
|
||||
3. Uploads result to GCS
|
||||
4. Used for real-time preview in the web UI
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **FFmpeg**: Static build from John Van Sickle
|
||||
- **Python 3.13**: Built from source (required for karaoke-gen)
|
||||
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
|
||||
|
||||
## Deployment
|
||||
|
||||
This service is deployed via the VM's startup script. The script:
|
||||
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
|
||||
2. Builds Python 3.13 from source
|
||||
3. Creates venv and installs dependencies
|
||||
4. Writes `main.py` to `/opt/encoding-worker/`
|
||||
5. Creates systemd service
|
||||
6. Starts the service on port 8080
|
||||
|
||||
## Local Development
|
||||
|
||||
This code is extracted from the infrastructure for maintainability. To test locally:
|
||||
|
||||
```bash
|
||||
cd backend/services/gce_encoding
|
||||
pip install -r requirements.txt
|
||||
pip install ../../../ # Install karaoke-gen wheel
|
||||
ENCODING_API_KEY=test python main.py
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
|
||||
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
|
||||
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
|
||||
- **GCS integration**: Direct download/upload via google-cloud-storage client
|
||||
@@ -0,0 +1,22 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This package contains the FastAPI application that runs on the GCE encoding worker VM.
|
||||
It provides HTTP endpoints for video encoding jobs.
|
||||
|
||||
The service:
|
||||
- Downloads input files from GCS
|
||||
- Runs FFmpeg encoding via LocalEncodingService
|
||||
- Uploads results back to GCS
|
||||
- Provides job status tracking
|
||||
|
||||
Endpoints:
|
||||
- POST /encode - Submit a full encoding job
|
||||
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
|
||||
- GET /status/{job_id} - Get job status
|
||||
- GET /health - Health check
|
||||
"""
|
||||
|
||||
from .main import app
|
||||
|
||||
__all__ = ["app"]
|
||||
@@ -0,0 +1,603 @@
|
||||
import asyncio
|
||||
import glob as glob_module
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import uuid
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
|
||||
from google.cloud import storage
|
||||
from pydantic import BaseModel
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
app = FastAPI(title="Encoding Worker", version="1.0.0")
|
||||
|
||||
# API key authentication
|
||||
API_KEY = os.environ.get("ENCODING_API_KEY", "")
|
||||
|
||||
|
||||
async def verify_api_key(x_api_key: str = Header(None)):
|
||||
# Verify API key for authentication
|
||||
if not API_KEY:
|
||||
logger.warning("No API key configured - authentication disabled")
|
||||
return True
|
||||
if x_api_key != API_KEY:
|
||||
raise HTTPException(status_code=401, detail="Invalid API key")
|
||||
return True
|
||||
|
||||
# Job tracking
|
||||
jobs: dict[str, dict] = {}
|
||||
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
|
||||
|
||||
# GCS client
|
||||
storage_client = storage.Client()
|
||||
|
||||
class EncodeRequest(BaseModel):
|
||||
job_id: str
|
||||
input_gcs_path: str # gs://bucket/path/to/inputs/
|
||||
output_gcs_path: str # gs://bucket/path/to/outputs/
|
||||
encoding_config: dict # Video formats to generate
|
||||
|
||||
|
||||
class EncodePreviewRequest(BaseModel):
|
||||
job_id: str
|
||||
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
|
||||
audio_gcs_path: str # gs://bucket/path/to/audio.flac
|
||||
output_gcs_path: str # gs://bucket/path/to/output.mp4
|
||||
background_color: str = "black"
|
||||
background_image_gcs_path: Optional[str] = None
|
||||
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
|
||||
|
||||
|
||||
class JobStatus(BaseModel):
|
||||
job_id: str
|
||||
status: str # pending, running, complete, failed
|
||||
progress: int # 0-100
|
||||
error: Optional[str] = None
|
||||
output_files: Optional[list[str]] = None
|
||||
|
||||
|
||||
def download_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a file or folder from GCS
|
||||
# Parse gs://bucket/path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blobs = list(bucket.list_blobs(prefix=prefix))
|
||||
|
||||
for blob in blobs:
|
||||
# Get relative path from prefix
|
||||
rel_path = blob.name[len(prefix):].lstrip("/")
|
||||
if not rel_path:
|
||||
continue
|
||||
dest = local_path / rel_path
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(dest))
|
||||
logger.info(f"Downloaded: {blob.name} -> {dest}")
|
||||
|
||||
|
||||
def upload_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a file or folder to GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
|
||||
if local_path.is_file():
|
||||
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
|
||||
else:
|
||||
for file in local_path.rglob("*"):
|
||||
if file.is_file():
|
||||
rel_path = file.relative_to(local_path)
|
||||
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(file))
|
||||
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
|
||||
|
||||
|
||||
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
|
||||
# Download a single file from GCS
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
local_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
blob.download_to_filename(str(local_path))
|
||||
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
|
||||
|
||||
|
||||
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
|
||||
# Upload a single file to a specific GCS path
|
||||
parts = gcs_uri.replace("gs://", "").split("/", 1)
|
||||
bucket_name = parts[0]
|
||||
blob_name = parts[1] if len(parts) > 1 else ""
|
||||
|
||||
bucket = storage_client.bucket(bucket_name)
|
||||
blob = bucket.blob(blob_name)
|
||||
blob.upload_from_filename(str(local_path))
|
||||
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
|
||||
|
||||
|
||||
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
|
||||
# Run FFmpeg encoding for preview video (480x270, fast settings)
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Download input files
|
||||
ass_path = work_dir / "subtitles.ass"
|
||||
audio_path = work_dir / "audio.flac"
|
||||
|
||||
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
|
||||
jobs[job_id]["progress"] = 30
|
||||
|
||||
# Download background image if provided
|
||||
bg_image_path = None
|
||||
if request.background_image_gcs_path:
|
||||
bg_image_path = work_dir / "background.png"
|
||||
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
|
||||
|
||||
# Download custom font if provided and register with fontconfig
|
||||
if request.font_gcs_path:
|
||||
# Use standard fontconfig location that's already in the search path
|
||||
fonts_dir = Path("/usr/local/share/fonts/custom")
|
||||
fonts_dir.mkdir(parents=True, exist_ok=True)
|
||||
font_filename = request.font_gcs_path.split("/")[-1]
|
||||
font_path = fonts_dir / font_filename
|
||||
download_single_file_from_gcs(request.font_gcs_path, font_path)
|
||||
logger.info(f"Downloaded custom font: {font_path}")
|
||||
# Update fontconfig cache so libass can find the font
|
||||
subprocess.run(["fc-cache", "-fv"], capture_output=True)
|
||||
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
|
||||
|
||||
# Build FFmpeg command
|
||||
output_path = work_dir / "preview.mp4"
|
||||
|
||||
# Escape special characters in path for FFmpeg filter syntax
|
||||
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
|
||||
def escape_ffmpeg_filter_path(path: str) -> str:
|
||||
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
|
||||
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
|
||||
|
||||
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
|
||||
|
||||
# Base command with frame rate
|
||||
cmd = ["ffmpeg", "-y", "-r", "24"]
|
||||
|
||||
# Video input: background image or solid color
|
||||
if bg_image_path and bg_image_path.exists():
|
||||
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
|
||||
# Scale and pad background to 480x270
|
||||
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
|
||||
else:
|
||||
# Solid color background
|
||||
color = request.background_color or "black"
|
||||
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
|
||||
vf = f"ass={escaped_ass_path}"
|
||||
|
||||
# Audio input
|
||||
cmd.extend(["-i", str(audio_path)])
|
||||
|
||||
# Video filter and encoding settings
|
||||
cmd.extend([
|
||||
"-vf", vf,
|
||||
"-c:a", "aac", "-b:a", "96k",
|
||||
"-c:v", "libx264",
|
||||
"-preset", "superfast",
|
||||
"-crf", "28",
|
||||
"-pix_fmt", "yuv420p",
|
||||
"-movflags", "+faststart",
|
||||
"-threads", "8",
|
||||
"-shortest",
|
||||
str(output_path)
|
||||
])
|
||||
|
||||
jobs[job_id]["progress"] = 40
|
||||
logger.info(f"Running preview encoding: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error(f"FFmpeg failed: {result.stderr}")
|
||||
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
|
||||
|
||||
jobs[job_id]["progress"] = 80
|
||||
logger.info(f"Preview encoded: {output_path}")
|
||||
|
||||
# Upload output to GCS
|
||||
upload_single_file_to_gcs(output_path, request.output_gcs_path)
|
||||
jobs[job_id]["progress"] = 95
|
||||
|
||||
jobs[job_id]["output_files"] = [request.output_gcs_path]
|
||||
return output_path
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
def ensure_latest_wheel():
|
||||
'''Download and install latest karaoke-gen wheel from GCS.
|
||||
|
||||
Called at the start of each job to enable hot code updates without restart.
|
||||
In-progress jobs continue with their version, new jobs get latest code.
|
||||
'''
|
||||
try:
|
||||
logger.info("Checking for latest karaoke-gen wheel in GCS...")
|
||||
|
||||
# Download latest wheel
|
||||
result = subprocess.run(
|
||||
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
|
||||
capture_output=True, text=True, timeout=60
|
||||
)
|
||||
|
||||
# Find the downloaded wheel (get the latest by version sorting)
|
||||
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
|
||||
if not wheels:
|
||||
logger.warning("No wheel found in GCS, using fallback encoding logic")
|
||||
return False
|
||||
|
||||
# Sort to get latest version
|
||||
wheel_path = sorted(wheels)[-1]
|
||||
logger.info(f"Installing wheel: {wheel_path}")
|
||||
|
||||
# Install (or upgrade) the wheel
|
||||
# Use 5-minute timeout - first install at job start may need to resolve dependencies
|
||||
# Subsequent installs are faster since dependencies are cached
|
||||
install_result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
|
||||
capture_output=True, text=True, timeout=300
|
||||
)
|
||||
|
||||
if install_result.returncode != 0:
|
||||
logger.warning(f"Wheel installation failed: {install_result.stderr}")
|
||||
return False
|
||||
|
||||
logger.info(f"Successfully installed wheel: {wheel_path}")
|
||||
return True
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("Wheel download/install timed out, using fallback")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to ensure latest wheel: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
def run_encoding(job_id: str, work_dir: Path, config: dict):
|
||||
'''Run encoding using LocalEncodingService (single source of truth).
|
||||
|
||||
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
|
||||
output files match local CLI exactly:
|
||||
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
- Concatenated title + karaoke + end screens
|
||||
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
|
||||
|
||||
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
|
||||
'''
|
||||
jobs[job_id]["status"] = "running"
|
||||
jobs[job_id]["progress"] = 10
|
||||
|
||||
try:
|
||||
# Import LocalEncodingService from installed wheel (required, no fallback)
|
||||
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
|
||||
logger.info("Using LocalEncodingService from installed wheel")
|
||||
|
||||
# Get artist/title from config for proper naming
|
||||
artist = config.get("artist", "Unknown Artist")
|
||||
title = config.get("title", "Unknown Title")
|
||||
base_name = f"{artist} - {title}"
|
||||
logger.info(f"Encoding for: {base_name}")
|
||||
|
||||
# Find input files in work_dir
|
||||
# Title/end screens are in screens/ subdirectory
|
||||
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
|
||||
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
|
||||
|
||||
# Karaoke video - search for With Vocals or main karaoke video
|
||||
karaoke_video = find_file(
|
||||
work_dir,
|
||||
"*With Vocals*.mov", "*With Vocals*.mkv",
|
||||
"*Vocals*.mov", "*Vocals*.mkv",
|
||||
"*.mkv", "*.mov"
|
||||
)
|
||||
# Exclude title/end/output videos
|
||||
if karaoke_video:
|
||||
name_lower = karaoke_video.name.lower()
|
||||
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
|
||||
# Search more specifically for karaoke video
|
||||
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
|
||||
|
||||
# Instrumental audio - respect user's selection from encoding config
|
||||
instrumental_selection = config.get("instrumental_selection", "clean")
|
||||
logger.info(f"Instrumental selection from config: {instrumental_selection}")
|
||||
|
||||
if instrumental_selection == "with_backing":
|
||||
# User selected instrumental with backing vocals
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
else:
|
||||
# Default to clean instrumental
|
||||
instrumental = find_file(
|
||||
work_dir,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
logger.info(f"Found files:")
|
||||
logger.info(f" Title video: {title_video}")
|
||||
logger.info(f" Karaoke video: {karaoke_video}")
|
||||
logger.info(f" End video: {end_video}")
|
||||
logger.info(f" Instrumental ({instrumental_selection}): {instrumental}")
|
||||
|
||||
# Validate required files
|
||||
if not title_video:
|
||||
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
|
||||
if not karaoke_video:
|
||||
raise ValueError(f"No karaoke video found in {work_dir}")
|
||||
if not instrumental:
|
||||
raise ValueError(f"No instrumental audio found in {work_dir}")
|
||||
|
||||
output_dir = work_dir / "outputs"
|
||||
output_dir.mkdir(exist_ok=True)
|
||||
|
||||
jobs[job_id]["progress"] = 20
|
||||
|
||||
# Build encoding config with proper file names
|
||||
encoding_config = EncodingConfig(
|
||||
title_video=str(title_video),
|
||||
karaoke_video=str(karaoke_video),
|
||||
instrumental_audio=str(instrumental),
|
||||
end_video=str(end_video) if end_video else None,
|
||||
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
)
|
||||
|
||||
# Create service and run encoding
|
||||
service = LocalEncodingService(logger=logger)
|
||||
|
||||
jobs[job_id]["progress"] = 30
|
||||
logger.info("Starting LocalEncodingService.encode_all_formats()")
|
||||
|
||||
result = service.encode_all_formats(encoding_config)
|
||||
|
||||
if not result.success:
|
||||
raise RuntimeError(f"Encoding failed: {result.error}")
|
||||
|
||||
jobs[job_id]["progress"] = 90
|
||||
|
||||
# Collect output files
|
||||
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
|
||||
jobs[job_id]["output_files"] = output_files
|
||||
|
||||
logger.info(f"Encoding complete. Output files: {output_files}")
|
||||
return output_dir
|
||||
|
||||
except ImportError as e:
|
||||
# No fallback - wheel must be installed
|
||||
error_msg = (
|
||||
f"LocalEncodingService not available: {e}. "
|
||||
"The karaoke-gen wheel must be installed. "
|
||||
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
|
||||
)
|
||||
logger.error(error_msg)
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = error_msg
|
||||
raise RuntimeError(error_msg) from e
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Encoding failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
raise
|
||||
|
||||
|
||||
async def process_job(job_id: str, request: EncodeRequest):
|
||||
# Process an encoding job asynchronously
|
||||
try:
|
||||
# Download and install latest wheel at job start (allows hot updates without restart)
|
||||
# This means in-progress jobs continue with their version, new jobs get latest code
|
||||
ensure_latest_wheel()
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Download input files
|
||||
jobs[job_id]["progress"] = 5
|
||||
logger.info(f"Downloading from {request.input_gcs_path}")
|
||||
download_from_gcs(request.input_gcs_path, work_dir)
|
||||
|
||||
# Run encoding in thread pool (CPU-bound)
|
||||
loop = asyncio.get_event_loop()
|
||||
output_dir = await loop.run_in_executor(
|
||||
executor,
|
||||
run_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request.encoding_config
|
||||
)
|
||||
|
||||
# Upload outputs
|
||||
jobs[job_id]["progress"] = 95
|
||||
logger.info(f"Uploading to {request.output_gcs_path}")
|
||||
upload_to_gcs(output_dir, request.output_gcs_path)
|
||||
|
||||
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
|
||||
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
|
||||
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
gcs_path = request.output_gcs_path.replace("gs://", "")
|
||||
parts = gcs_path.split("/", 1)
|
||||
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
|
||||
local_output_files = jobs[job_id].get("output_files", [])
|
||||
blob_paths = []
|
||||
for local_path in local_output_files:
|
||||
filename = Path(local_path).name
|
||||
blob_path = f"{prefix}/{filename}" if prefix else filename
|
||||
blob_paths.append(blob_path)
|
||||
jobs[job_id]["output_files"] = blob_paths
|
||||
logger.info(f"Output files (blob paths): {blob_paths}")
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
|
||||
# Process a preview encoding job asynchronously
|
||||
try:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
work_dir = Path(temp_dir) / "work"
|
||||
work_dir.mkdir()
|
||||
|
||||
# Run preview encoding in thread pool (CPU-bound)
|
||||
# Note: run_preview_encoding handles download/upload internally
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(
|
||||
executor,
|
||||
run_preview_encoding,
|
||||
job_id,
|
||||
work_dir,
|
||||
request
|
||||
)
|
||||
|
||||
jobs[job_id]["status"] = "complete"
|
||||
jobs[job_id]["progress"] = 100
|
||||
logger.info(f"Preview job {job_id} complete")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Preview job {job_id} failed: {e}")
|
||||
jobs[job_id]["status"] = "failed"
|
||||
jobs[job_id]["error"] = str(e)
|
||||
|
||||
|
||||
@app.post("/encode-preview")
|
||||
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit a preview encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
# If job already exists, return cached result or current status
|
||||
if job_id in jobs:
|
||||
existing_job = jobs[job_id]
|
||||
if existing_job["status"] == "complete":
|
||||
# Return cached result - preview already encoded
|
||||
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
|
||||
elif existing_job["status"] == "failed":
|
||||
# Previous attempt failed, allow retry by replacing the job
|
||||
pass
|
||||
else:
|
||||
# Job is still in progress
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_preview_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.post("/encode")
|
||||
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
|
||||
# Submit an encoding job
|
||||
job_id = request.job_id
|
||||
|
||||
if job_id in jobs:
|
||||
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
|
||||
|
||||
jobs[job_id] = {
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"progress": 0,
|
||||
"error": None,
|
||||
"output_files": None,
|
||||
}
|
||||
|
||||
background_tasks.add_task(process_job, job_id, request)
|
||||
|
||||
return {"status": "accepted", "job_id": job_id}
|
||||
|
||||
|
||||
@app.get("/status/{job_id}")
|
||||
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
|
||||
# Get the status of an encoding job
|
||||
if job_id not in jobs:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
return JobStatus(**jobs[job_id])
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
async def health_check():
|
||||
# Health check endpoint
|
||||
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
|
||||
|
||||
# Get karaoke-gen wheel version if installed
|
||||
wheel_version = None
|
||||
try:
|
||||
from importlib.metadata import version as get_version
|
||||
wheel_version = get_version("karaoke-gen")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"active_jobs": active_jobs,
|
||||
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
|
||||
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
|
||||
"wheel_version": wheel_version,
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
@@ -0,0 +1,16 @@
|
||||
# GCE Encoding Worker dependencies
|
||||
# These are installed in the VM's Python virtual environment
|
||||
|
||||
# Web framework
|
||||
fastapi>=0.109.0
|
||||
uvicorn>=0.27.0
|
||||
|
||||
# GCS client
|
||||
google-cloud-storage>=2.14.0
|
||||
|
||||
# Async utilities (optional, for future improvements)
|
||||
aiofiles>=23.2.0
|
||||
aiohttp>=3.9.0
|
||||
|
||||
# Note: The karaoke-gen wheel is installed separately from GCS.
|
||||
# It provides LocalEncodingService which does the actual encoding.
|
||||
@@ -15,6 +15,7 @@ import os
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
from karaoke_gen.utils import sanitize_filename
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -136,8 +137,10 @@ class GoogleDriveService:
|
||||
logger.info(f"Looking for folder '{folder_name}' in parent {parent_id}")
|
||||
|
||||
# Search for existing folder
|
||||
# Escape single quotes in folder name for Google Drive API query syntax
|
||||
escaped_folder_name = folder_name.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{folder_name}' and '{parent_id}' in parents "
|
||||
f"name='{escaped_folder_name}' and '{parent_id}' in parents "
|
||||
f"and mimeType='application/vnd.google-apps.folder' and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id, name)").execute()
|
||||
@@ -203,8 +206,10 @@ class GoogleDriveService:
|
||||
|
||||
# Check for existing file with same name
|
||||
if replace_existing:
|
||||
# Escape single quotes in filename for Google Drive API query syntax
|
||||
escaped_filename = filename.replace("'", "\\'")
|
||||
query = (
|
||||
f"name='{filename}' and '{parent_id}' in parents and trashed=false"
|
||||
f"name='{escaped_filename}' and '{parent_id}' in parents and trashed=false"
|
||||
)
|
||||
results = self.service.files().list(q=query, fields="files(id)").execute()
|
||||
for existing_file in results.get("files", []):
|
||||
@@ -249,7 +254,10 @@ class GoogleDriveService:
|
||||
Returns:
|
||||
Dictionary mapping category to uploaded file ID
|
||||
"""
|
||||
filename_base = f"{brand_code} - {base_name}"
|
||||
# Sanitize base_name to handle Unicode characters (curly quotes, em dashes, etc.)
|
||||
# that could cause issues with Google Drive API queries and file naming
|
||||
safe_base_name = sanitize_filename(base_name) if base_name else base_name
|
||||
filename_base = f"{brand_code} - {safe_base_name}"
|
||||
uploaded_files = {}
|
||||
|
||||
logger.info(
|
||||
@@ -296,6 +304,45 @@ class GoogleDriveService:
|
||||
logger.info(f"Public share upload complete: {len(uploaded_files)} files uploaded")
|
||||
return uploaded_files
|
||||
|
||||
def delete_file(self, file_id: str) -> bool:
|
||||
"""
|
||||
Delete a file from Google Drive.
|
||||
|
||||
Args:
|
||||
file_id: Google Drive file ID to delete
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
logger.info(f"Deleting Google Drive file: {file_id}")
|
||||
|
||||
try:
|
||||
self.service.files().delete(fileId=file_id).execute()
|
||||
logger.info(f"Successfully deleted file: {file_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
# Check if it's a 404 (already deleted)
|
||||
if hasattr(e, 'resp') and e.resp.status == 404:
|
||||
logger.warning(f"File not found (already deleted?): {file_id}")
|
||||
return True
|
||||
logger.error(f"Failed to delete Google Drive file: {e}")
|
||||
return False
|
||||
|
||||
def delete_files(self, file_ids: list[str]) -> dict[str, bool]:
|
||||
"""
|
||||
Delete multiple files from Google Drive.
|
||||
|
||||
Args:
|
||||
file_ids: List of Google Drive file IDs to delete
|
||||
|
||||
Returns:
|
||||
Dictionary mapping file_id to success status
|
||||
"""
|
||||
results = {}
|
||||
for file_id in file_ids:
|
||||
results[file_id] = self.delete_file(file_id)
|
||||
return results
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_gdrive_service: Optional[GoogleDriveService] = None
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
"""
|
||||
Centralized job defaults service.
|
||||
|
||||
This module provides consistent handling of job creation defaults across all
|
||||
endpoints (file_upload, audio_search, made-for-you webhook, etc.).
|
||||
|
||||
Centralizing these defaults prevents divergence between code paths and ensures
|
||||
all jobs receive the same default configuration.
|
||||
"""
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
@dataclass
|
||||
class EffectiveDistributionSettings:
|
||||
"""Distribution settings with defaults applied from environment variables."""
|
||||
dropbox_path: Optional[str]
|
||||
gdrive_folder_id: Optional[str]
|
||||
discord_webhook_url: Optional[str]
|
||||
brand_prefix: Optional[str]
|
||||
enable_youtube_upload: bool
|
||||
youtube_description: Optional[str]
|
||||
|
||||
|
||||
def get_effective_distribution_settings(
|
||||
dropbox_path: Optional[str] = None,
|
||||
gdrive_folder_id: Optional[str] = None,
|
||||
discord_webhook_url: Optional[str] = None,
|
||||
brand_prefix: Optional[str] = None,
|
||||
enable_youtube_upload: Optional[bool] = None,
|
||||
youtube_description: Optional[str] = None,
|
||||
) -> EffectiveDistributionSettings:
|
||||
"""
|
||||
Get distribution settings with defaults applied from environment variables.
|
||||
|
||||
This ensures consistent handling of defaults across all job creation endpoints.
|
||||
Each parameter, if not provided (None), falls back to the corresponding
|
||||
environment variable configured in settings.
|
||||
|
||||
Args:
|
||||
dropbox_path: Explicit Dropbox path or None for default
|
||||
gdrive_folder_id: Explicit Google Drive folder ID or None for default
|
||||
discord_webhook_url: Explicit Discord webhook URL or None for default
|
||||
brand_prefix: Explicit brand prefix or None for default
|
||||
enable_youtube_upload: Explicit flag or None for default
|
||||
youtube_description: Explicit description or None for default
|
||||
|
||||
Returns:
|
||||
EffectiveDistributionSettings with defaults applied
|
||||
"""
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=dropbox_path or settings.default_dropbox_path,
|
||||
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
|
||||
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
|
||||
brand_prefix=brand_prefix or settings.default_brand_prefix,
|
||||
enable_youtube_upload=enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload,
|
||||
youtube_description=youtube_description or settings.default_youtube_description,
|
||||
)
|
||||
|
||||
|
||||
def resolve_cdg_txt_defaults(
|
||||
theme_id: Optional[str],
|
||||
enable_cdg: Optional[bool] = None,
|
||||
enable_txt: Optional[bool] = None,
|
||||
) -> Tuple[bool, bool]:
|
||||
"""
|
||||
Resolve CDG/TXT settings based on theme and explicit settings.
|
||||
|
||||
The resolution logic is:
|
||||
1. If explicit True/False is provided, use that value
|
||||
2. Otherwise, if a theme is set, use the server defaults (settings.default_enable_cdg/txt)
|
||||
3. If no theme is set, default to False (CDG/TXT require style configuration)
|
||||
|
||||
This ensures CDG/TXT are only enabled when:
|
||||
- A theme is configured (provides necessary style params), AND
|
||||
- The server defaults allow it (DEFAULT_ENABLE_CDG=true by default)
|
||||
|
||||
Args:
|
||||
theme_id: Theme identifier (if any)
|
||||
enable_cdg: Explicit CDG setting (None means use default)
|
||||
enable_txt: Explicit TXT setting (None means use default)
|
||||
|
||||
Returns:
|
||||
Tuple of (resolved_enable_cdg, resolved_enable_txt)
|
||||
"""
|
||||
settings = get_settings()
|
||||
|
||||
# Default based on whether theme is set AND server defaults
|
||||
# Theme is required because CDG/TXT need style configuration
|
||||
theme_is_set = theme_id is not None
|
||||
default_cdg = theme_is_set and settings.default_enable_cdg
|
||||
default_txt = theme_is_set and settings.default_enable_txt
|
||||
|
||||
# Explicit values override defaults, None uses computed default
|
||||
resolved_cdg = enable_cdg if enable_cdg is not None else default_cdg
|
||||
resolved_txt = enable_txt if enable_txt is not None else default_txt
|
||||
|
||||
return resolved_cdg, resolved_txt
|
||||
|
||||
|
||||
# Singleton instance (optional, for convenience)
|
||||
_service_instance = None
|
||||
|
||||
|
||||
def get_job_defaults_service():
|
||||
"""Get the job defaults service (module-level functions work fine, this is for consistency)."""
|
||||
return {
|
||||
'get_effective_distribution_settings': get_effective_distribution_settings,
|
||||
'resolve_cdg_txt_defaults': resolve_cdg_txt_defaults,
|
||||
}
|
||||
+381
-35
@@ -13,7 +13,10 @@ import uuid
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict, Any, List
|
||||
|
||||
from backend.config import settings
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
@@ -21,6 +24,16 @@ from backend.services.storage_service import StorageService
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobManager:
|
||||
"""Manager for job lifecycle and state."""
|
||||
|
||||
@@ -29,13 +42,52 @@ class JobManager:
|
||||
self.firestore = FirestoreService()
|
||||
self.storage = StorageService()
|
||||
|
||||
def create_job(self, job_create: JobCreate) -> Job:
|
||||
def create_job(self, job_create: JobCreate, is_admin: bool = False) -> Job:
|
||||
"""
|
||||
Create a new job with initial state PENDING.
|
||||
|
||||
|
||||
Jobs start in PENDING state and transition to DOWNLOADING
|
||||
when a worker picks them up.
|
||||
|
||||
Args:
|
||||
job_create: Job creation parameters
|
||||
is_admin: Whether the requesting user is an admin (bypasses rate limits)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme_id is not provided (all jobs require a theme)
|
||||
RateLimitExceededError: If user has exceeded their daily job limit
|
||||
"""
|
||||
# Check rate limit FIRST (before any other validation)
|
||||
# This prevents wasted work if user is rate limited
|
||||
if job_create.user_email:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
allowed, remaining, message = rate_limit_service.check_user_job_limit(
|
||||
user_email=job_create.user_email,
|
||||
is_admin=is_admin
|
||||
)
|
||||
if not allowed:
|
||||
from backend.services.rate_limit_service import _seconds_until_midnight_utc
|
||||
|
||||
# Get actual current count - remaining is clamped to 0 which loses info
|
||||
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
|
||||
raise RateLimitExceededError(
|
||||
message=message,
|
||||
limit_type="jobs_per_day",
|
||||
remaining_seconds=_seconds_until_midnight_utc(),
|
||||
current_count=current_count,
|
||||
limit_value=settings.rate_limit_jobs_per_day
|
||||
)
|
||||
|
||||
# Enforce theme requirement - all jobs must have a theme
|
||||
# This prevents unstyled videos from ever being generated
|
||||
if not job_create.theme_id:
|
||||
raise ValueError(
|
||||
"theme_id is required for all jobs. "
|
||||
"Use get_theme_service().get_default_theme_id() to get the default theme."
|
||||
)
|
||||
|
||||
job_id = str(uuid.uuid4())[:8]
|
||||
|
||||
now = datetime.utcnow()
|
||||
@@ -61,17 +113,38 @@ class JobManager:
|
||||
discord_webhook_url=job_create.discord_webhook_url,
|
||||
dropbox_path=job_create.dropbox_path,
|
||||
gdrive_folder_id=job_create.gdrive_folder_id,
|
||||
# Theme configuration
|
||||
theme_id=job_create.theme_id,
|
||||
color_overrides=job_create.color_overrides,
|
||||
style_params_gcs_path=job_create.style_params_gcs_path,
|
||||
style_assets=job_create.style_assets,
|
||||
# Two-phase workflow (Batch 6)
|
||||
prep_only=job_create.prep_only,
|
||||
finalise_only=job_create.finalise_only,
|
||||
keep_brand_code=job_create.keep_brand_code,
|
||||
# Request metadata (for tracking and filtering)
|
||||
request_metadata=job_create.request_metadata,
|
||||
# Tenant scoping
|
||||
tenant_id=job_create.tenant_id,
|
||||
# Made-for-you order fields
|
||||
made_for_you=job_create.made_for_you,
|
||||
customer_email=job_create.customer_email,
|
||||
customer_notes=job_create.customer_notes,
|
||||
)
|
||||
|
||||
self.firestore.create_job(job)
|
||||
logger.info(f"Created new job {job_id} with status PENDING")
|
||||
|
||||
|
||||
# Record job creation for rate limiting (after successful persistence)
|
||||
if job_create.user_email:
|
||||
try:
|
||||
from backend.services.rate_limit_service import get_rate_limit_service
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
rate_limit_service.record_job_creation(job_create.user_email, job_id)
|
||||
except Exception as e:
|
||||
# Don't fail job creation if rate limit recording fails
|
||||
logger.warning(f"Failed to record job creation for rate limiting: {e}")
|
||||
|
||||
return job
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
@@ -106,19 +179,23 @@ class JobManager:
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filtering.
|
||||
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id (customer identifier)
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (job owner)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters
|
||||
"""
|
||||
@@ -128,6 +205,8 @@ class JobManager:
|
||||
client_id=client_id,
|
||||
created_after=created_after,
|
||||
created_before=created_before,
|
||||
user_email=user_email,
|
||||
tenant_id=tenant_id,
|
||||
limit=limit
|
||||
)
|
||||
|
||||
@@ -216,7 +295,7 @@ class JobManager:
|
||||
return urls
|
||||
|
||||
def delete_job(self, job_id: str, delete_files: bool = True) -> None:
|
||||
"""Delete a job and optionally its files."""
|
||||
"""Delete a job, its files, and its logs subcollection."""
|
||||
if delete_files:
|
||||
job = self.get_job(job_id)
|
||||
if job and job.output_files:
|
||||
@@ -225,7 +304,15 @@ class JobManager:
|
||||
self.storage.delete_file(gcs_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting file {gcs_path}: {e}")
|
||||
|
||||
|
||||
# Delete logs subcollection first (must be done before deleting parent doc)
|
||||
try:
|
||||
deleted_logs = self.firestore.delete_logs_subcollection(job_id)
|
||||
if deleted_logs > 0:
|
||||
logger.info(f"Deleted {deleted_logs} log entries for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
|
||||
self.firestore.delete_job(job_id)
|
||||
logger.info(f"Deleted job {job_id}")
|
||||
|
||||
@@ -341,10 +428,217 @@ class JobManager:
|
||||
'instrumental_token': updates['instrumental_token'],
|
||||
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
|
||||
})
|
||||
|
||||
|
||||
logger.info(f"Job {job_id} transitioned to {new_status}")
|
||||
|
||||
# Trigger notifications asynchronously (fire-and-forget)
|
||||
self._trigger_state_notifications(job_id, new_status)
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Trigger email and push notifications based on state transitions.
|
||||
|
||||
This is fire-and-forget - notification failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
new_status: New job status
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
# Get the job to access user info
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.user_email:
|
||||
logger.debug(f"No user email for job {job_id}, skipping notifications")
|
||||
return
|
||||
|
||||
# Job completion notification
|
||||
if new_status == JobStatus.COMPLETE:
|
||||
self._schedule_completion_email(job)
|
||||
self._send_push_notification(job, "complete")
|
||||
|
||||
# Idle reminder scheduling for blocking states
|
||||
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
|
||||
self._schedule_idle_reminder(job, new_status)
|
||||
# Send push notification for blocking states
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
self._send_push_notification(job, action_type)
|
||||
|
||||
except Exception as e:
|
||||
# Never let notification failures affect job processing
|
||||
logger.error(f"Error triggering notifications for job {job_id}: {e}")
|
||||
|
||||
def _schedule_completion_email(self, job: Job) -> None:
|
||||
"""
|
||||
Schedule sending a job completion email.
|
||||
|
||||
For made-for-you orders, also transfers ownership from admin to customer.
|
||||
Uses asyncio to fire-and-forget the email sending.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
|
||||
state_data = job.state_data or {}
|
||||
youtube_url = state_data.get('youtube_url')
|
||||
dropbox_url = state_data.get('dropbox_link')
|
||||
brand_code = state_data.get('brand_code')
|
||||
|
||||
# For made-for-you orders, send to customer and transfer ownership
|
||||
recipient_email = job.user_email
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
# Transfer ownership from admin to customer (non-blocking - email still goes out if this fails)
|
||||
try:
|
||||
self.update_job(job.job_id, {'user_email': job.customer_email})
|
||||
logger.info(f"Transferred ownership of made-for-you job {job.job_id} to {_mask_email(job.customer_email)}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to transfer ownership for job {job.job_id}: {e}")
|
||||
|
||||
# Create async task (fire-and-forget)
|
||||
async def send_email():
|
||||
await notification_service.send_job_completion_email(
|
||||
job_id=job.job_id,
|
||||
user_email=recipient_email,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
brand_code=brand_code,
|
||||
)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
# If we're in an async context, create a task
|
||||
loop.create_task(send_email())
|
||||
except RuntimeError:
|
||||
# No event loop - we're likely in a sync context
|
||||
# Use daemon thread to avoid blocking job completion
|
||||
def run_in_thread():
|
||||
asyncio.run(send_email())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled completion email for job {job.job_id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
|
||||
|
||||
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
|
||||
"""
|
||||
Schedule an idle reminder for a blocking state.
|
||||
|
||||
Records the timestamp when the blocking state was entered and
|
||||
schedules a Cloud Tasks task for 5 minutes later.
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
# Record when we entered blocking state (for idle detection)
|
||||
blocking_entered_at = datetime.utcnow().isoformat()
|
||||
|
||||
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
|
||||
|
||||
# Update state_data with blocking state info (handle None state_data)
|
||||
existing_state_data = job.state_data or {}
|
||||
state_data_update = {
|
||||
'blocking_state_entered_at': blocking_entered_at,
|
||||
'blocking_action_type': action_type,
|
||||
'reminder_sent': False, # Will be set to True after reminder is sent
|
||||
}
|
||||
|
||||
self.firestore.update_job(job.job_id, {
|
||||
'state_data': {**existing_state_data, **state_data_update}
|
||||
})
|
||||
|
||||
# Schedule the idle reminder check via worker service (5 min delay)
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
async def schedule_reminder():
|
||||
worker_service = get_worker_service()
|
||||
await worker_service.schedule_idle_reminder(job.job_id)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(schedule_reminder())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
# Use daemon thread to avoid blocking job processing
|
||||
def run_in_thread():
|
||||
asyncio.run(schedule_reminder())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
|
||||
|
||||
def _send_push_notification(self, job: Job, action_type: str) -> None:
|
||||
"""
|
||||
Send a push notification for job state changes.
|
||||
|
||||
Fire-and-forget - failures don't affect job processing.
|
||||
|
||||
Args:
|
||||
job: Job object
|
||||
action_type: Type of notification ("lyrics", "instrumental", or "complete")
|
||||
"""
|
||||
import asyncio
|
||||
import threading
|
||||
|
||||
try:
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Skip if push notifications not enabled
|
||||
if not push_service.is_enabled():
|
||||
logger.debug("Push notifications not enabled, skipping")
|
||||
return
|
||||
|
||||
# Build job dict for notification service
|
||||
job_dict = {
|
||||
"job_id": job.job_id,
|
||||
"user_email": job.user_email,
|
||||
"artist": job.artist,
|
||||
"title": job.title,
|
||||
}
|
||||
|
||||
async def send_notification():
|
||||
if action_type == "complete":
|
||||
await push_service.send_completion_notification(job_dict)
|
||||
else:
|
||||
await push_service.send_blocking_notification(job_dict, action_type)
|
||||
|
||||
# Try to get existing event loop, create new one if none exists
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(send_notification())
|
||||
except RuntimeError:
|
||||
# No event loop - we're in a sync context
|
||||
def run_in_thread():
|
||||
asyncio.run(send_notification())
|
||||
thread = threading.Thread(target=run_in_thread, daemon=True)
|
||||
thread.start()
|
||||
|
||||
logger.debug(f"Scheduled push notification for job {job.job_id} ({action_type})")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to send push notification for job {job.job_id}: {e}")
|
||||
|
||||
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
|
||||
"""
|
||||
Update a specific key in the job's state_data field.
|
||||
@@ -571,33 +865,45 @@ class JobManager:
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
max_logs: int = 500
|
||||
max_logs: int = 500,
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
) -> None:
|
||||
"""
|
||||
Append a log entry to the job's worker_logs atomically.
|
||||
|
||||
Uses Firestore ArrayUnion for atomic append to avoid race conditions
|
||||
when multiple workers (audio + lyrics) are logging concurrently.
|
||||
|
||||
Note: max_logs is not enforced per-append (would require transactions).
|
||||
Logs may grow beyond max_logs; trim periodically if needed.
|
||||
|
||||
Append a log entry to the job's logs.
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are stored in a Firestore
|
||||
subcollection (jobs/{job_id}/logs) to avoid the 1MB document size limit.
|
||||
|
||||
For backward compatibility (USE_LOG_SUBCOLLECTION=false), logs are stored
|
||||
in the embedded worker_logs array using atomic ArrayUnion.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
worker: Worker name (audio, lyrics, screens, video, render)
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
max_logs: Not used (kept for API compatibility)
|
||||
metadata: Optional additional metadata dict
|
||||
"""
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
|
||||
# Use atomic ArrayUnion to avoid race conditions
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
if settings.use_log_subcollection:
|
||||
# New subcollection approach - avoids 1MB limit
|
||||
log_entry = WorkerLogEntry.create(
|
||||
job_id=job_id,
|
||||
worker=worker,
|
||||
level=level,
|
||||
message=message,
|
||||
metadata=metadata
|
||||
)
|
||||
self.firestore.append_log_to_subcollection(job_id, log_entry)
|
||||
else:
|
||||
# Legacy embedded array approach
|
||||
log_entry = {
|
||||
'timestamp': datetime.utcnow().isoformat() + 'Z',
|
||||
'level': level,
|
||||
'worker': worker,
|
||||
'message': message[:1000] # Truncate long messages
|
||||
}
|
||||
self.firestore.append_worker_log(job_id, log_entry)
|
||||
|
||||
def get_worker_logs(
|
||||
self,
|
||||
@@ -607,28 +913,68 @@ class JobManager:
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get worker logs for a job, optionally filtered by worker and index.
|
||||
|
||||
|
||||
By default (USE_LOG_SUBCOLLECTION=true), logs are read from the
|
||||
subcollection. Falls back to embedded array for older jobs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
since_index: Return only logs after this index (for pagination)
|
||||
worker: Filter by worker name (optional)
|
||||
|
||||
|
||||
Returns:
|
||||
List of log entries as dicts
|
||||
List of log entries as dicts (in legacy format for API compatibility)
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
subcollection_logs = self.firestore.get_logs_from_subcollection(
|
||||
job_id=job_id,
|
||||
offset=since_index,
|
||||
worker=worker,
|
||||
limit=500
|
||||
)
|
||||
if subcollection_logs:
|
||||
# Convert to legacy format for API compatibility
|
||||
return [log.to_legacy_dict() for log in subcollection_logs]
|
||||
# Fall through to check embedded array for older jobs
|
||||
|
||||
# Embedded array approach (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return []
|
||||
|
||||
|
||||
logs = [log.dict() if hasattr(log, 'dict') else log for log in job.worker_logs]
|
||||
|
||||
|
||||
# Filter by index
|
||||
if since_index > 0:
|
||||
logs = logs[since_index:]
|
||||
|
||||
|
||||
# Filter by worker
|
||||
if worker:
|
||||
logs = [log for log in logs if log.get('worker') == worker]
|
||||
|
||||
|
||||
return logs
|
||||
|
||||
def get_worker_logs_count(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of worker logs for a job.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of logs
|
||||
"""
|
||||
if settings.use_log_subcollection:
|
||||
# Try subcollection first
|
||||
count = self.firestore.get_logs_count_from_subcollection(job_id)
|
||||
if count > 0:
|
||||
return count
|
||||
# Fall through to check embedded array
|
||||
|
||||
# Embedded array (legacy jobs or fallback)
|
||||
job = self.get_job(job_id)
|
||||
if not job or not job.worker_logs:
|
||||
return 0
|
||||
return len(job.worker_logs)
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
"""
|
||||
Job notification service for sending emails on job state changes.
|
||||
|
||||
Handles:
|
||||
- Job completion emails (when job enters COMPLETE status)
|
||||
- Action reminder emails (when user is idle at blocking states)
|
||||
|
||||
This service orchestrates between the template service and email service,
|
||||
building the complete notification flow for jobs.
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.template_service import get_template_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Environment variable to enable/disable auto emails
|
||||
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
|
||||
|
||||
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
|
||||
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask email for logging to protect PII. Shows first char + domain."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 1:
|
||||
return f"*@{domain}"
|
||||
return f"{local[0]}***@{domain}"
|
||||
|
||||
|
||||
class JobNotificationService:
|
||||
"""
|
||||
Service for sending job-related email notifications.
|
||||
|
||||
Coordinates between template service (for message rendering) and
|
||||
email service (for sending).
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize notification service."""
|
||||
self.email_service = get_email_service()
|
||||
self.template_service = get_template_service()
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
|
||||
|
||||
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
|
||||
"""Build the lyrics review URL for a job."""
|
||||
import urllib.parse
|
||||
|
||||
review_ui_url = os.getenv("REVIEW_UI_URL", f"{self.frontend_url}/lyrics/")
|
||||
base_api_url = f"{self.backend_url}/api/review/{job_id}"
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{review_ui_url}?baseApiUrl={encoded_api_url}"
|
||||
if audio_hash:
|
||||
url += f"&audioHash={urllib.parse.quote(audio_hash, safe='')}"
|
||||
if review_token:
|
||||
url += f"&reviewToken={urllib.parse.quote(review_token, safe='')}"
|
||||
return url
|
||||
|
||||
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
|
||||
"""Build the instrumental selection URL for a job."""
|
||||
base_api_url = f"{self.backend_url}/api/jobs/{job_id}"
|
||||
|
||||
import urllib.parse
|
||||
encoded_api_url = urllib.parse.quote(base_api_url, safe='')
|
||||
|
||||
url = f"{self.frontend_url}/instrumental/?baseApiUrl={encoded_api_url}"
|
||||
if instrumental_token:
|
||||
url += f"&instrumentalToken={urllib.parse.quote(instrumental_token, safe='')}"
|
||||
return url
|
||||
|
||||
async def send_job_completion_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
brand_code: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send job completion email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
user_name: User's display name (optional)
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
brand_code: Release ID (e.g., "NOMAD-1178")
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping completion email")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the completion message using template service
|
||||
message_content = self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
# Send the email with CC to admin
|
||||
success = self.email_service.send_job_completion(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
artist=artist,
|
||||
title=title,
|
||||
brand_code=brand_code,
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send completion email for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending completion email for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
async def send_action_reminder_email(
|
||||
self,
|
||||
job_id: str,
|
||||
user_email: str,
|
||||
action_type: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
audio_hash: Optional[str] = None,
|
||||
review_token: Optional[str] = None,
|
||||
instrumental_token: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Send action-needed reminder email to user.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_email: User's email address
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
audio_hash: Audio hash for review URL
|
||||
review_token: Review token for URL
|
||||
instrumental_token: Instrumental token for URL
|
||||
|
||||
Returns:
|
||||
True if email was sent successfully
|
||||
"""
|
||||
if not ENABLE_AUTO_EMAILS:
|
||||
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
|
||||
return False
|
||||
|
||||
if not user_email:
|
||||
logger.warning(f"No user email for job {job_id}, skipping reminder")
|
||||
return False
|
||||
|
||||
try:
|
||||
# Render the appropriate template
|
||||
if action_type == "lyrics":
|
||||
review_url = self._build_review_url(job_id, audio_hash, review_token)
|
||||
message_content = self.template_service.render_action_needed_lyrics(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
review_url=review_url,
|
||||
)
|
||||
elif action_type == "instrumental":
|
||||
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
|
||||
message_content = self.template_service.render_action_needed_instrumental(
|
||||
name=user_name,
|
||||
artist=artist,
|
||||
title=title,
|
||||
instrumental_url=instrumental_url,
|
||||
)
|
||||
else:
|
||||
logger.error(f"Unknown action type: {action_type}")
|
||||
return False
|
||||
|
||||
# Send the reminder email (no CC)
|
||||
success = self.email_service.send_action_reminder(
|
||||
to_email=user_email,
|
||||
message_content=message_content,
|
||||
action_type=action_type,
|
||||
artist=artist,
|
||||
title=title,
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
|
||||
else:
|
||||
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
|
||||
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
|
||||
return False
|
||||
|
||||
def get_completion_message(
|
||||
self,
|
||||
job_id: str,
|
||||
user_name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Get the rendered completion message for a job (for admin copy functionality).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
user_name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
|
||||
Returns:
|
||||
Rendered message content as plain text
|
||||
"""
|
||||
return self.template_service.render_job_completion(
|
||||
name=user_name,
|
||||
youtube_url=youtube_url,
|
||||
dropbox_url=dropbox_url,
|
||||
artist=artist,
|
||||
title=title,
|
||||
job_id=job_id,
|
||||
feedback_url=FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
# Global instance
|
||||
_job_notification_service: Optional[JobNotificationService] = None
|
||||
|
||||
|
||||
def get_job_notification_service() -> JobNotificationService:
|
||||
"""Get the global job notification service instance."""
|
||||
global _job_notification_service
|
||||
if _job_notification_service is None:
|
||||
_job_notification_service = JobNotificationService()
|
||||
return _job_notification_service
|
||||
@@ -0,0 +1,98 @@
|
||||
"""Langfuse callback handler preloader for container startup.
|
||||
|
||||
Initializes the Langfuse callback handler at container startup to avoid
|
||||
slow initialization during request processing. The CallbackHandler()
|
||||
constructor makes blocking network calls to the Langfuse API, which can
|
||||
take 3+ minutes on Cloud Run cold starts.
|
||||
|
||||
See docs/archive/2026-01-08-performance-investigation.md for background.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded handler
|
||||
_preloaded_handler: Optional[Any] = None
|
||||
_initialization_attempted: bool = False
|
||||
|
||||
|
||||
def preload_langfuse_handler() -> Optional[Any]:
|
||||
"""Preload Langfuse callback handler at startup.
|
||||
|
||||
Only initializes if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY
|
||||
environment variables are set.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not configured
|
||||
"""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
|
||||
if _initialization_attempted:
|
||||
logger.debug("Langfuse initialization already attempted")
|
||||
return _preloaded_handler
|
||||
|
||||
_initialization_attempted = True
|
||||
|
||||
# Check if Langfuse is configured
|
||||
public_key = os.getenv("LANGFUSE_PUBLIC_KEY")
|
||||
secret_key = os.getenv("LANGFUSE_SECRET_KEY")
|
||||
|
||||
if not (public_key and secret_key):
|
||||
logger.info("Langfuse not configured (missing keys), skipping preload")
|
||||
return None
|
||||
|
||||
logger.info("Preloading Langfuse callback handler...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
from langfuse.langchain import CallbackHandler
|
||||
|
||||
# Initialize the handler - this is the slow part that makes network calls
|
||||
_preloaded_handler = CallbackHandler()
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
host = os.getenv("LANGFUSE_HOST", "cloud.langfuse.com")
|
||||
logger.info(f"Langfuse handler preloaded in {elapsed:.2f}s (host: {host})")
|
||||
|
||||
return _preloaded_handler
|
||||
|
||||
except ImportError:
|
||||
logger.warning("langfuse package not installed, skipping preload")
|
||||
return None
|
||||
except Exception as e:
|
||||
elapsed = time.time() - start_time
|
||||
logger.error(f"Failed to preload Langfuse handler after {elapsed:.2f}s: {e}")
|
||||
# Don't raise - Langfuse is optional
|
||||
return None
|
||||
|
||||
|
||||
def get_preloaded_langfuse_handler() -> Optional[Any]:
|
||||
"""Get the preloaded Langfuse callback handler if available.
|
||||
|
||||
Returns:
|
||||
The preloaded CallbackHandler, or None if not preloaded/configured
|
||||
"""
|
||||
return _preloaded_handler
|
||||
|
||||
|
||||
def is_langfuse_preloaded() -> bool:
|
||||
"""Check if Langfuse handler has been preloaded."""
|
||||
return _preloaded_handler is not None
|
||||
|
||||
|
||||
def is_langfuse_configured() -> bool:
|
||||
"""Check if Langfuse environment variables are configured."""
|
||||
return bool(
|
||||
os.getenv("LANGFUSE_PUBLIC_KEY") and os.getenv("LANGFUSE_SECRET_KEY")
|
||||
)
|
||||
|
||||
|
||||
def clear_preloaded_handler() -> None:
|
||||
"""Clear preloaded handler. Useful for testing."""
|
||||
global _preloaded_handler, _initialization_attempted
|
||||
_preloaded_handler = None
|
||||
_initialization_attempted = False
|
||||
@@ -0,0 +1,590 @@
|
||||
"""
|
||||
Local Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based video encoding functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker as a fallback) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Concatenating title, karaoke, and end videos
|
||||
- Encoding to multiple output formats (4K lossless, 4K lossy, 720p, MKV)
|
||||
- Hardware acceleration detection and fallback to software encoding
|
||||
- Remuxing video with instrumental audio
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingConfig:
|
||||
"""Configuration for video encoding."""
|
||||
title_video: str # Path to title video
|
||||
karaoke_video: str # Path to karaoke video (with vocals)
|
||||
instrumental_audio: str # Path to instrumental audio
|
||||
end_video: Optional[str] = None # Optional path to end credits video
|
||||
|
||||
# Output paths
|
||||
output_karaoke_mp4: Optional[str] = None
|
||||
output_with_vocals_mp4: Optional[str] = None
|
||||
output_lossless_4k_mp4: Optional[str] = None
|
||||
output_lossy_4k_mp4: Optional[str] = None
|
||||
output_lossless_mkv: Optional[str] = None
|
||||
output_720p_mp4: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingResult:
|
||||
"""Result of video encoding operation."""
|
||||
success: bool
|
||||
output_files: Dict[str, str]
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based video encoding.
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# MP4 flags for better compatibility
|
||||
MP4_FLAGS = "-movflags +faststart"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
log_level: int = logging.INFO,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding service.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log commands without executing them
|
||||
log_level: Logging level (affects FFmpeg verbosity)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.log_level = log_level
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._hwaccel_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._scale_filter: Optional[str] = None
|
||||
self._hwaccel_decode_flags: Optional[str] = None
|
||||
self._aac_codec: Optional[str] = None
|
||||
|
||||
# Build FFmpeg base command
|
||||
self._ffmpeg_base_command = self._build_ffmpeg_base_command()
|
||||
|
||||
def _build_ffmpeg_base_command(self) -> str:
|
||||
"""Build the FFmpeg base command with appropriate flags."""
|
||||
# Use bundled FFmpeg for frozen builds
|
||||
ffmpeg_path = os.path.join(sys._MEIPASS, "ffmpeg.exe") if getattr(sys, "frozen", False) else "ffmpeg"
|
||||
|
||||
base_cmd = f"{ffmpeg_path} -hide_banner -nostats -y"
|
||||
|
||||
if self.log_level == logging.DEBUG:
|
||||
base_cmd += " -loglevel verbose"
|
||||
else:
|
||||
base_cmd += " -loglevel fatal"
|
||||
|
||||
return base_cmd
|
||||
|
||||
def _detect_hardware_acceleration(self) -> Tuple[bool, str, str, str, str]:
|
||||
"""
|
||||
Detect available hardware acceleration.
|
||||
|
||||
Returns:
|
||||
Tuple of (hwaccel_available, video_encoder, scale_filter,
|
||||
hwaccel_decode_flags, aac_codec)
|
||||
"""
|
||||
self.logger.info("Detecting hardware acceleration capabilities...")
|
||||
|
||||
# Try NVENC (NVIDIA)
|
||||
try:
|
||||
test_cmd = f"{self._ffmpeg_base_command} -hide_banner -loglevel error " \
|
||||
f"-f lavfi -i testsrc=duration=1:size=320x240:rate=1 " \
|
||||
f"-c:v h264_nvenc -f null -"
|
||||
subprocess.run(
|
||||
test_cmd, shell=True, check=True,
|
||||
capture_output=True, timeout=30
|
||||
)
|
||||
self.logger.info("NVIDIA NVENC hardware acceleration available")
|
||||
return (
|
||||
True,
|
||||
"h264_nvenc",
|
||||
"scale_cuda",
|
||||
"-hwaccel cuda -hwaccel_output_format cuda",
|
||||
"aac"
|
||||
)
|
||||
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# No hardware acceleration available
|
||||
self.logger.info("No hardware acceleration available, using software encoding")
|
||||
return (False, "libx264", "scale", "", "aac")
|
||||
|
||||
@property
|
||||
def hwaccel_available(self) -> bool:
|
||||
"""Check if hardware acceleration is available."""
|
||||
if self._hwaccel_available is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def scale_filter(self) -> str:
|
||||
"""Get the scale filter to use."""
|
||||
if self._scale_filter is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._scale_filter
|
||||
|
||||
@property
|
||||
def hwaccel_decode_flags(self) -> str:
|
||||
"""Get hardware acceleration decode flags."""
|
||||
if self._hwaccel_decode_flags is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._hwaccel_decode_flags
|
||||
|
||||
@property
|
||||
def aac_codec(self) -> str:
|
||||
"""Get the AAC codec to use."""
|
||||
if self._aac_codec is None:
|
||||
self._detect_and_set_hwaccel()
|
||||
return self._aac_codec
|
||||
|
||||
def _detect_and_set_hwaccel(self) -> None:
|
||||
"""Detect and set hardware acceleration settings."""
|
||||
(
|
||||
self._hwaccel_available,
|
||||
self._video_encoder,
|
||||
self._scale_filter,
|
||||
self._hwaccel_decode_flags,
|
||||
self._aac_codec
|
||||
) = self._detect_hardware_acceleration()
|
||||
|
||||
def _get_nvenc_quality_settings(self, preset: str = "medium") -> str:
|
||||
"""Get NVENC quality settings for different presets."""
|
||||
if not self.hwaccel_available:
|
||||
return ""
|
||||
|
||||
settings = {
|
||||
"lossless": "-preset p7 -tune hq -rc vbr -cq 0 -qmin 0 -qmax 0",
|
||||
"high": "-preset p7 -tune hq -rc vbr -cq 19 -b:v 0",
|
||||
"medium": "-preset p4 -tune hq -rc vbr -cq 23 -b:v 0",
|
||||
"fast": "-preset p2 -tune ll -rc vbr -cq 28 -b:v 0",
|
||||
}
|
||||
return settings.get(preset, settings["medium"])
|
||||
|
||||
def _execute_command(
|
||||
self,
|
||||
command: str,
|
||||
description: str,
|
||||
timeout: int = 3600, # 1 hour default
|
||||
) -> bool:
|
||||
"""
|
||||
Execute an FFmpeg command.
|
||||
|
||||
Args:
|
||||
command: The FFmpeg command to execute
|
||||
description: Human-readable description of the operation
|
||||
timeout: Command timeout in seconds
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Executing: {description}")
|
||||
self.logger.debug(f"Command: {command}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would execute: {command}")
|
||||
return True
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
command,
|
||||
shell=True,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout
|
||||
)
|
||||
self.logger.info(f"Completed: {description}")
|
||||
return True
|
||||
except subprocess.CalledProcessError as e:
|
||||
self.logger.error(f"Failed: {description}")
|
||||
self.logger.error(f"Error: {e.stderr}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error(f"Timeout: {description}")
|
||||
return False
|
||||
|
||||
def _execute_command_with_fallback(
|
||||
self,
|
||||
gpu_command: str,
|
||||
cpu_command: str,
|
||||
description: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Execute a command with GPU, falling back to CPU if it fails.
|
||||
|
||||
Args:
|
||||
gpu_command: Hardware-accelerated command
|
||||
cpu_command: Software fallback command
|
||||
description: Human-readable description
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
if self.hwaccel_available:
|
||||
self.logger.info(f"Trying hardware-accelerated encoding for: {description}")
|
||||
if self._execute_command(gpu_command, description):
|
||||
return True
|
||||
self.logger.warning(f"Hardware encoding failed, falling back to software")
|
||||
|
||||
return self._execute_command(cpu_command, description)
|
||||
|
||||
def remux_with_instrumental(
|
||||
self,
|
||||
input_video: str,
|
||||
instrumental_audio: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Remux video with instrumental audio track.
|
||||
|
||||
Args:
|
||||
input_video: Path to input video file
|
||||
instrumental_audio: Path to instrumental audio file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_video}" '
|
||||
f'-i "{instrumental_audio}" -map 0:v -map 1:a -c copy '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Remuxing with instrumental audio")
|
||||
|
||||
def convert_mov_to_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Convert MOV to MP4 format.
|
||||
|
||||
Args:
|
||||
input_file: Path to input MOV file
|
||||
output_file: Path for output MP4 file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -c:a copy {self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Converting MOV to MP4"
|
||||
)
|
||||
|
||||
def encode_lossless_mp4(
|
||||
self,
|
||||
title_video: str,
|
||||
karaoke_video: str,
|
||||
output_file: str,
|
||||
end_video: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossless 4K MP4 by concatenating title, karaoke, and optionally end videos.
|
||||
|
||||
Args:
|
||||
title_video: Path to title video
|
||||
karaoke_video: Path to karaoke video
|
||||
output_file: Path for output file
|
||||
end_video: Optional path to end credits video
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
# Quote file paths
|
||||
title_quoted = shlex.quote(os.path.abspath(title_video))
|
||||
karaoke_quoted = shlex.quote(os.path.abspath(karaoke_video))
|
||||
|
||||
# Build filter and inputs for concatenation
|
||||
if end_video and os.path.isfile(end_video):
|
||||
end_quoted = shlex.quote(os.path.abspath(end_video))
|
||||
extra_input = f"-i {end_quoted}"
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0][2:v:0][2:a:0]'
|
||||
'concat=n=3:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
else:
|
||||
extra_input = ""
|
||||
concat_filter = (
|
||||
'-filter_complex "[0:v:0][0:a:0][1:v:0][1:a:0]'
|
||||
'concat=n=2:v=1:a=1[outv][outa]"'
|
||||
)
|
||||
|
||||
gpu_command = (
|
||||
f"{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i {title_quoted} "
|
||||
f"{self.hwaccel_decode_flags} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v {self.video_encoder} '
|
||||
f'{self._get_nvenc_quality_settings("lossless")} -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f"{self._ffmpeg_base_command} -i {title_quoted} -i {karaoke_quoted} {extra_input} "
|
||||
f'{concat_filter} -map "[outv]" -map "[outa]" -c:v libx264 -c:a pcm_s16le '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding lossless 4K MP4"
|
||||
)
|
||||
|
||||
def encode_lossy_mp4(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create lossy 4K MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file (typically lossless MP4)
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a {self.aac_codec} -ar 48000 -b:a 320k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Encoding lossy 4K MP4 with AAC")
|
||||
|
||||
def encode_lossless_mkv(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create MKV with FLAC audio (for YouTube upload).
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v copy -c:a flac "{output_file}"'
|
||||
)
|
||||
return self._execute_command(command, "Creating MKV with FLAC for YouTube")
|
||||
|
||||
def encode_720p(
|
||||
self,
|
||||
input_file: str,
|
||||
output_file: str,
|
||||
) -> bool:
|
||||
"""
|
||||
Create 720p MP4 with AAC audio.
|
||||
|
||||
Args:
|
||||
input_file: Path to input file
|
||||
output_file: Path for output file
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
gpu_command = (
|
||||
f'{self._ffmpeg_base_command} {self.hwaccel_decode_flags} -i "{input_file}" '
|
||||
f'-c:v {self.video_encoder} -vf "{self.scale_filter}=1280:720" '
|
||||
f'{self._get_nvenc_quality_settings("medium")} -b:v 2000k '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
cpu_command = (
|
||||
f'{self._ffmpeg_base_command} -i "{input_file}" '
|
||||
f'-c:v libx264 -vf "scale=1280:720" -b:v 2000k -preset medium -tune animation '
|
||||
f'-c:a {self.aac_codec} -ar 48000 -b:a 128k '
|
||||
f'{self.MP4_FLAGS} "{output_file}"'
|
||||
)
|
||||
return self._execute_command_with_fallback(
|
||||
gpu_command, cpu_command, "Encoding 720p MP4"
|
||||
)
|
||||
|
||||
def encode_all_formats(
|
||||
self,
|
||||
config: EncodingConfig,
|
||||
) -> EncodingResult:
|
||||
"""
|
||||
Encode video to all output formats.
|
||||
|
||||
This performs the full encoding pipeline:
|
||||
1. Remux with instrumental audio
|
||||
2. Convert to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenated)
|
||||
4. Encode lossy 4K MP4
|
||||
5. Encode lossless MKV (for YouTube)
|
||||
6. Encode 720p MP4
|
||||
|
||||
Args:
|
||||
config: Encoding configuration with input/output paths
|
||||
|
||||
Returns:
|
||||
EncodingResult with success status and output file paths
|
||||
"""
|
||||
output_files = {}
|
||||
|
||||
try:
|
||||
# Step 1: Remux with instrumental audio
|
||||
if config.output_karaoke_mp4:
|
||||
self.logger.info("[Step 1/6] Remuxing video with instrumental audio...")
|
||||
if not self.remux_with_instrumental(
|
||||
config.karaoke_video,
|
||||
config.instrumental_audio,
|
||||
config.output_karaoke_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to remux with instrumental audio"
|
||||
)
|
||||
output_files["karaoke_mp4"] = config.output_karaoke_mp4
|
||||
|
||||
# Step 2: Convert to MP4 if needed
|
||||
if config.output_with_vocals_mp4:
|
||||
if not config.karaoke_video.endswith(".mp4"):
|
||||
self.logger.info("[Step 2/6] Converting karaoke video to MP4...")
|
||||
if not self.convert_mov_to_mp4(
|
||||
config.karaoke_video,
|
||||
config.output_with_vocals_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to convert to MP4"
|
||||
)
|
||||
output_files["with_vocals_mp4"] = config.output_with_vocals_mp4
|
||||
else:
|
||||
self.logger.info("[Step 2/6] Skipped - video already MP4")
|
||||
|
||||
# Step 3: Encode lossless 4K MP4
|
||||
if config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 3/6] Encoding lossless 4K MP4...")
|
||||
karaoke_for_concat = config.output_karaoke_mp4 or config.karaoke_video
|
||||
if not self.encode_lossless_mp4(
|
||||
config.title_video,
|
||||
karaoke_for_concat,
|
||||
config.output_lossless_4k_mp4,
|
||||
config.end_video
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossless 4K MP4"
|
||||
)
|
||||
output_files["lossless_4k_mp4"] = config.output_lossless_4k_mp4
|
||||
|
||||
# Step 4: Encode lossy 4K MP4
|
||||
if config.output_lossy_4k_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 4/6] Encoding lossy 4K MP4...")
|
||||
if not self.encode_lossy_mp4(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossy_4k_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode lossy 4K MP4"
|
||||
)
|
||||
output_files["lossy_4k_mp4"] = config.output_lossy_4k_mp4
|
||||
|
||||
# Step 5: Create MKV with FLAC audio
|
||||
if config.output_lossless_mkv and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 5/6] Creating MKV with FLAC audio...")
|
||||
if not self.encode_lossless_mkv(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_lossless_mkv
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to create MKV"
|
||||
)
|
||||
output_files["lossless_mkv"] = config.output_lossless_mkv
|
||||
|
||||
# Step 6: Encode 720p version
|
||||
if config.output_720p_mp4 and config.output_lossless_4k_mp4:
|
||||
self.logger.info("[Step 6/6] Encoding 720p MP4...")
|
||||
if not self.encode_720p(
|
||||
config.output_lossless_4k_mp4,
|
||||
config.output_720p_mp4
|
||||
):
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error="Failed to encode 720p"
|
||||
)
|
||||
output_files["720p_mp4"] = config.output_720p_mp4
|
||||
|
||||
self.logger.info("All encoding steps completed successfully")
|
||||
return EncodingResult(success=True, output_files=output_files)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Encoding failed with exception: {e}")
|
||||
return EncodingResult(
|
||||
success=False,
|
||||
output_files=output_files,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_local_encoding_service: Optional[LocalEncodingService] = None
|
||||
|
||||
|
||||
def get_local_encoding_service(**kwargs) -> LocalEncodingService:
|
||||
"""
|
||||
Get a local encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalEncodingService
|
||||
|
||||
Returns:
|
||||
LocalEncodingService instance
|
||||
"""
|
||||
global _local_encoding_service
|
||||
|
||||
if _local_encoding_service is None:
|
||||
_local_encoding_service = LocalEncodingService(**kwargs)
|
||||
|
||||
return _local_encoding_service
|
||||
@@ -0,0 +1,407 @@
|
||||
"""
|
||||
Local Preview Encoding Service.
|
||||
|
||||
Provides local FFmpeg-based preview video encoding functionality, extracted from
|
||||
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
|
||||
is used across local CLI, Cloud Run, and GCE worker environments.
|
||||
|
||||
This service handles:
|
||||
- Generating preview videos with ASS subtitle overlay
|
||||
- Hardware acceleration detection (NVENC) with fallback to software encoding
|
||||
- Background image/color support
|
||||
- Custom font support
|
||||
- Optimized settings for fast preview generation
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingConfig:
|
||||
"""Configuration for preview video encoding."""
|
||||
ass_path: str # Path to ASS subtitles file
|
||||
audio_path: str # Path to audio file
|
||||
output_path: str # Path for output video file
|
||||
|
||||
# Optional background settings
|
||||
background_image_path: Optional[str] = None # Path to background image
|
||||
background_color: str = "black" # Fallback background color
|
||||
|
||||
# Optional font settings
|
||||
font_path: Optional[str] = None # Path to custom font file
|
||||
|
||||
|
||||
@dataclass
|
||||
class PreviewEncodingResult:
|
||||
"""Result of preview encoding operation."""
|
||||
success: bool
|
||||
output_path: Optional[str] = None
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class LocalPreviewEncodingService:
|
||||
"""
|
||||
Service for local FFmpeg-based preview video encoding.
|
||||
|
||||
This is the single source of truth for preview encoding logic, used by:
|
||||
- Local CLI (via VideoGenerator which delegates here)
|
||||
- Cloud Run (when GCE is unavailable)
|
||||
- GCE worker (via installed wheel)
|
||||
|
||||
Supports hardware acceleration (NVENC) with automatic fallback
|
||||
to software encoding (libx264) when unavailable.
|
||||
"""
|
||||
|
||||
# Preview video settings - these are the canonical values
|
||||
PREVIEW_WIDTH = 480
|
||||
PREVIEW_HEIGHT = 270
|
||||
PREVIEW_FPS = 24
|
||||
PREVIEW_AUDIO_BITRATE = "96k"
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local preview encoding service.
|
||||
|
||||
Args:
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Hardware acceleration settings (detected on first use)
|
||||
self._nvenc_available: Optional[bool] = None
|
||||
self._video_encoder: Optional[str] = None
|
||||
self._hwaccel_flags: Optional[List[str]] = None
|
||||
|
||||
def _detect_nvenc_support(self) -> bool:
|
||||
"""
|
||||
Detect if NVENC hardware encoding is available.
|
||||
|
||||
Returns:
|
||||
True if NVENC is available, False otherwise
|
||||
"""
|
||||
try:
|
||||
self.logger.info("Detecting NVENC hardware acceleration...")
|
||||
|
||||
# Test h264_nvenc encoder directly
|
||||
test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
|
||||
"-c:v", "h264_nvenc", "-f", "null", "-"
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
# Try alternative test with different source
|
||||
alt_test_cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
|
||||
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
|
||||
]
|
||||
|
||||
alt_result = subprocess.run(
|
||||
alt_test_cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if alt_result.returncode == 0:
|
||||
self.logger.info("NVENC hardware acceleration available")
|
||||
return True
|
||||
|
||||
self.logger.info("NVENC not available, using software encoding")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.debug("NVENC detection timed out")
|
||||
return False
|
||||
except Exception as e:
|
||||
self.logger.debug(f"NVENC detection failed: {e}")
|
||||
return False
|
||||
|
||||
def _configure_hardware_acceleration(self) -> None:
|
||||
"""Configure hardware acceleration settings based on detected capabilities."""
|
||||
if self._nvenc_available is None:
|
||||
self._nvenc_available = self._detect_nvenc_support()
|
||||
|
||||
if self._nvenc_available:
|
||||
self._video_encoder = "h264_nvenc"
|
||||
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
|
||||
self.logger.info("Using NVENC hardware acceleration for preview encoding")
|
||||
else:
|
||||
self._video_encoder = "libx264"
|
||||
self._hwaccel_flags = []
|
||||
self.logger.info("Using software encoding (libx264) for preview")
|
||||
|
||||
@property
|
||||
def nvenc_available(self) -> bool:
|
||||
"""Check if NVENC hardware acceleration is available."""
|
||||
if self._nvenc_available is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._nvenc_available
|
||||
|
||||
@property
|
||||
def video_encoder(self) -> str:
|
||||
"""Get the video encoder to use."""
|
||||
if self._video_encoder is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._video_encoder
|
||||
|
||||
@property
|
||||
def hwaccel_flags(self) -> List[str]:
|
||||
"""Get hardware acceleration flags."""
|
||||
if self._hwaccel_flags is None:
|
||||
self._configure_hardware_acceleration()
|
||||
return self._hwaccel_flags
|
||||
|
||||
def _escape_ffmpeg_filter_path(self, path: str) -> str:
|
||||
"""
|
||||
Escape a path for FFmpeg filter expressions (for subprocess without shell).
|
||||
|
||||
When using subprocess with a command list (no shell), FFmpeg receives the
|
||||
filter string directly. FFmpeg's filter parser requires escaping:
|
||||
- Backslashes: double them (\\ -> \\\\)
|
||||
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
|
||||
- Spaces: escape with backslash ( -> \\ )
|
||||
- Special characters: :,[];
|
||||
|
||||
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
|
||||
"""
|
||||
# First escape existing backslashes
|
||||
escaped = path.replace("\\", "\\\\")
|
||||
# Escape single quotes
|
||||
escaped = escaped.replace("'", "\\\\\\'")
|
||||
# Escape spaces
|
||||
escaped = escaped.replace(" ", "\\ ")
|
||||
# Escape FFmpeg filter special characters
|
||||
escaped = escaped.replace(":", "\\:")
|
||||
escaped = escaped.replace(",", "\\,")
|
||||
escaped = escaped.replace("[", "\\[")
|
||||
escaped = escaped.replace("]", "\\]")
|
||||
escaped = escaped.replace(";", "\\;")
|
||||
return escaped
|
||||
|
||||
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
|
||||
"""
|
||||
Build ASS filter with optional font directory support.
|
||||
|
||||
Args:
|
||||
ass_path: Path to ASS subtitles file
|
||||
font_path: Optional path to custom font file
|
||||
|
||||
Returns:
|
||||
FFmpeg ASS filter string
|
||||
"""
|
||||
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
|
||||
ass_filter = f"ass={escaped_ass_path}"
|
||||
|
||||
if font_path and os.path.isfile(font_path):
|
||||
font_dir = os.path.dirname(font_path)
|
||||
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
|
||||
ass_filter += f":fontsdir={escaped_font_dir}"
|
||||
self.logger.debug(f"Using font directory: {font_dir}")
|
||||
|
||||
return ass_filter
|
||||
|
||||
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
|
||||
"""
|
||||
Build FFmpeg command for preview video generation.
|
||||
|
||||
This is the canonical preview encoding command, ensuring consistency
|
||||
across all environments (local CLI, Cloud Run, GCE worker).
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
FFmpeg command as a list of arguments
|
||||
"""
|
||||
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
"-hide_banner",
|
||||
"-loglevel", "error",
|
||||
"-r", str(self.PREVIEW_FPS),
|
||||
]
|
||||
|
||||
# Add hardware acceleration flags if available
|
||||
cmd.extend(self.hwaccel_flags)
|
||||
|
||||
# Input source (background image or solid color)
|
||||
if config.background_image_path and os.path.isfile(config.background_image_path):
|
||||
self.logger.debug(f"Using background image: {config.background_image_path}")
|
||||
cmd.extend([
|
||||
"-loop", "1",
|
||||
"-i", config.background_image_path,
|
||||
])
|
||||
# Build video filter with scaling and ASS subtitles
|
||||
video_filter = (
|
||||
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
|
||||
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
|
||||
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f"Using solid {config.background_color} background")
|
||||
cmd.extend([
|
||||
"-f", "lavfi",
|
||||
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
|
||||
])
|
||||
# Just ASS subtitles, no scaling needed
|
||||
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
|
||||
|
||||
cmd.extend([
|
||||
"-i", config.audio_path,
|
||||
"-vf", video_filter,
|
||||
"-c:a", "aac",
|
||||
"-b:a", self.PREVIEW_AUDIO_BITRATE,
|
||||
"-c:v", self.video_encoder,
|
||||
])
|
||||
|
||||
# Add encoder-specific settings optimized for speed
|
||||
if self.nvenc_available:
|
||||
cmd.extend([
|
||||
"-preset", "p1", # Fastest NVENC preset
|
||||
"-tune", "ll", # Low latency
|
||||
"-rc", "cbr", # Constant bitrate for speed
|
||||
"-b:v", "800k", # Lower bitrate for speed
|
||||
"-profile:v", "baseline", # Most compatible profile
|
||||
"-level", "3.1", # Lower level for speed
|
||||
])
|
||||
self.logger.debug("Using NVENC with maximum speed settings")
|
||||
else:
|
||||
cmd.extend([
|
||||
"-profile:v", "baseline",
|
||||
"-level", "3.0",
|
||||
"-preset", "superfast",
|
||||
"-tune", "fastdecode",
|
||||
"-b:v", "600k",
|
||||
"-maxrate", "800k",
|
||||
"-bufsize", "1200k",
|
||||
"-crf", "28",
|
||||
])
|
||||
self.logger.debug("Using software encoding with maximum speed settings")
|
||||
|
||||
cmd.extend([
|
||||
"-pix_fmt", "yuv420p", # Required for browser compatibility
|
||||
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
|
||||
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
|
||||
"-keyint_min", "48",
|
||||
"-sc_threshold", "0", # Disable scene change detection for speed
|
||||
"-threads", "0", # Use all available CPU threads
|
||||
"-shortest",
|
||||
"-y",
|
||||
config.output_path,
|
||||
])
|
||||
|
||||
return cmd
|
||||
|
||||
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
|
||||
"""
|
||||
Encode a preview video.
|
||||
|
||||
Args:
|
||||
config: Preview encoding configuration
|
||||
|
||||
Returns:
|
||||
PreviewEncodingResult with success status and output path
|
||||
"""
|
||||
self.logger.info(f"Encoding preview video: {config.output_path}")
|
||||
|
||||
# Validate input files
|
||||
if not os.path.isfile(config.ass_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"ASS subtitles file not found: {config.ass_path}"
|
||||
)
|
||||
|
||||
if not os.path.isfile(config.audio_path):
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"Audio file not found: {config.audio_path}"
|
||||
)
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir = os.path.dirname(config.output_path)
|
||||
if output_dir:
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
try:
|
||||
# Build and execute FFmpeg command
|
||||
cmd = self._build_preview_ffmpeg_command(config)
|
||||
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=300 # 5 minute timeout for preview encoding
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
|
||||
self.logger.error(f"FFmpeg failed: {error_msg}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=f"FFmpeg preview encoding failed: {error_msg}"
|
||||
)
|
||||
|
||||
self.logger.info(f"Preview encoded successfully: {config.output_path}")
|
||||
return PreviewEncodingResult(
|
||||
success=True,
|
||||
output_path=config.output_path
|
||||
)
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
self.logger.error("Preview encoding timed out")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error="Preview encoding timed out after 5 minutes"
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.error(f"Preview encoding failed: {e}")
|
||||
return PreviewEncodingResult(
|
||||
success=False,
|
||||
error=str(e)
|
||||
)
|
||||
|
||||
|
||||
# Singleton instance and factory function
|
||||
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
|
||||
|
||||
|
||||
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
|
||||
"""
|
||||
Get a local preview encoding service instance.
|
||||
|
||||
Args:
|
||||
**kwargs: Arguments passed to LocalPreviewEncodingService
|
||||
|
||||
Returns:
|
||||
LocalPreviewEncodingService instance
|
||||
"""
|
||||
global _local_preview_encoding_service
|
||||
|
||||
if _local_preview_encoding_service is None:
|
||||
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
|
||||
|
||||
return _local_preview_encoding_service
|
||||
@@ -0,0 +1,216 @@
|
||||
"""
|
||||
LyricsTranscriber cache synchronization with GCS.
|
||||
|
||||
This service persists LyricsTranscriber's cache files to GCS so that
|
||||
cloud workers (Cloud Run instances) can share cache across containers.
|
||||
|
||||
Cache files are stored flat in GCS under lyrics-transcriber-cache/ prefix:
|
||||
- Transcription: {provider}_{audio_hash}_raw.json, {provider}_{audio_hash}_converted.json
|
||||
- Lyrics: {provider}_{artist_title_hash}_raw.json, {provider}_{artist_title_hash}_converted.json
|
||||
|
||||
Hash computation matches LyricsTranscriber's implementation exactly:
|
||||
- Audio hash: MD5 of audio file bytes
|
||||
- Lyrics hash: MD5 of "{artist.lower()}_{title.lower()}"
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Providers that use audio file hash as cache key
|
||||
TRANSCRIPTION_PROVIDERS = ["audioshake", "whisper", "localwhisper"]
|
||||
|
||||
# Providers that use artist+title hash as cache key
|
||||
LYRICS_PROVIDERS = ["genius", "spotify", "lrclib", "musixmatch"]
|
||||
|
||||
# Cache file suffixes
|
||||
CACHE_SUFFIXES = ["raw", "converted"]
|
||||
|
||||
|
||||
class LyricsCacheService:
|
||||
"""Service to sync LyricsTranscriber cache with GCS."""
|
||||
|
||||
GCS_CACHE_PREFIX = "lyrics-transcriber-cache/"
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""Initialize the cache service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance. If None, creates a new one.
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
|
||||
def compute_audio_hash(self, audio_path: str) -> str:
|
||||
"""Compute MD5 hash of audio file bytes.
|
||||
|
||||
This matches LyricsTranscriber's _get_file_hash() method exactly.
|
||||
|
||||
Args:
|
||||
audio_path: Path to the audio file.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of the file contents.
|
||||
"""
|
||||
md5_hash = hashlib.md5()
|
||||
with open(audio_path, "rb") as f:
|
||||
for chunk in iter(lambda: f.read(4096), b""):
|
||||
md5_hash.update(chunk)
|
||||
return md5_hash.hexdigest()
|
||||
|
||||
def compute_lyrics_hash(self, artist: str, title: str) -> str:
|
||||
"""Compute MD5 hash of artist and title.
|
||||
|
||||
This matches LyricsTranscriber's _get_artist_title_hash() method exactly.
|
||||
|
||||
Args:
|
||||
artist: Artist name.
|
||||
title: Track title.
|
||||
|
||||
Returns:
|
||||
MD5 hex digest of "{artist.lower()}_{title.lower()}".
|
||||
"""
|
||||
combined = f"{artist.lower()}_{title.lower()}"
|
||||
return hashlib.md5(combined.encode()).hexdigest()
|
||||
|
||||
def _get_cache_filenames(
|
||||
self, providers: List[str], hash_value: str
|
||||
) -> List[str]:
|
||||
"""Generate list of possible cache filenames for given providers and hash.
|
||||
|
||||
Args:
|
||||
providers: List of provider names (e.g., ["audioshake", "whisper"]).
|
||||
hash_value: The hash to use in filenames.
|
||||
|
||||
Returns:
|
||||
List of filenames like ["audioshake_abc123_raw.json", ...].
|
||||
"""
|
||||
filenames = []
|
||||
for provider in providers:
|
||||
for suffix in CACHE_SUFFIXES:
|
||||
filenames.append(f"{provider}_{hash_value}_{suffix}.json")
|
||||
return filenames
|
||||
|
||||
def sync_cache_from_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Download relevant cache files from GCS to local directory.
|
||||
|
||||
Downloads cache files for both transcription (audio hash) and
|
||||
lyrics (artist+title hash) providers.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory to download cache files to.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"downloaded": N, "not_found": M, "errors": E}
|
||||
"""
|
||||
os.makedirs(local_cache_dir, exist_ok=True)
|
||||
|
||||
stats = {"downloaded": 0, "not_found": 0, "errors": 0}
|
||||
|
||||
# Get all possible cache filenames
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
all_files = transcription_files + lyrics_files
|
||||
|
||||
for filename in all_files:
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
|
||||
try:
|
||||
if self.storage.file_exists(gcs_path):
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
logger.info(f"Cache hit: downloaded {filename}")
|
||||
stats["downloaded"] += 1
|
||||
else:
|
||||
logger.debug(f"Cache miss: {filename} not in GCS")
|
||||
stats["not_found"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error downloading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync from GCS complete: "
|
||||
f"{stats['downloaded']} downloaded, "
|
||||
f"{stats['not_found']} not found, "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
|
||||
def sync_cache_to_gcs(
|
||||
self,
|
||||
local_cache_dir: str,
|
||||
audio_hash: str,
|
||||
lyrics_hash: str,
|
||||
) -> Dict[str, int]:
|
||||
"""Upload new cache files from local directory to GCS.
|
||||
|
||||
Only uploads files that match expected cache patterns for the given
|
||||
hashes and don't already exist in GCS.
|
||||
|
||||
Args:
|
||||
local_cache_dir: Local directory with cache files.
|
||||
audio_hash: MD5 hash of audio file.
|
||||
lyrics_hash: MD5 hash of artist+title.
|
||||
|
||||
Returns:
|
||||
Dict with counts: {"uploaded": N, "skipped": M, "errors": E}
|
||||
"""
|
||||
stats = {"uploaded": 0, "skipped": 0, "errors": 0}
|
||||
|
||||
if not os.path.exists(local_cache_dir):
|
||||
logger.warning(f"Local cache dir does not exist: {local_cache_dir}")
|
||||
return stats
|
||||
|
||||
# Get all possible cache filenames we're interested in
|
||||
transcription_files = self._get_cache_filenames(
|
||||
TRANSCRIPTION_PROVIDERS, audio_hash
|
||||
)
|
||||
lyrics_files = self._get_cache_filenames(LYRICS_PROVIDERS, lyrics_hash)
|
||||
expected_files = set(transcription_files + lyrics_files)
|
||||
|
||||
# Check each file in local cache dir
|
||||
for filename in os.listdir(local_cache_dir):
|
||||
# Only process files we expect (matching our hash patterns)
|
||||
if filename not in expected_files:
|
||||
continue
|
||||
|
||||
local_path = os.path.join(local_cache_dir, filename)
|
||||
if not os.path.isfile(local_path):
|
||||
continue
|
||||
|
||||
gcs_path = f"{self.GCS_CACHE_PREFIX}{filename}"
|
||||
|
||||
try:
|
||||
# Skip if already exists in GCS (same hash = same content)
|
||||
if self.storage.file_exists(gcs_path):
|
||||
logger.debug(f"Cache file already in GCS: {filename}")
|
||||
stats["skipped"] += 1
|
||||
continue
|
||||
|
||||
self.storage.upload_file(local_path, gcs_path)
|
||||
logger.info(f"Uploaded cache file: {filename}")
|
||||
stats["uploaded"] += 1
|
||||
except Exception as e:
|
||||
logger.warning(f"Error uploading cache file {filename}: {e}")
|
||||
stats["errors"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cache sync to GCS complete: "
|
||||
f"{stats['uploaded']} uploaded, "
|
||||
f"{stats['skipped']} skipped (already exist), "
|
||||
f"{stats['errors']} errors"
|
||||
)
|
||||
return stats
|
||||
@@ -0,0 +1,122 @@
|
||||
"""NLTK resource preloader for container startup.
|
||||
|
||||
Loads NLTK data at container startup to avoid slow downloads during request processing.
|
||||
Cloud Run's ephemeral filesystem means NLTK data must be re-downloaded on each cold start,
|
||||
which can take 30-100+ seconds for cmudict.
|
||||
|
||||
See docs/archive/2026-01-08-performance-investigation.md for background.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded resources
|
||||
_preloaded_resources: Dict[str, Any] = {}
|
||||
|
||||
|
||||
def preload_nltk_cmudict() -> None:
|
||||
"""Preload NLTK's CMU Pronouncing Dictionary at startup.
|
||||
|
||||
The cmudict is used by SyllablesMatchHandler for syllable counting.
|
||||
Without preloading, each SyllablesMatchHandler init downloads ~30MB,
|
||||
which took 50-100+ seconds in Cloud Run.
|
||||
"""
|
||||
global _preloaded_resources
|
||||
|
||||
if "cmudict" in _preloaded_resources:
|
||||
logger.info("NLTK cmudict already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK cmudict...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
# Ensure the data is downloaded
|
||||
try:
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Try to access it - will raise LookupError if not downloaded
|
||||
_ = cmudict.dict()
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK cmudict data...")
|
||||
nltk.download("cmudict", quiet=True)
|
||||
from nltk.corpus import cmudict
|
||||
|
||||
# Load into memory
|
||||
cmu_dict = cmudict.dict()
|
||||
_preloaded_resources["cmudict"] = cmu_dict
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK cmudict preloaded in {elapsed:.2f}s ({len(cmu_dict)} entries)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload NLTK cmudict: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_cmudict() -> Optional[Dict]:
|
||||
"""Get the preloaded CMU dictionary if available.
|
||||
|
||||
Returns:
|
||||
The preloaded cmudict dictionary, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_resources.get("cmudict")
|
||||
|
||||
|
||||
def is_cmudict_preloaded() -> bool:
|
||||
"""Check if cmudict has been preloaded."""
|
||||
return "cmudict" in _preloaded_resources
|
||||
|
||||
|
||||
def preload_nltk_punkt() -> None:
|
||||
"""Preload NLTK's punkt tokenizer (optional, used for sentence tokenization)."""
|
||||
global _preloaded_resources
|
||||
|
||||
if "punkt" in _preloaded_resources:
|
||||
logger.info("NLTK punkt already preloaded")
|
||||
return
|
||||
|
||||
logger.info("Preloading NLTK punkt tokenizer...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import nltk
|
||||
|
||||
try:
|
||||
from nltk.tokenize import word_tokenize
|
||||
|
||||
# Test it works
|
||||
_ = word_tokenize("test")
|
||||
except LookupError:
|
||||
logger.info("Downloading NLTK punkt data...")
|
||||
nltk.download("punkt", quiet=True)
|
||||
nltk.download("punkt_tab", quiet=True)
|
||||
|
||||
_preloaded_resources["punkt"] = True
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"NLTK punkt preloaded in {elapsed:.2f}s")
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to preload NLTK punkt (non-critical): {e}")
|
||||
|
||||
|
||||
def preload_all_nltk_resources() -> None:
|
||||
"""Preload all NLTK resources used by the application."""
|
||||
preload_nltk_cmudict()
|
||||
# punkt is optional and less critical
|
||||
try:
|
||||
preload_nltk_punkt()
|
||||
except Exception:
|
||||
pass # Non-critical
|
||||
|
||||
|
||||
def clear_preloaded_resources() -> None:
|
||||
"""Clear all preloaded resources. Useful for testing."""
|
||||
global _preloaded_resources
|
||||
_preloaded_resources.clear()
|
||||
@@ -0,0 +1,287 @@
|
||||
"""
|
||||
Packaging Service.
|
||||
|
||||
Provides CDG and TXT package generation functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- CDG (CD+G) file generation from LRC files
|
||||
- TXT lyric file generation from LRC files
|
||||
- ZIP packaging of CDG/MP3 and TXT/MP3 pairs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import zipfile
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PackagingService:
|
||||
"""
|
||||
Service for creating CDG and TXT karaoke packages.
|
||||
|
||||
CDG (CD+Graphics) is a format used by karaoke machines.
|
||||
TXT packages are used by software karaoke players.
|
||||
Both formats are packaged as ZIP files with an MP3 audio track.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
dry_run: bool = False,
|
||||
non_interactive: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the packaging service.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration for CDG generation
|
||||
dry_run: If True, log actions without performing them
|
||||
non_interactive: If True, skip interactive prompts
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.cdg_styles = cdg_styles
|
||||
self.dry_run = dry_run
|
||||
self.non_interactive = non_interactive
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
def create_cdg_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
audio_file: str,
|
||||
output_zip_path: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_mp3_path: Optional[str] = None,
|
||||
output_cdg_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Create a CDG package (ZIP containing CDG and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
audio_file: Path to the instrumental audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
artist: Artist name for metadata
|
||||
title: Song title for metadata
|
||||
output_mp3_path: Optional path for the extracted MP3 file
|
||||
output_cdg_path: Optional path for the extracted CDG file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, mp3_path, cdg_path)
|
||||
|
||||
Raises:
|
||||
ValueError: If CDG styles are not configured
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If CDG generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating CDG package for {artist} - {title}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(audio_file):
|
||||
raise FileNotFoundError(f"Audio file not found: {audio_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"CDG ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"CDG ZIP already exists: {output_zip_path}")
|
||||
|
||||
# Check if individual files exist (allows skipping generation)
|
||||
if output_mp3_path and output_cdg_path:
|
||||
if os.path.isfile(output_mp3_path) and os.path.isfile(output_cdg_path):
|
||||
self.logger.info("Found existing MP3 and CDG files, creating ZIP directly")
|
||||
if not self.dry_run:
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(output_mp3_path, os.path.basename(output_mp3_path)),
|
||||
(output_cdg_path, os.path.basename(output_cdg_path))]
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would generate CDG package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_mp3_path, output_cdg_path
|
||||
|
||||
# Generate CDG files
|
||||
if self.cdg_styles is None:
|
||||
raise ValueError(
|
||||
"CDG styles configuration is required for CDG generation"
|
||||
)
|
||||
|
||||
self.logger.info("Generating CDG and MP3 files")
|
||||
from lyrics_transcriber.output.cdg import CDGGenerator
|
||||
|
||||
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
|
||||
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
|
||||
|
||||
cdg_file, mp3_file, zip_file = generator.generate_cdg_from_lrc(
|
||||
lrc_file=lrc_file,
|
||||
audio_file=audio_file,
|
||||
title=title,
|
||||
artist=artist,
|
||||
cdg_styles=self.cdg_styles,
|
||||
)
|
||||
|
||||
# Rename ZIP to expected output path if different
|
||||
if os.path.isfile(zip_file) and zip_file != output_zip_path:
|
||||
os.rename(zip_file, output_zip_path)
|
||||
self.logger.info(f"Renamed CDG ZIP: {zip_file} -> {output_zip_path}")
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create CDG ZIP file: {output_zip_path}")
|
||||
|
||||
# Extract the ZIP to get individual files if paths provided
|
||||
extracted_mp3 = None
|
||||
extracted_cdg = None
|
||||
if output_mp3_path or output_cdg_path:
|
||||
self.logger.info(f"Extracting CDG ZIP file: {output_zip_path}")
|
||||
with zipfile.ZipFile(output_zip_path, "r") as zip_ref:
|
||||
zip_ref.extractall(output_dir)
|
||||
|
||||
# Find extracted files
|
||||
if output_mp3_path and os.path.isfile(output_mp3_path):
|
||||
extracted_mp3 = output_mp3_path
|
||||
self.logger.info(f"Extracted MP3: {extracted_mp3}")
|
||||
if output_cdg_path and os.path.isfile(output_cdg_path):
|
||||
extracted_cdg = output_cdg_path
|
||||
self.logger.info(f"Extracted CDG: {extracted_cdg}")
|
||||
|
||||
self.logger.info(f"CDG package created: {output_zip_path}")
|
||||
return output_zip_path, extracted_mp3, extracted_cdg
|
||||
|
||||
def create_txt_package(
|
||||
self,
|
||||
lrc_file: str,
|
||||
mp3_file: str,
|
||||
output_zip_path: str,
|
||||
output_txt_path: Optional[str] = None,
|
||||
) -> Tuple[str, Optional[str]]:
|
||||
"""
|
||||
Create a TXT package (ZIP containing TXT lyrics and MP3 files).
|
||||
|
||||
Args:
|
||||
lrc_file: Path to the LRC lyrics file
|
||||
mp3_file: Path to the MP3 audio file
|
||||
output_zip_path: Path for the output ZIP file
|
||||
output_txt_path: Optional path for the generated TXT file
|
||||
|
||||
Returns:
|
||||
Tuple of (zip_path, txt_path)
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If input files are missing
|
||||
Exception: If TXT generation fails
|
||||
"""
|
||||
self.logger.info(f"Creating TXT package from {lrc_file}")
|
||||
|
||||
# Validate inputs
|
||||
if not os.path.isfile(lrc_file):
|
||||
raise FileNotFoundError(f"LRC file not found: {lrc_file}")
|
||||
if not os.path.isfile(mp3_file):
|
||||
raise FileNotFoundError(f"MP3 file not found: {mp3_file}")
|
||||
|
||||
# Check if ZIP already exists
|
||||
if os.path.isfile(output_zip_path):
|
||||
if self.non_interactive:
|
||||
self.logger.info(
|
||||
f"TXT ZIP exists, will be overwritten: {output_zip_path}"
|
||||
)
|
||||
else:
|
||||
self.logger.info(f"TXT ZIP already exists: {output_zip_path}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would create TXT package: {output_zip_path}"
|
||||
)
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
# Generate TXT from LRC
|
||||
self.logger.info(f"Converting LRC to TXT format: {lrc_file}")
|
||||
from lyrics_converter import LyricsConverter
|
||||
|
||||
txt_converter = LyricsConverter(output_format="txt", filepath=lrc_file)
|
||||
converted_txt = txt_converter.convert_file()
|
||||
|
||||
# Write TXT file
|
||||
if output_txt_path is None:
|
||||
# Default to same name as ZIP but with .txt extension
|
||||
output_txt_path = output_zip_path.replace(".zip", ".txt")
|
||||
|
||||
with open(output_txt_path, "w") as txt_file:
|
||||
txt_file.write(converted_txt)
|
||||
self.logger.info(f"TXT file written: {output_txt_path}")
|
||||
|
||||
# Create ZIP containing MP3 and TXT
|
||||
self.logger.info(f"Creating TXT ZIP: {output_zip_path}")
|
||||
self._create_zip_from_files(
|
||||
output_zip_path,
|
||||
[(mp3_file, os.path.basename(mp3_file)),
|
||||
(output_txt_path, os.path.basename(output_txt_path))]
|
||||
)
|
||||
|
||||
if not os.path.isfile(output_zip_path):
|
||||
raise Exception(f"Failed to create TXT ZIP file: {output_zip_path}")
|
||||
|
||||
self.logger.info(f"TXT package created: {output_zip_path}")
|
||||
return output_zip_path, output_txt_path
|
||||
|
||||
def _create_zip_from_files(
|
||||
self,
|
||||
zip_path: str,
|
||||
files: list,
|
||||
) -> None:
|
||||
"""
|
||||
Create a ZIP file from a list of files.
|
||||
|
||||
Args:
|
||||
zip_path: Path for the output ZIP file
|
||||
files: List of (file_path, archive_name) tuples
|
||||
"""
|
||||
with zipfile.ZipFile(zip_path, "w") as zipf:
|
||||
for file_path, archive_name in files:
|
||||
if os.path.isfile(file_path):
|
||||
zipf.write(file_path, archive_name)
|
||||
self.logger.debug(f"Added to ZIP: {archive_name}")
|
||||
else:
|
||||
self.logger.warning(f"File not found for ZIP: {file_path}")
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_packaging_service: Optional[PackagingService] = None
|
||||
|
||||
|
||||
def get_packaging_service(
|
||||
cdg_styles: Optional[Dict[str, Any]] = None,
|
||||
**kwargs
|
||||
) -> PackagingService:
|
||||
"""
|
||||
Get a packaging service instance.
|
||||
|
||||
Args:
|
||||
cdg_styles: CDG style configuration
|
||||
**kwargs: Additional arguments passed to PackagingService
|
||||
|
||||
Returns:
|
||||
PackagingService instance
|
||||
"""
|
||||
global _packaging_service
|
||||
|
||||
# Create new instance if settings changed
|
||||
if _packaging_service is None or cdg_styles:
|
||||
_packaging_service = PackagingService(
|
||||
cdg_styles=cdg_styles,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _packaging_service
|
||||
@@ -0,0 +1,409 @@
|
||||
"""
|
||||
Push Notification Service for Web Push Notifications.
|
||||
|
||||
Handles sending push notifications to users' browsers/devices using the Web Push protocol.
|
||||
Uses pywebpush library for VAPID authentication and push message encryption.
|
||||
"""
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, List
|
||||
|
||||
from pywebpush import webpush, WebPushException
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import User, PushSubscription
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SubscriptionGoneError(Exception):
|
||||
"""Raised when a push subscription is no longer valid (410 Gone)."""
|
||||
pass
|
||||
|
||||
|
||||
class PushNotificationService:
|
||||
"""
|
||||
Service for sending Web Push notifications.
|
||||
|
||||
Requires VAPID keys to be configured in Secret Manager:
|
||||
- vapid-public-key: Base64-encoded public key
|
||||
- vapid-private-key: Base64-encoded private key
|
||||
"""
|
||||
|
||||
def __init__(self, db=None):
|
||||
"""
|
||||
Initialize the push notification service.
|
||||
|
||||
Args:
|
||||
db: Firestore client (optional, will use singleton if not provided)
|
||||
"""
|
||||
self.settings = get_settings()
|
||||
self._vapid_private_key: Optional[str] = None
|
||||
self._vapid_public_key: Optional[str] = None
|
||||
self._db = db
|
||||
|
||||
@property
|
||||
def db(self):
|
||||
"""Get Firestore client (lazy initialization)."""
|
||||
if self._db is None:
|
||||
from google.cloud import firestore
|
||||
self._db = firestore.Client()
|
||||
return self._db
|
||||
|
||||
@property
|
||||
def vapid_private_key(self) -> Optional[str]:
|
||||
"""Get VAPID private key from Secret Manager (cached)."""
|
||||
if self._vapid_private_key is None:
|
||||
self._vapid_private_key = self.settings.get_secret("vapid-private-key")
|
||||
return self._vapid_private_key
|
||||
|
||||
@property
|
||||
def vapid_public_key(self) -> Optional[str]:
|
||||
"""Get VAPID public key from Secret Manager (cached)."""
|
||||
if self._vapid_public_key is None:
|
||||
self._vapid_public_key = self.settings.get_secret("vapid-public-key")
|
||||
return self._vapid_public_key
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if push notifications are enabled and properly configured."""
|
||||
return (
|
||||
self.settings.enable_push_notifications and
|
||||
self.vapid_private_key is not None and
|
||||
self.vapid_public_key is not None
|
||||
)
|
||||
|
||||
def get_public_key(self) -> Optional[str]:
|
||||
"""Get the VAPID public key for client-side subscription."""
|
||||
if not self.settings.enable_push_notifications:
|
||||
return None
|
||||
return self.vapid_public_key
|
||||
|
||||
async def send_push(
|
||||
self,
|
||||
user_email: str,
|
||||
title: str,
|
||||
body: str,
|
||||
url: str = "/app/",
|
||||
tag: Optional[str] = None
|
||||
) -> int:
|
||||
"""
|
||||
Send a push notification to all of a user's subscribed devices.
|
||||
|
||||
Args:
|
||||
user_email: The user's email address
|
||||
title: Notification title
|
||||
body: Notification body text
|
||||
url: URL to open when notification is clicked
|
||||
tag: Optional tag for notification grouping (replaces notifications with same tag)
|
||||
|
||||
Returns:
|
||||
Number of notifications successfully sent
|
||||
"""
|
||||
if not self.is_enabled():
|
||||
logger.debug("Push notifications not enabled, skipping")
|
||||
return 0
|
||||
|
||||
# Get user from Firestore
|
||||
user_doc = self.db.collection("gen_users").document(user_email).get()
|
||||
if not user_doc.exists:
|
||||
logger.warning(f"User {user_email} not found for push notification")
|
||||
return 0
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
if not subscriptions:
|
||||
logger.debug(f"User {user_email} has no push subscriptions")
|
||||
return 0
|
||||
|
||||
# Build notification payload
|
||||
import json
|
||||
payload = json.dumps({
|
||||
"title": title,
|
||||
"body": body,
|
||||
"url": url,
|
||||
"tag": tag or "default",
|
||||
"icon": "/nomad-logo.png"
|
||||
})
|
||||
|
||||
# Send to each subscription
|
||||
success_count = 0
|
||||
invalid_subscriptions = []
|
||||
|
||||
for sub in subscriptions:
|
||||
try:
|
||||
webpush(
|
||||
subscription_info={
|
||||
"endpoint": sub["endpoint"],
|
||||
"keys": sub["keys"]
|
||||
},
|
||||
data=payload,
|
||||
vapid_private_key=self.vapid_private_key,
|
||||
vapid_claims={
|
||||
"sub": self.settings.vapid_subject
|
||||
}
|
||||
)
|
||||
success_count += 1
|
||||
logger.debug(f"Push sent to {sub.get('device_name', 'unknown device')}")
|
||||
|
||||
except WebPushException as e:
|
||||
logger.warning(f"Push failed: {e}")
|
||||
# Check if subscription is gone (410) or unauthorized (401/403)
|
||||
if e.response and e.response.status_code in (404, 410):
|
||||
logger.info(f"Subscription gone, will remove: {sub['endpoint'][:50]}...")
|
||||
invalid_subscriptions.append(sub["endpoint"])
|
||||
elif e.response and e.response.status_code in (401, 403):
|
||||
logger.warning(f"Push auth failed for {sub['endpoint'][:50]}...")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Unexpected error sending push: {e}")
|
||||
|
||||
# Remove invalid subscriptions
|
||||
if invalid_subscriptions:
|
||||
await self._remove_invalid_subscriptions(user_email, invalid_subscriptions)
|
||||
|
||||
logger.info(f"Push notifications sent to {success_count}/{len(subscriptions)} devices for {user_email}")
|
||||
return success_count
|
||||
|
||||
async def _remove_invalid_subscriptions(self, user_email: str, endpoints: List[str]) -> None:
|
||||
"""Remove invalid subscriptions from user's list."""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
if not user_doc.exists:
|
||||
return
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
current_subs = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Filter out invalid subscriptions
|
||||
valid_subs = [s for s in current_subs if s["endpoint"] not in endpoints]
|
||||
|
||||
if len(valid_subs) < len(current_subs):
|
||||
user_ref.update({"push_subscriptions": valid_subs})
|
||||
logger.info(f"Removed {len(current_subs) - len(valid_subs)} invalid subscriptions for {user_email}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to remove invalid subscriptions: {e}")
|
||||
|
||||
async def send_blocking_notification(
|
||||
self,
|
||||
job: dict,
|
||||
action_type: str # "lyrics" or "instrumental"
|
||||
) -> int:
|
||||
"""
|
||||
Send notification when a job enters a blocking state requiring user action.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
action_type: Type of action needed ("lyrics" or "instrumental")
|
||||
|
||||
Returns:
|
||||
Number of notifications sent
|
||||
"""
|
||||
user_email = job.get("user_email")
|
||||
if not user_email:
|
||||
return 0
|
||||
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
|
||||
if action_type == "lyrics":
|
||||
notif_title = "Review Lyrics"
|
||||
notif_body = f'"{title}" by {artist} needs lyrics review'
|
||||
url = f"/review/{job_id}"
|
||||
tag = f"lyrics-{job_id}"
|
||||
else: # instrumental
|
||||
notif_title = "Select Instrumental"
|
||||
notif_body = f'"{title}" by {artist} needs instrumental selection'
|
||||
url = f"/instrumental/{job_id}"
|
||||
tag = f"instrumental-{job_id}"
|
||||
|
||||
return await self.send_push(
|
||||
user_email=user_email,
|
||||
title=notif_title,
|
||||
body=notif_body,
|
||||
url=url,
|
||||
tag=tag
|
||||
)
|
||||
|
||||
async def send_completion_notification(self, job: dict) -> int:
|
||||
"""
|
||||
Send notification when a job completes successfully.
|
||||
|
||||
Args:
|
||||
job: Job dictionary with job_id, user_email, artist, title
|
||||
|
||||
Returns:
|
||||
Number of notifications sent
|
||||
"""
|
||||
user_email = job.get("user_email")
|
||||
if not user_email:
|
||||
return 0
|
||||
|
||||
job_id = job.get("job_id", "unknown")
|
||||
artist = job.get("artist", "Unknown Artist")
|
||||
title = job.get("title", "Unknown Title")
|
||||
|
||||
return await self.send_push(
|
||||
user_email=user_email,
|
||||
title="Video Ready!",
|
||||
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
|
||||
url=f"/app/?job={job_id}",
|
||||
tag=f"complete-{job_id}"
|
||||
)
|
||||
|
||||
async def add_subscription(
|
||||
self,
|
||||
user_email: str,
|
||||
endpoint: str,
|
||||
keys: dict,
|
||||
device_name: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Add a push subscription for a user.
|
||||
|
||||
Enforces max subscriptions per user - oldest removed if limit exceeded.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
endpoint: Push service endpoint URL
|
||||
keys: Encryption keys (p256dh, auth)
|
||||
device_name: Optional device identifier
|
||||
|
||||
Returns:
|
||||
True if subscription was added successfully
|
||||
"""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
|
||||
if not user_doc.exists:
|
||||
logger.warning(f"User {user_email} not found")
|
||||
return False
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Check if this endpoint already exists (update it)
|
||||
existing_idx = None
|
||||
for i, sub in enumerate(subscriptions):
|
||||
if sub["endpoint"] == endpoint:
|
||||
existing_idx = i
|
||||
break
|
||||
|
||||
new_sub = {
|
||||
"endpoint": endpoint,
|
||||
"keys": keys,
|
||||
"device_name": device_name,
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"last_used_at": None
|
||||
}
|
||||
|
||||
if existing_idx is not None:
|
||||
# Update existing subscription
|
||||
subscriptions[existing_idx] = new_sub
|
||||
logger.info(f"Updated existing push subscription for {user_email}")
|
||||
else:
|
||||
# Add new subscription
|
||||
subscriptions.append(new_sub)
|
||||
logger.info(f"Added new push subscription for {user_email}")
|
||||
|
||||
# Enforce max subscriptions (remove oldest)
|
||||
max_subs = self.settings.max_push_subscriptions_per_user
|
||||
if len(subscriptions) > max_subs:
|
||||
# Sort by created_at and keep only the newest
|
||||
subscriptions.sort(key=lambda s: s.get("created_at", ""), reverse=True)
|
||||
removed = subscriptions[max_subs:]
|
||||
subscriptions = subscriptions[:max_subs]
|
||||
logger.info(f"Removed {len(removed)} old subscriptions for {user_email} (max {max_subs})")
|
||||
|
||||
user_ref.update({"push_subscriptions": subscriptions})
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to add subscription for {user_email}: {e}")
|
||||
return False
|
||||
|
||||
async def remove_subscription(self, user_email: str, endpoint: str) -> bool:
|
||||
"""
|
||||
Remove a push subscription for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
endpoint: Push service endpoint URL to remove
|
||||
|
||||
Returns:
|
||||
True if subscription was removed
|
||||
"""
|
||||
try:
|
||||
user_ref = self.db.collection("gen_users").document(user_email)
|
||||
user_doc = user_ref.get()
|
||||
|
||||
if not user_doc.exists:
|
||||
return False
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Filter out the subscription
|
||||
new_subs = [s for s in subscriptions if s["endpoint"] != endpoint]
|
||||
|
||||
if len(new_subs) < len(subscriptions):
|
||||
user_ref.update({"push_subscriptions": new_subs})
|
||||
logger.info(f"Removed push subscription for {user_email}")
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to remove subscription for {user_email}: {e}")
|
||||
return False
|
||||
|
||||
async def list_subscriptions(self, user_email: str) -> List[dict]:
|
||||
"""
|
||||
List all push subscriptions for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
|
||||
Returns:
|
||||
List of subscription info (endpoint, device_name, created_at, last_used_at)
|
||||
"""
|
||||
try:
|
||||
user_doc = self.db.collection("gen_users").document(user_email).get()
|
||||
|
||||
if not user_doc.exists:
|
||||
return []
|
||||
|
||||
user_data = user_doc.to_dict()
|
||||
subscriptions = user_data.get("push_subscriptions", [])
|
||||
|
||||
# Return safe subset of subscription data
|
||||
return [
|
||||
{
|
||||
"endpoint": s["endpoint"],
|
||||
"device_name": s.get("device_name"),
|
||||
"created_at": s.get("created_at"),
|
||||
"last_used_at": s.get("last_used_at")
|
||||
}
|
||||
for s in subscriptions
|
||||
]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list subscriptions for {user_email}: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_push_service: Optional[PushNotificationService] = None
|
||||
|
||||
|
||||
def get_push_notification_service(db=None) -> PushNotificationService:
|
||||
"""Get the push notification service singleton."""
|
||||
global _push_service
|
||||
if _push_service is None:
|
||||
_push_service = PushNotificationService(db)
|
||||
return _push_service
|
||||
@@ -0,0 +1,641 @@
|
||||
"""
|
||||
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
|
||||
|
||||
Uses Firestore for distributed rate limit tracking with date-based document IDs
|
||||
that automatically reset at UTC midnight.
|
||||
"""
|
||||
import hashlib
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Tuple, Optional, List, Dict, Any
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Firestore collection names
|
||||
RATE_LIMITS_COLLECTION = "rate_limits"
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
OVERRIDES_COLLECTION = "overrides"
|
||||
|
||||
|
||||
def _get_today_date_str() -> str:
|
||||
"""Get today's date in UTC as YYYY-MM-DD string."""
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
|
||||
|
||||
def _seconds_until_midnight_utc() -> int:
|
||||
"""Calculate seconds until UTC midnight (when rate limits reset)."""
|
||||
now = datetime.now(timezone.utc)
|
||||
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
# Add 1 day to get next midnight
|
||||
from datetime import timedelta
|
||||
next_midnight = midnight + timedelta(days=1)
|
||||
return int((next_midnight - now).total_seconds())
|
||||
|
||||
|
||||
def _hash_ip(ip_address: str) -> str:
|
||||
"""Hash an IP address for storage (privacy-preserving)."""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
|
||||
|
||||
|
||||
class RateLimitService:
|
||||
"""
|
||||
Service for managing rate limits across the platform.
|
||||
|
||||
Rate limits use Firestore for distributed tracking with automatic
|
||||
reset at UTC midnight via date-based document IDs.
|
||||
"""
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize rate limit service with Firestore client."""
|
||||
if db is None:
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
else:
|
||||
self.db = db
|
||||
self._blocklist_cache: Optional[Dict[str, Any]] = None
|
||||
self._blocklist_cache_time: Optional[datetime] = None
|
||||
self._cache_ttl_seconds = 300 # 5 minute cache
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Job Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if a user can create a new job.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
is_admin: Whether the user is an admin (admins bypass limits)
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
- allowed: True if user can create job
|
||||
- remaining: Number of jobs remaining today
|
||||
- message: Human-readable status message
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
if is_admin:
|
||||
return True, -1, "Admin users bypass rate limits"
|
||||
|
||||
limit = settings.rate_limit_jobs_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No job limit configured"
|
||||
|
||||
# Check for user override
|
||||
override = self.get_user_override(user_email)
|
||||
if override and override.get("bypass_job_limit"):
|
||||
return True, -1, "User has rate limit bypass"
|
||||
if override and override.get("custom_daily_job_limit"):
|
||||
limit = override["custom_daily_job_limit"]
|
||||
|
||||
# Get current count
|
||||
current_count = self.get_user_job_count_today(user_email)
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} jobs remaining today"
|
||||
|
||||
def get_user_job_count_today(self, user_email: str) -> int:
|
||||
"""Get the number of jobs created by a user today."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_job_creation(self, user_email: str, job_id: str) -> None:
|
||||
"""
|
||||
Record a job creation for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"user_jobs_{user_email}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
job_ids = data.get("job_ids", [])
|
||||
job_ids.append(job_id)
|
||||
else:
|
||||
count = 1
|
||||
job_ids = [job_id]
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"user_email": user_email,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"job_ids": job_ids,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# YouTube Upload Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if the system can perform a YouTube upload.
|
||||
|
||||
YouTube uploads are limited system-wide due to API quota constraints.
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_youtube_uploads_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No YouTube upload limit configured"
|
||||
|
||||
current_count = self.get_youtube_uploads_today()
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} YouTube uploads remaining today"
|
||||
|
||||
def get_youtube_uploads_today(self) -> int:
|
||||
"""Get the number of YouTube uploads performed today (system-wide)."""
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return 0
|
||||
|
||||
return doc.to_dict().get("count", 0)
|
||||
|
||||
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
|
||||
"""
|
||||
Record a YouTube upload for rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"youtube_uploads_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
uploads = data.get("uploads", [])
|
||||
else:
|
||||
count = 1
|
||||
uploads = []
|
||||
|
||||
uploads.append({
|
||||
"job_id": job_id,
|
||||
"user_email": user_email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"uploads": uploads,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded YouTube upload for job {job_id}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Beta Enrollment IP Rate Limiting
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Check if an IP address can enroll in the beta program.
|
||||
|
||||
Args:
|
||||
ip_address: Client IP address
|
||||
|
||||
Returns:
|
||||
Tuple of (allowed, remaining, message)
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return True, -1, "Rate limiting disabled"
|
||||
|
||||
limit = settings.rate_limit_beta_ip_per_day
|
||||
if limit == 0:
|
||||
return True, -1, "No beta IP limit configured"
|
||||
|
||||
# Check today's enrollment count for this IP
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
|
||||
current_count = 0
|
||||
if doc.exists:
|
||||
current_count = doc.to_dict().get("count", 0)
|
||||
|
||||
remaining = max(0, limit - current_count)
|
||||
|
||||
if current_count >= limit:
|
||||
seconds = _seconds_until_midnight_utc()
|
||||
logger.warning(
|
||||
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
|
||||
)
|
||||
return (
|
||||
False,
|
||||
0,
|
||||
f"Too many beta enrollments from this network today. Please try again tomorrow."
|
||||
)
|
||||
|
||||
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
|
||||
|
||||
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
|
||||
"""
|
||||
Record a beta enrollment for IP rate limiting.
|
||||
|
||||
Uses Firestore transactions for atomic increment.
|
||||
"""
|
||||
if not settings.enable_rate_limiting:
|
||||
return
|
||||
|
||||
ip_hash = _hash_ip(ip_address)
|
||||
date_str = _get_today_date_str()
|
||||
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
|
||||
f"beta_ip_{ip_hash}_{date_str}"
|
||||
)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
count = data.get("count", 0) + 1
|
||||
enrollments = data.get("enrollments", [])
|
||||
else:
|
||||
count = 1
|
||||
enrollments = []
|
||||
|
||||
enrollments.append({
|
||||
"email": email,
|
||||
"timestamp": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction.set(doc_ref, {
|
||||
"ip_hash": ip_hash,
|
||||
"date": date_str,
|
||||
"count": count,
|
||||
"enrollments": enrollments,
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# User Overrides (Whitelist)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get rate limit override settings for a user.
|
||||
|
||||
Returns:
|
||||
Override settings dict or None if no override exists
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
|
||||
def set_user_override(
|
||||
self,
|
||||
user_email: str,
|
||||
bypass_job_limit: bool = False,
|
||||
custom_daily_job_limit: Optional[int] = None,
|
||||
reason: str = "",
|
||||
admin_email: str = ""
|
||||
) -> None:
|
||||
"""
|
||||
Set rate limit override for a user.
|
||||
|
||||
Args:
|
||||
user_email: User to override
|
||||
bypass_job_limit: If True, user bypasses all job limits
|
||||
custom_daily_job_limit: Custom limit (None = use default)
|
||||
reason: Reason for override
|
||||
admin_email: Admin who set the override
|
||||
"""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc_ref.set({
|
||||
"email": user_email.lower(),
|
||||
"bypass_job_limit": bypass_job_limit,
|
||||
"custom_daily_job_limit": custom_daily_job_limit,
|
||||
"reason": reason,
|
||||
"created_by": admin_email,
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
})
|
||||
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
|
||||
|
||||
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
|
||||
"""Remove rate limit override for a user."""
|
||||
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return False
|
||||
|
||||
doc_ref.delete()
|
||||
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def list_user_overrides(self) -> List[Dict[str, Any]]:
|
||||
"""List all user rate limit overrides."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get all user rate limit overrides as a dict keyed by email."""
|
||||
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
|
||||
return {doc.id: doc.to_dict() for doc in docs}
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Load blocklist from Firestore with caching.
|
||||
|
||||
Returns:
|
||||
Dict with keys: disposable_domains, blocked_emails, blocked_ips
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache
|
||||
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
|
||||
cache_age = (now - self._blocklist_cache_time).total_seconds()
|
||||
if cache_age < self._cache_ttl_seconds:
|
||||
return self._blocklist_cache
|
||||
|
||||
# Load from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
# Initialize with empty blocklist
|
||||
self._blocklist_cache = {
|
||||
"disposable_domains": [],
|
||||
"blocked_emails": [],
|
||||
"blocked_ips": [],
|
||||
}
|
||||
else:
|
||||
self._blocklist_cache = doc.to_dict()
|
||||
|
||||
self._blocklist_cache_time = now
|
||||
return self._blocklist_cache
|
||||
|
||||
def is_disposable_domain(self, domain: str) -> bool:
|
||||
"""Check if a domain is in the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist()
|
||||
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
|
||||
|
||||
def is_blocked_email(self, email: str) -> bool:
|
||||
"""Check if an email is explicitly blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
|
||||
|
||||
def is_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Check if an IP address is blocked."""
|
||||
blocklist = self._load_blocklist()
|
||||
return ip_address in blocklist.get("blocked_ips", [])
|
||||
|
||||
def get_blocklist(self) -> Dict[str, Any]:
|
||||
"""Get all blocklist data."""
|
||||
return self._load_blocklist(force_refresh=True)
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
domains.add(domain.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None # Invalidate cache
|
||||
logger.info(f"Added disposable domain {domain} by {admin_email}")
|
||||
|
||||
def remove_disposable_domain(self, domain: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
domains = set(blocklist.get("disposable_domains", []))
|
||||
|
||||
if domain.lower() not in [d.lower() for d in domains]:
|
||||
return False
|
||||
|
||||
domains = {d for d in domains if d.lower() != domain.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"disposable_domains": list(domains),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed disposable domain {domain}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> None:
|
||||
"""Add an email to the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
emails.add(email.lower())
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked email {email} by {admin_email}")
|
||||
|
||||
def remove_blocked_email(self, email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
emails = set(blocklist.get("blocked_emails", []))
|
||||
|
||||
if email.lower() not in [e.lower() for e in emails]:
|
||||
return False
|
||||
|
||||
emails = {e for e in emails if e.lower() != email.lower()}
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_emails": list(emails),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked email {email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
ips.add(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
"updated_by": admin_email,
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
blocklist = self._load_blocklist(force_refresh=True)
|
||||
ips = set(blocklist.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return False
|
||||
|
||||
ips.discard(ip_address)
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
|
||||
doc_ref.set({
|
||||
**blocklist,
|
||||
"blocked_ips": list(ips),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}, merge=True)
|
||||
|
||||
self._blocklist_cache = None
|
||||
logger.info(f"Removed blocked IP {ip_address}")
|
||||
return True
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Stats
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def get_rate_limit_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns:
|
||||
Dict with current usage stats
|
||||
"""
|
||||
date_str = _get_today_date_str()
|
||||
|
||||
# Get YouTube upload count
|
||||
youtube_count = self.get_youtube_uploads_today()
|
||||
|
||||
# Count unique users with jobs today
|
||||
users_with_jobs = set()
|
||||
total_jobs_today = 0
|
||||
|
||||
# Query all user job documents for today
|
||||
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
|
||||
"date", "==", date_str
|
||||
).stream()
|
||||
|
||||
for doc in docs:
|
||||
data = doc.to_dict()
|
||||
if data.get("user_email"):
|
||||
users_with_jobs.add(data["user_email"])
|
||||
total_jobs_today += data.get("count", 0)
|
||||
|
||||
return {
|
||||
"date": date_str,
|
||||
"youtube_uploads_today": youtube_count,
|
||||
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
|
||||
"total_jobs_today": total_jobs_today,
|
||||
"users_with_jobs_today": len(users_with_jobs),
|
||||
"job_limit_per_user": settings.rate_limit_jobs_per_day,
|
||||
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
|
||||
"rate_limiting_enabled": settings.enable_rate_limiting,
|
||||
"seconds_until_reset": _seconds_until_midnight_utc(),
|
||||
}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_rate_limit_service: Optional[RateLimitService] = None
|
||||
|
||||
|
||||
def get_rate_limit_service() -> RateLimitService:
|
||||
"""Get the singleton RateLimitService instance."""
|
||||
global _rate_limit_service
|
||||
if _rate_limit_service is None:
|
||||
_rate_limit_service = RateLimitService()
|
||||
return _rate_limit_service
|
||||
@@ -0,0 +1,65 @@
|
||||
"""SpaCy model preloader for container startup.
|
||||
|
||||
Loads SpaCy models at container startup to avoid slow loading during request processing.
|
||||
Cloud Run filesystem I/O can cause 60+ second delays when loading SpaCy models lazily.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Singleton storage for preloaded models
|
||||
_preloaded_models: dict = {}
|
||||
|
||||
|
||||
def preload_spacy_model(model_name: str = "en_core_web_sm") -> None:
|
||||
"""Preload a SpaCy model at startup.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model to load (default: en_core_web_sm)
|
||||
"""
|
||||
global _preloaded_models
|
||||
|
||||
if model_name in _preloaded_models:
|
||||
logger.info(f"SpaCy model '{model_name}' already preloaded")
|
||||
return
|
||||
|
||||
logger.info(f"Preloading SpaCy model '{model_name}'...")
|
||||
start_time = time.time()
|
||||
|
||||
try:
|
||||
import spacy
|
||||
|
||||
nlp = spacy.load(model_name)
|
||||
_preloaded_models[model_name] = nlp
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
logger.info(f"SpaCy model '{model_name}' preloaded in {elapsed:.2f}s")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to preload SpaCy model '{model_name}': {e}")
|
||||
raise
|
||||
|
||||
|
||||
def get_preloaded_model(model_name: str = "en_core_web_sm") -> Optional[object]:
|
||||
"""Get a preloaded SpaCy model if available.
|
||||
|
||||
Args:
|
||||
model_name: The SpaCy model name
|
||||
|
||||
Returns:
|
||||
The preloaded SpaCy Language object, or None if not preloaded
|
||||
"""
|
||||
return _preloaded_models.get(model_name)
|
||||
|
||||
|
||||
def is_model_preloaded(model_name: str = "en_core_web_sm") -> bool:
|
||||
"""Check if a SpaCy model has been preloaded."""
|
||||
return model_name in _preloaded_models
|
||||
|
||||
|
||||
def clear_preloaded_models() -> None:
|
||||
"""Clear all preloaded models. Useful for testing."""
|
||||
global _preloaded_models
|
||||
_preloaded_models.clear()
|
||||
@@ -0,0 +1,397 @@
|
||||
"""
|
||||
Stripe service for payment processing.
|
||||
|
||||
Handles:
|
||||
- Creating checkout sessions for credit purchases
|
||||
- Processing webhook events
|
||||
- Managing customer records
|
||||
"""
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
import stripe
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Credit packages available for purchase
|
||||
CREDIT_PACKAGES = {
|
||||
"1_credit": {
|
||||
"credits": 1,
|
||||
"price_cents": 500, # $5.00
|
||||
"name": "1 Karaoke Credit",
|
||||
"description": "Create 1 professional karaoke video",
|
||||
},
|
||||
"3_credits": {
|
||||
"credits": 3,
|
||||
"price_cents": 1200, # $12.00 (20% discount)
|
||||
"name": "3 Karaoke Credits",
|
||||
"description": "Create 3 professional karaoke videos (Save 20%)",
|
||||
},
|
||||
"5_credits": {
|
||||
"credits": 5,
|
||||
"price_cents": 1750, # $17.50 (30% discount)
|
||||
"name": "5 Karaoke Credits",
|
||||
"description": "Create 5 professional karaoke videos (Save 30%)",
|
||||
},
|
||||
"10_credits": {
|
||||
"credits": 10,
|
||||
"price_cents": 3000, # $30.00 (40% discount)
|
||||
"name": "10 Karaoke Credits",
|
||||
"description": "Create 10 professional karaoke videos (Save 40%)",
|
||||
},
|
||||
}
|
||||
|
||||
# Made-for-you service package (not a credit package - creates a job directly)
|
||||
MADE_FOR_YOU_PACKAGE = {
|
||||
"price_cents": 1500, # $15.00
|
||||
"name": "Made For You Karaoke Video",
|
||||
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
|
||||
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
|
||||
}
|
||||
|
||||
|
||||
class StripeService:
|
||||
"""Service for Stripe payment processing."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Stripe with API key."""
|
||||
self.settings = get_settings()
|
||||
self.secret_key = os.getenv("STRIPE_SECRET_KEY")
|
||||
self.webhook_secret = os.getenv("STRIPE_WEBHOOK_SECRET")
|
||||
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
|
||||
# After consolidation, buy URL is the same as frontend URL
|
||||
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
|
||||
# Payment method configuration ID from Stripe Dashboard
|
||||
# Enables Google Pay, Apple Pay, Link, and other wallet methods
|
||||
# Get this from: Dashboard > Settings > Payment methods > [Your Config] > Copy ID
|
||||
self.payment_method_config = os.getenv("STRIPE_PAYMENT_METHOD_CONFIG")
|
||||
|
||||
if self.secret_key:
|
||||
stripe.api_key = self.secret_key
|
||||
logger.info("Stripe initialized with API key")
|
||||
else:
|
||||
logger.warning("STRIPE_SECRET_KEY not set - payments disabled")
|
||||
|
||||
if self.payment_method_config:
|
||||
logger.info(f"Using payment method config: {self.payment_method_config}")
|
||||
else:
|
||||
logger.info("No STRIPE_PAYMENT_METHOD_CONFIG set - using Stripe defaults")
|
||||
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Stripe is properly configured."""
|
||||
return bool(self.secret_key)
|
||||
|
||||
def get_credit_packages(self) -> Dict[str, Dict[str, Any]]:
|
||||
"""Get available credit packages."""
|
||||
return CREDIT_PACKAGES
|
||||
|
||||
def create_checkout_session(
|
||||
self,
|
||||
package_id: str,
|
||||
user_email: str,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for purchasing credits.
|
||||
|
||||
Args:
|
||||
package_id: ID of the credit package to purchase
|
||||
user_email: Email of the purchasing user
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
package = CREDIT_PACKAGES.get(package_id)
|
||||
if not package:
|
||||
return False, None, f"Invalid package: {package_id}"
|
||||
|
||||
try:
|
||||
# Default URLs
|
||||
if not success_url:
|
||||
success_url = f"{self.frontend_url}/payment/success?session_id={{CHECKOUT_SESSION_ID}}"
|
||||
if not cancel_url:
|
||||
cancel_url = f"{self.buy_url}?cancelled=true"
|
||||
|
||||
# Build checkout session params
|
||||
session_params = {
|
||||
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
|
||||
'line_items': [{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': package['name'],
|
||||
'description': package['description'],
|
||||
'images': ['https://gen.nomadkaraoke.com/nomad-logo.png'],
|
||||
},
|
||||
'unit_amount': package['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
'mode': 'payment',
|
||||
'success_url': success_url,
|
||||
'cancel_url': cancel_url,
|
||||
'customer_email': user_email,
|
||||
'metadata': {
|
||||
'package_id': package_id,
|
||||
'credits': str(package['credits']),
|
||||
'user_email': user_email,
|
||||
},
|
||||
# Allow promotion codes
|
||||
'allow_promotion_codes': True,
|
||||
}
|
||||
|
||||
# Add payment method configuration if set (enables Google Pay, Link, etc.)
|
||||
if self.payment_method_config:
|
||||
session_params['payment_method_configuration'] = self.payment_method_config
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(**session_params)
|
||||
|
||||
logger.info(f"Created checkout session {session.id} for {user_email}, package {package_id}")
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def create_made_for_you_checkout_session(
|
||||
self,
|
||||
customer_email: str,
|
||||
artist: str,
|
||||
title: str,
|
||||
source_type: str = "search",
|
||||
youtube_url: Optional[str] = None,
|
||||
notes: Optional[str] = None,
|
||||
success_url: Optional[str] = None,
|
||||
cancel_url: Optional[str] = None,
|
||||
) -> Tuple[bool, Optional[str], str]:
|
||||
"""
|
||||
Create a Stripe Checkout session for a made-for-you order.
|
||||
|
||||
This is for the full-service karaoke video creation where Nomad Karaoke
|
||||
handles all the work (lyrics review, instrumental selection, etc.).
|
||||
|
||||
Args:
|
||||
customer_email: Customer's email for delivery
|
||||
artist: Song artist
|
||||
title: Song title
|
||||
source_type: Audio source type (search, youtube, upload)
|
||||
youtube_url: YouTube URL if source_type is youtube
|
||||
notes: Any special requests from customer
|
||||
success_url: URL to redirect to on success (optional)
|
||||
cancel_url: URL to redirect to on cancel (optional)
|
||||
|
||||
Returns:
|
||||
(success, checkout_url, message)
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return False, None, "Payment processing is not configured"
|
||||
|
||||
try:
|
||||
# Default URLs - redirect to gen app success page
|
||||
if not success_url:
|
||||
success_url = f"{self.frontend_url}/order/success?session_id={{CHECKOUT_SESSION_ID}}"
|
||||
if not cancel_url:
|
||||
cancel_url = "https://nomadkaraoke.com/#do-it-for-me"
|
||||
|
||||
# Build metadata for job creation after payment
|
||||
metadata = {
|
||||
'order_type': 'made_for_you',
|
||||
'customer_email': customer_email,
|
||||
'artist': artist,
|
||||
'title': title,
|
||||
'source_type': source_type,
|
||||
}
|
||||
if youtube_url:
|
||||
metadata['youtube_url'] = youtube_url
|
||||
if notes:
|
||||
# Truncate notes to fit Stripe's 500 char limit per metadata value
|
||||
metadata['notes'] = notes[:500] if len(notes) > 500 else notes
|
||||
|
||||
# Build checkout session params
|
||||
session_params = {
|
||||
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
|
||||
'line_items': [{
|
||||
'price_data': {
|
||||
'currency': 'usd',
|
||||
'product_data': {
|
||||
'name': f"Karaoke Video: {artist} - {title}",
|
||||
'description': MADE_FOR_YOU_PACKAGE['description'],
|
||||
'images': MADE_FOR_YOU_PACKAGE['images'],
|
||||
},
|
||||
'unit_amount': MADE_FOR_YOU_PACKAGE['price_cents'],
|
||||
},
|
||||
'quantity': 1,
|
||||
}],
|
||||
'mode': 'payment',
|
||||
'success_url': success_url,
|
||||
'cancel_url': cancel_url,
|
||||
'customer_email': customer_email,
|
||||
'metadata': metadata,
|
||||
'allow_promotion_codes': True,
|
||||
}
|
||||
|
||||
# Add payment method configuration if set (enables Google Pay, Link, etc.)
|
||||
if self.payment_method_config:
|
||||
session_params['payment_method_configuration'] = self.payment_method_config
|
||||
|
||||
# Create checkout session
|
||||
session = stripe.checkout.Session.create(**session_params)
|
||||
|
||||
logger.info(
|
||||
f"Created made-for-you checkout session {session.id} for {customer_email}, "
|
||||
f"song: {artist} - {title}"
|
||||
)
|
||||
return True, session.url, "Checkout session created"
|
||||
|
||||
except stripe.error.StripeError as e:
|
||||
logger.error(f"Stripe error creating made-for-you checkout session: {e}")
|
||||
return False, None, f"Payment error: {str(e)}"
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating made-for-you checkout session: {e}")
|
||||
return False, None, "Failed to create checkout session"
|
||||
|
||||
def verify_webhook_signature(self, payload: bytes, signature: str) -> Tuple[bool, Optional[Dict], str]:
|
||||
"""
|
||||
Verify a Stripe webhook signature.
|
||||
|
||||
Args:
|
||||
payload: Raw request body
|
||||
signature: Stripe-Signature header value
|
||||
|
||||
Returns:
|
||||
(valid, event_data, message)
|
||||
"""
|
||||
if not self.webhook_secret:
|
||||
logger.error("STRIPE_WEBHOOK_SECRET not configured")
|
||||
return False, None, "Webhook secret not configured"
|
||||
|
||||
try:
|
||||
event = stripe.Webhook.construct_event(
|
||||
payload, signature, self.webhook_secret
|
||||
)
|
||||
return True, event, "Webhook verified"
|
||||
except stripe.error.SignatureVerificationError as e:
|
||||
logger.error(f"Invalid webhook signature: {e}")
|
||||
return False, None, "Invalid signature"
|
||||
except Exception as e:
|
||||
logger.error(f"Error verifying webhook: {e}")
|
||||
return False, None, str(e)
|
||||
|
||||
def handle_checkout_completed(self, session: Dict) -> Tuple[bool, Optional[str], int, str]:
|
||||
"""
|
||||
Handle a completed checkout session.
|
||||
|
||||
Args:
|
||||
session: Stripe checkout session object
|
||||
|
||||
Returns:
|
||||
(success, user_email, credits_to_add, message)
|
||||
"""
|
||||
try:
|
||||
metadata = session.get('metadata', {})
|
||||
user_email = metadata.get('user_email') or session.get('customer_email')
|
||||
try:
|
||||
credits = int(metadata.get('credits', 0))
|
||||
except (ValueError, TypeError):
|
||||
logger.error(f"Invalid credits metadata in session {session.get('id')}: {metadata.get('credits')}")
|
||||
return False, user_email, 0, "Invalid credit amount in session metadata"
|
||||
package_id = metadata.get('package_id')
|
||||
|
||||
if not user_email:
|
||||
logger.error(f"No user email in checkout session {session.get('id')}")
|
||||
return False, None, 0, "No user email found"
|
||||
|
||||
if credits <= 0:
|
||||
logger.error(f"Invalid credits in checkout session {session.get('id')}")
|
||||
return False, user_email, 0, "Invalid credit amount"
|
||||
|
||||
logger.info(
|
||||
f"Checkout completed: {user_email} purchased {credits} credits "
|
||||
f"(package: {package_id}, session: {session.get('id')})"
|
||||
)
|
||||
|
||||
return True, user_email, credits, f"Successfully purchased {credits} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error handling checkout completed: {e}")
|
||||
return False, None, 0, str(e)
|
||||
|
||||
def get_session(self, session_id: str) -> Optional[Dict]:
|
||||
"""
|
||||
Get a checkout session by ID.
|
||||
|
||||
Args:
|
||||
session_id: Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
Session data or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
session = stripe.checkout.Session.retrieve(session_id)
|
||||
return dict(session)
|
||||
except Exception as e:
|
||||
logger.error(f"Error retrieving session {session_id}: {e}")
|
||||
return None
|
||||
|
||||
def create_customer(self, email: str, name: Optional[str] = None) -> Optional[str]:
|
||||
"""
|
||||
Create or get a Stripe customer for a user.
|
||||
|
||||
Args:
|
||||
email: User's email
|
||||
name: User's display name (optional)
|
||||
|
||||
Returns:
|
||||
Stripe customer ID or None
|
||||
"""
|
||||
if not self.is_configured():
|
||||
return None
|
||||
|
||||
try:
|
||||
# Check if customer already exists
|
||||
customers = stripe.Customer.list(email=email, limit=1)
|
||||
if customers.data:
|
||||
return customers.data[0].id
|
||||
|
||||
# Create new customer
|
||||
customer = stripe.Customer.create(
|
||||
email=email,
|
||||
name=name,
|
||||
metadata={'source': 'nomad_karaoke'},
|
||||
)
|
||||
logger.info(f"Created Stripe customer {customer.id} for {email}")
|
||||
return customer.id
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating Stripe customer: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global instance
|
||||
_stripe_service = None
|
||||
|
||||
|
||||
def get_stripe_service() -> StripeService:
|
||||
"""Get the global Stripe service instance."""
|
||||
global _stripe_service
|
||||
if _stripe_service is None:
|
||||
_stripe_service = StripeService()
|
||||
return _stripe_service
|
||||
@@ -91,7 +91,15 @@ class StructuredFormatter(logging.Formatter):
|
||||
|
||||
# Add custom fields from 'extra' dict
|
||||
# Common fields we want to extract from log records
|
||||
custom_fields = ["job_id", "worker", "operation", "duration", "status", "error"]
|
||||
custom_fields = [
|
||||
# Job-related fields
|
||||
"job_id", "worker", "operation", "duration", "status", "error",
|
||||
# Audit logging fields (from middleware and auth)
|
||||
"request_id", "user_email", "client_ip", "latency_ms",
|
||||
"audit_type", "method", "path", "status_code", "query_string",
|
||||
"user_agent", "user_type", "is_admin", "remaining_uses",
|
||||
"auth_message", "token_provided", "token_length", "auth_header_present",
|
||||
]
|
||||
for field in custom_fields:
|
||||
value = getattr(record, field, None)
|
||||
if value is not None:
|
||||
|
||||
@@ -0,0 +1,330 @@
|
||||
"""
|
||||
Template service for managing email templates stored in GCS.
|
||||
|
||||
Templates are stored in GCS bucket and rendered with job-specific variables.
|
||||
This allows updating email content without code deployment.
|
||||
|
||||
Template locations:
|
||||
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
|
||||
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
|
||||
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Default templates (fallback if GCS fetch fails)
|
||||
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
|
||||
|
||||
Our system has transcribed and synchronized the lyrics, but they may need some corrections. Please review and make any corrections necessary so we can finish generating your video.
|
||||
|
||||
Review your lyrics here:
|
||||
{review_url}
|
||||
|
||||
This usually takes just a few minutes.
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
|
||||
|
||||
Your karaoke video for "{artist} - {title}" is almost done!
|
||||
|
||||
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
|
||||
|
||||
Select your instrumental here:
|
||||
{instrumental_url}
|
||||
|
||||
Thanks!
|
||||
-Andrew
|
||||
"""
|
||||
|
||||
|
||||
class TemplateService:
|
||||
"""
|
||||
Service for fetching and rendering email templates from GCS.
|
||||
|
||||
Templates support the following variables:
|
||||
- {name} - User's display name or "there" if unknown
|
||||
- {youtube_url} - YouTube video URL
|
||||
- {dropbox_url} - Dropbox folder URL
|
||||
- {artist} - Artist name
|
||||
- {title} - Song title
|
||||
- {job_id} - Job ID
|
||||
- {review_url} - Lyrics review URL
|
||||
- {instrumental_url} - Instrumental selection URL
|
||||
- {feedback_url} - Feedback form URL
|
||||
"""
|
||||
|
||||
TEMPLATE_PREFIX = "templates/"
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize template service."""
|
||||
self.settings = get_settings()
|
||||
self._storage_client = None
|
||||
self._bucket = None
|
||||
|
||||
@property
|
||||
def storage_client(self):
|
||||
"""Lazy-initialize storage client."""
|
||||
if self._storage_client is None:
|
||||
from google.cloud import storage
|
||||
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
|
||||
return self._storage_client
|
||||
|
||||
@property
|
||||
def bucket(self):
|
||||
"""Get the GCS bucket for templates."""
|
||||
if self._bucket is None:
|
||||
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
|
||||
return self._bucket
|
||||
|
||||
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
|
||||
"""
|
||||
Fetch a template from GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
|
||||
Returns:
|
||||
Template content as string, or None if not found
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
if blob.exists():
|
||||
content = blob.download_as_text()
|
||||
logger.debug(f"Fetched template from GCS: {blob_path}")
|
||||
return content
|
||||
else:
|
||||
logger.warning(f"Template not found in GCS: {blob_path}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to fetch template {blob_path}: {e}")
|
||||
return None
|
||||
|
||||
def get_job_completion_template(self) -> str:
|
||||
"""Get the job completion email template."""
|
||||
template = self._fetch_template_from_gcs("job-completion.txt")
|
||||
if template is None:
|
||||
logger.info("Using default job completion template")
|
||||
return DEFAULT_JOB_COMPLETION_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_lyrics_template(self) -> str:
|
||||
"""Get the lyrics review reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
|
||||
if template is None:
|
||||
logger.info("Using default lyrics reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
|
||||
return template
|
||||
|
||||
def get_action_needed_instrumental_template(self) -> str:
|
||||
"""Get the instrumental selection reminder template."""
|
||||
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
|
||||
if template is None:
|
||||
logger.info("Using default instrumental reminder template")
|
||||
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
|
||||
return template
|
||||
|
||||
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
|
||||
"""
|
||||
Render a template with the given variables.
|
||||
|
||||
Missing variables are replaced with empty strings.
|
||||
Handles conditional sections like feedback URL.
|
||||
|
||||
Args:
|
||||
template: Template string with {variable} placeholders
|
||||
variables: Dictionary of variable values
|
||||
|
||||
Returns:
|
||||
Rendered template string
|
||||
"""
|
||||
result = template
|
||||
|
||||
# Handle feedback URL section - remove if not provided
|
||||
if not variables.get("feedback_url"):
|
||||
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
|
||||
result = re.sub(
|
||||
r'\n*If you have a moment.*?\{feedback_url\}\n*',
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["feedback_url"] = ""
|
||||
|
||||
# Replace all variables
|
||||
for key, value in variables.items():
|
||||
placeholder = "{" + key + "}"
|
||||
result = result.replace(placeholder, str(value) if value else "")
|
||||
|
||||
# Clean up any remaining unreplaced placeholders
|
||||
result = re.sub(r'\{[a-z_]+\}', '', result)
|
||||
|
||||
return result.strip()
|
||||
|
||||
def render_job_completion(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
youtube_url: Optional[str] = None,
|
||||
dropbox_url: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
job_id: Optional[str] = None,
|
||||
feedback_url: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Render the job completion email template.
|
||||
|
||||
Args:
|
||||
name: User's display name (defaults to "there")
|
||||
youtube_url: YouTube video URL
|
||||
dropbox_url: Dropbox folder URL
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
job_id: Job ID
|
||||
feedback_url: Feedback form URL (optional)
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_job_completion_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"youtube_url": youtube_url or "[YouTube URL not available]",
|
||||
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"job_id": job_id or "",
|
||||
"feedback_url": feedback_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_lyrics(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
review_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the lyrics review reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
review_url: Lyrics review URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_lyrics_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"review_url": review_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def render_action_needed_instrumental(
|
||||
self,
|
||||
name: Optional[str] = None,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
instrumental_url: str = "",
|
||||
) -> str:
|
||||
"""
|
||||
Render the instrumental selection reminder template.
|
||||
|
||||
Args:
|
||||
name: User's display name
|
||||
artist: Artist name
|
||||
title: Song title
|
||||
instrumental_url: Instrumental selection URL
|
||||
|
||||
Returns:
|
||||
Rendered email content
|
||||
"""
|
||||
template = self.get_action_needed_instrumental_template()
|
||||
variables = {
|
||||
"name": name or "there",
|
||||
"artist": artist or "Unknown Artist",
|
||||
"title": title or "Unknown Title",
|
||||
"instrumental_url": instrumental_url,
|
||||
}
|
||||
return self.render_template(template, variables)
|
||||
|
||||
def upload_template(self, template_name: str, content: str) -> bool:
|
||||
"""
|
||||
Upload a template to GCS.
|
||||
|
||||
Args:
|
||||
template_name: Name of template file (e.g., "job-completion.txt")
|
||||
content: Template content
|
||||
|
||||
Returns:
|
||||
True if upload successful
|
||||
"""
|
||||
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
|
||||
try:
|
||||
blob = self.bucket.blob(blob_path)
|
||||
blob.upload_from_string(content, content_type="text/plain")
|
||||
logger.info(f"Uploaded template to GCS: {blob_path}")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to upload template {blob_path}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_template_service: Optional[TemplateService] = None
|
||||
|
||||
|
||||
def get_template_service() -> TemplateService:
|
||||
"""Get the global template service instance."""
|
||||
global _template_service
|
||||
if _template_service is None:
|
||||
_template_service = TemplateService()
|
||||
return _template_service
|
||||
@@ -0,0 +1,285 @@
|
||||
"""
|
||||
Tenant service for managing white-label B2B portal configurations.
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json.
|
||||
The service provides:
|
||||
- Tenant config loading with caching
|
||||
- Subdomain to tenant ID resolution
|
||||
- Tenant validation
|
||||
|
||||
Similar to ThemeService, configs are cached for 5 minutes to reduce GCS reads.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from typing import Dict, Optional
|
||||
|
||||
from backend.models.tenant import TenantConfig, TenantPublicConfig
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for tenant configs
|
||||
TENANTS_PREFIX = "tenants"
|
||||
|
||||
# Default sender email (shared with EmailService for consistency)
|
||||
DEFAULT_SENDER_EMAIL = "gen@nomadkaraoke.com"
|
||||
|
||||
|
||||
class TenantService:
|
||||
"""Service for managing tenant configurations from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the tenant service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._config_cache: Dict[str, TenantConfig] = {}
|
||||
self._cache_times: Dict[str, datetime] = {}
|
||||
self._subdomain_map: Dict[str, str] = {} # subdomain -> tenant_id
|
||||
self._subdomain_map_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache
|
||||
|
||||
def _get_config_path(self, tenant_id: str) -> str:
|
||||
"""Get the GCS path for a tenant's config file."""
|
||||
return f"{TENANTS_PREFIX}/{tenant_id}/config.json"
|
||||
|
||||
def _is_cache_valid(self, tenant_id: str) -> bool:
|
||||
"""Check if the config cache for a tenant is still valid."""
|
||||
if tenant_id not in self._config_cache or tenant_id not in self._cache_times:
|
||||
return False
|
||||
age = datetime.now() - self._cache_times[tenant_id]
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _is_subdomain_map_valid(self) -> bool:
|
||||
"""Check if the subdomain map cache is still valid."""
|
||||
if not self._subdomain_map or self._subdomain_map_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._subdomain_map_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def get_tenant_config(
|
||||
self, tenant_id: str, force_refresh: bool = False
|
||||
) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Load tenant configuration from GCS with caching.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
TenantConfig if found, None otherwise
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid(tenant_id):
|
||||
return self._config_cache[tenant_id]
|
||||
|
||||
try:
|
||||
config_path = self._get_config_path(tenant_id)
|
||||
if not self.storage.file_exists(config_path):
|
||||
logger.debug(f"Tenant config not found: {tenant_id}")
|
||||
return None
|
||||
|
||||
data = self.storage.download_json(config_path)
|
||||
config = TenantConfig(**data)
|
||||
|
||||
# Update cache
|
||||
self._config_cache[tenant_id] = config
|
||||
self._cache_times[tenant_id] = datetime.now()
|
||||
|
||||
# Update subdomain map and its timestamp
|
||||
self._subdomain_map[config.subdomain.lower()] = tenant_id
|
||||
self._subdomain_map_time = datetime.now()
|
||||
|
||||
logger.info(f"Loaded tenant config: {tenant_id}")
|
||||
return config
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load tenant config {tenant_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_tenant_by_subdomain(self, subdomain: str) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Get tenant config by subdomain.
|
||||
|
||||
Args:
|
||||
subdomain: Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')
|
||||
|
||||
Returns:
|
||||
TenantConfig if found, None otherwise
|
||||
"""
|
||||
subdomain_lower = subdomain.lower()
|
||||
|
||||
# Check cache first
|
||||
if subdomain_lower in self._subdomain_map:
|
||||
tenant_id = self._subdomain_map[subdomain_lower]
|
||||
return self.get_tenant_config(tenant_id)
|
||||
|
||||
# Try to find tenant by listing all configs
|
||||
# This is a fallback for when we don't have the subdomain cached
|
||||
tenant_id = self._resolve_subdomain_to_tenant_id(subdomain_lower)
|
||||
if tenant_id:
|
||||
return self.get_tenant_config(tenant_id)
|
||||
|
||||
return None
|
||||
|
||||
def _resolve_subdomain_to_tenant_id(self, subdomain: str) -> Optional[str]:
|
||||
"""
|
||||
Resolve a subdomain to tenant ID by checking GCS.
|
||||
|
||||
This extracts the tenant ID from the subdomain pattern:
|
||||
- 'vocalstar.nomadkaraoke.com' -> 'vocalstar'
|
||||
- 'vocalstar.gen.nomadkaraoke.com' -> 'vocalstar'
|
||||
"""
|
||||
# Extract potential tenant ID from subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
parts = subdomain.split(".")
|
||||
|
||||
if len(parts) >= 3:
|
||||
# First part is likely the tenant ID
|
||||
potential_tenant_id = parts[0]
|
||||
|
||||
# Check if config exists
|
||||
config_path = self._get_config_path(potential_tenant_id)
|
||||
if self.storage.file_exists(config_path):
|
||||
return potential_tenant_id
|
||||
|
||||
return None
|
||||
|
||||
def get_public_config(self, tenant_id: str) -> Optional[TenantPublicConfig]:
|
||||
"""
|
||||
Get the public (frontend-safe) tenant configuration.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
TenantPublicConfig if found, None otherwise
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if config:
|
||||
return TenantPublicConfig.from_config(config)
|
||||
return None
|
||||
|
||||
def get_public_config_by_subdomain(
|
||||
self, subdomain: str
|
||||
) -> Optional[TenantPublicConfig]:
|
||||
"""
|
||||
Get the public tenant configuration by subdomain.
|
||||
|
||||
Args:
|
||||
subdomain: Full subdomain
|
||||
|
||||
Returns:
|
||||
TenantPublicConfig if found, None otherwise
|
||||
"""
|
||||
config = self.get_tenant_by_subdomain(subdomain)
|
||||
if config:
|
||||
return TenantPublicConfig.from_config(config)
|
||||
return None
|
||||
|
||||
def tenant_exists(self, tenant_id: str) -> bool:
|
||||
"""
|
||||
Check if a tenant exists.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
True if tenant config exists, False otherwise
|
||||
"""
|
||||
config_path = self._get_config_path(tenant_id)
|
||||
return self.storage.file_exists(config_path)
|
||||
|
||||
def is_email_allowed_for_tenant(self, tenant_id: str, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is allowed for a specific tenant.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if email is allowed, False otherwise
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if not config:
|
||||
return False
|
||||
return config.is_email_allowed(email)
|
||||
|
||||
def get_tenant_sender_email(self, tenant_id: str) -> str:
|
||||
"""
|
||||
Get the email sender address for a tenant.
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
|
||||
Returns:
|
||||
Sender email address
|
||||
"""
|
||||
config = self.get_tenant_config(tenant_id)
|
||||
if config:
|
||||
return config.get_sender_email()
|
||||
# Default fallback (consistent with EmailService)
|
||||
return DEFAULT_SENDER_EMAIL
|
||||
|
||||
def invalidate_cache(self, tenant_id: Optional[str] = None) -> None:
|
||||
"""
|
||||
Invalidate tenant config cache.
|
||||
|
||||
Args:
|
||||
tenant_id: Specific tenant to invalidate, or None for all
|
||||
"""
|
||||
if tenant_id:
|
||||
self._config_cache.pop(tenant_id, None)
|
||||
self._cache_times.pop(tenant_id, None)
|
||||
# Also remove any subdomain map entries pointing to this tenant
|
||||
subdomains_to_remove = [
|
||||
subdomain
|
||||
for subdomain, tid in self._subdomain_map.items()
|
||||
if tid == tenant_id
|
||||
]
|
||||
for subdomain in subdomains_to_remove:
|
||||
self._subdomain_map.pop(subdomain, None)
|
||||
logger.info(f"Tenant config cache invalidated: {tenant_id}")
|
||||
else:
|
||||
self._config_cache.clear()
|
||||
self._cache_times.clear()
|
||||
self._subdomain_map.clear()
|
||||
self._subdomain_map_time = None
|
||||
logger.info("All tenant config caches invalidated")
|
||||
|
||||
def get_asset_url(self, tenant_id: str, asset_name: str) -> Optional[str]:
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
Args:
|
||||
tenant_id: The tenant identifier
|
||||
asset_name: Asset filename (e.g., 'logo.png')
|
||||
|
||||
Returns:
|
||||
Signed URL if asset exists, None otherwise
|
||||
"""
|
||||
asset_path = f"{TENANTS_PREFIX}/{tenant_id}/{asset_name}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
return self.storage.generate_signed_url(asset_path, expiration_minutes=60)
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_tenant_service: Optional[TenantService] = None
|
||||
_tenant_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_tenant_service() -> TenantService:
|
||||
"""Get or create the singleton TenantService instance (thread-safe)."""
|
||||
global _tenant_service
|
||||
if _tenant_service is None:
|
||||
with _tenant_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _tenant_service is None:
|
||||
_tenant_service = TenantService()
|
||||
return _tenant_service
|
||||
@@ -0,0 +1,469 @@
|
||||
"""
|
||||
Theme service for managing pre-created style themes.
|
||||
|
||||
Themes are stored in GCS at themes/{theme_id}/ with:
|
||||
- style_params.json: Complete style configuration
|
||||
- preview.png: Preview image for theme selection UI
|
||||
- youtube_description.txt: Optional YouTube description template
|
||||
- assets/: Fonts, backgrounds, and other assets
|
||||
|
||||
The service provides:
|
||||
- Theme listing with signed preview URLs
|
||||
- Theme detail retrieval
|
||||
- Color override application
|
||||
- Job style preparation (copying theme to job folder)
|
||||
"""
|
||||
|
||||
import copy
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from backend.models.theme import (
|
||||
ColorOverrides,
|
||||
ThemeDetail,
|
||||
ThemeMetadata,
|
||||
ThemeRegistry,
|
||||
ThemeSummary,
|
||||
hex_to_rgba,
|
||||
)
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# GCS paths for themes
|
||||
THEMES_PREFIX = "themes"
|
||||
METADATA_FILE = f"{THEMES_PREFIX}/_metadata.json"
|
||||
|
||||
|
||||
class ThemeService:
|
||||
"""Service for managing themes from GCS."""
|
||||
|
||||
def __init__(self, storage: Optional[StorageService] = None):
|
||||
"""
|
||||
Initialize the theme service.
|
||||
|
||||
Args:
|
||||
storage: StorageService instance (creates new one if not provided)
|
||||
"""
|
||||
self.storage = storage or StorageService()
|
||||
self._metadata_cache: Optional[ThemeRegistry] = None
|
||||
self._cache_time: Optional[datetime] = None
|
||||
self.CACHE_TTL_SECONDS = 300 # 5 minute cache for theme metadata
|
||||
|
||||
def _get_theme_path(self, theme_id: str, filename: str = "") -> str:
|
||||
"""Get the GCS path for a theme file."""
|
||||
if filename:
|
||||
return f"{THEMES_PREFIX}/{theme_id}/{filename}"
|
||||
return f"{THEMES_PREFIX}/{theme_id}"
|
||||
|
||||
def _is_cache_valid(self) -> bool:
|
||||
"""Check if the metadata cache is still valid."""
|
||||
if self._metadata_cache is None or self._cache_time is None:
|
||||
return False
|
||||
age = datetime.now() - self._cache_time
|
||||
return age.total_seconds() < self.CACHE_TTL_SECONDS
|
||||
|
||||
def _load_metadata(self, force_refresh: bool = False) -> ThemeRegistry:
|
||||
"""
|
||||
Load theme metadata from GCS with caching.
|
||||
|
||||
Args:
|
||||
force_refresh: Force reload from GCS even if cache is valid
|
||||
|
||||
Returns:
|
||||
ThemeRegistry containing all theme metadata
|
||||
"""
|
||||
if not force_refresh and self._is_cache_valid():
|
||||
return self._metadata_cache # type: ignore
|
||||
|
||||
try:
|
||||
data = self.storage.download_json(METADATA_FILE)
|
||||
self._metadata_cache = ThemeRegistry(**data)
|
||||
self._cache_time = datetime.now()
|
||||
logger.info(f"Loaded theme metadata: {len(self._metadata_cache.themes)} themes")
|
||||
return self._metadata_cache
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load theme metadata: {e}")
|
||||
# Return empty registry on error
|
||||
return ThemeRegistry(version=1, themes=[])
|
||||
|
||||
def list_themes(self) -> List[ThemeSummary]:
|
||||
"""
|
||||
List all available themes with signed preview URLs.
|
||||
|
||||
Returns:
|
||||
List of ThemeSummary objects with preview URLs
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
summaries = []
|
||||
|
||||
for theme in metadata.themes:
|
||||
try:
|
||||
# Generate signed URLs for preview images
|
||||
preview_path = self._get_theme_path(theme.id, "preview.png")
|
||||
thumbnail_path = self._get_theme_path(theme.id, "preview_thumbnail.png")
|
||||
|
||||
preview_url = None
|
||||
thumbnail_url = None
|
||||
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
# Try thumbnail, fall back to main preview
|
||||
if self.storage.file_exists(thumbnail_path):
|
||||
thumbnail_url = self.storage.generate_signed_url(thumbnail_path, expiration_minutes=60)
|
||||
elif preview_url:
|
||||
thumbnail_url = preview_url
|
||||
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
preview_url=preview_url,
|
||||
thumbnail_url=thumbnail_url,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Error processing theme {theme.id}: {e}")
|
||||
# Still include theme but without preview URLs
|
||||
summaries.append(
|
||||
ThemeSummary(
|
||||
id=theme.id,
|
||||
name=theme.name,
|
||||
description=theme.description,
|
||||
is_default=theme.is_default,
|
||||
)
|
||||
)
|
||||
|
||||
return summaries
|
||||
|
||||
def get_theme(self, theme_id: str) -> Optional[ThemeDetail]:
|
||||
"""
|
||||
Get full theme details including style parameters.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
ThemeDetail if found, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
|
||||
# Find theme in metadata
|
||||
theme_meta = next((t for t in metadata.themes if t.id == theme_id), None)
|
||||
if not theme_meta:
|
||||
logger.warning(f"Theme not found: {theme_id}")
|
||||
return None
|
||||
|
||||
try:
|
||||
# Load style_params.json
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
style_params = self.storage.download_json(style_params_path)
|
||||
|
||||
# Check for YouTube description
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
has_youtube_desc = self.storage.file_exists(youtube_desc_path)
|
||||
|
||||
# Generate preview URL
|
||||
preview_path = self._get_theme_path(theme_id, "preview.png")
|
||||
preview_url = None
|
||||
if self.storage.file_exists(preview_path):
|
||||
preview_url = self.storage.generate_signed_url(preview_path, expiration_minutes=60)
|
||||
|
||||
return ThemeDetail(
|
||||
id=theme_id,
|
||||
name=theme_meta.name,
|
||||
description=theme_meta.description,
|
||||
preview_url=preview_url,
|
||||
is_default=theme_meta.is_default,
|
||||
style_params=style_params,
|
||||
has_youtube_description=has_youtube_desc,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_theme_style_params(self, theme_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get just the style_params.json for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
Style parameters dict if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
style_params_path = self._get_theme_path(theme_id, "style_params.json")
|
||||
return self.storage.download_json(style_params_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading style params for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_description(self, theme_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
YouTube description text if found, None otherwise
|
||||
"""
|
||||
try:
|
||||
youtube_desc_path = self._get_theme_path(theme_id, "youtube_description.txt")
|
||||
if not self.storage.file_exists(youtube_desc_path):
|
||||
return None
|
||||
|
||||
# Download to temp file and read
|
||||
with tempfile.NamedTemporaryFile(mode="w", delete=False, suffix=".txt") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
try:
|
||||
self.storage.download_file(youtube_desc_path, tmp_path)
|
||||
with open(tmp_path, "r") as f:
|
||||
return f.read()
|
||||
finally:
|
||||
if os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
except Exception as e:
|
||||
logger.error(f"Error loading YouTube description for theme {theme_id}: {e}")
|
||||
return None
|
||||
|
||||
def apply_color_overrides(
|
||||
self, style_params: Dict[str, Any], overrides: ColorOverrides
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Apply user color overrides to theme style parameters.
|
||||
|
||||
Args:
|
||||
style_params: Base style parameters from theme
|
||||
overrides: User color overrides
|
||||
|
||||
Returns:
|
||||
Modified style parameters with overrides applied
|
||||
"""
|
||||
if not overrides.has_overrides():
|
||||
return style_params
|
||||
|
||||
# Deep copy to avoid modifying original
|
||||
result = copy.deepcopy(style_params)
|
||||
|
||||
# Apply artist_color to intro, end, and CDG sections
|
||||
if overrides.artist_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["artist_color"] = overrides.artist_color
|
||||
if "end" in result:
|
||||
result["end"]["artist_color"] = overrides.artist_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["artist_color"] = overrides.artist_color
|
||||
|
||||
# Apply title_color to intro, end, and CDG sections
|
||||
if overrides.title_color:
|
||||
if "intro" in result:
|
||||
result["intro"]["title_color"] = overrides.title_color
|
||||
if "end" in result:
|
||||
result["end"]["title_color"] = overrides.title_color
|
||||
if "cdg" in result:
|
||||
result["cdg"]["title_color"] = overrides.title_color
|
||||
|
||||
# Apply sung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.sung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["primary_color"] = hex_to_rgba(overrides.sung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["active_fill"] = overrides.sung_lyrics_color
|
||||
|
||||
# Apply unsung_lyrics_color (karaoke uses RGBA, CDG uses hex)
|
||||
if overrides.unsung_lyrics_color:
|
||||
if "karaoke" in result:
|
||||
result["karaoke"]["secondary_color"] = hex_to_rgba(overrides.unsung_lyrics_color)
|
||||
if "cdg" in result:
|
||||
result["cdg"]["inactive_fill"] = overrides.unsung_lyrics_color
|
||||
|
||||
return result
|
||||
|
||||
def prepare_job_style(
|
||||
self,
|
||||
job_id: str,
|
||||
theme_id: str,
|
||||
color_overrides: Optional[ColorOverrides] = None,
|
||||
) -> Tuple[str, Dict[str, str]]:
|
||||
"""
|
||||
Prepare style files for a job by copying theme to job folder.
|
||||
|
||||
This method:
|
||||
1. Loads the theme's style_params.json
|
||||
2. Applies any color overrides
|
||||
3. Updates asset paths to point to theme assets (shared, not copied)
|
||||
4. Uploads modified style_params.json to job folder
|
||||
5. Returns the GCS path and style_assets mapping
|
||||
|
||||
Args:
|
||||
job_id: The job ID
|
||||
theme_id: The theme to use
|
||||
color_overrides: Optional color overrides
|
||||
|
||||
Returns:
|
||||
Tuple of (style_params_gcs_path, style_assets dict)
|
||||
|
||||
Raises:
|
||||
ValueError: If theme not found
|
||||
"""
|
||||
# Load theme style params
|
||||
style_params = self.get_theme_style_params(theme_id)
|
||||
if style_params is None:
|
||||
raise ValueError(f"Theme not found: {theme_id}")
|
||||
|
||||
# Apply color overrides if provided
|
||||
if color_overrides:
|
||||
style_params = self.apply_color_overrides(style_params, color_overrides)
|
||||
|
||||
# Update asset paths to point to theme's shared assets
|
||||
# Theme assets stay in themes/{theme_id}/assets/ - they're shared
|
||||
style_assets = self._build_style_assets_mapping(theme_id, style_params)
|
||||
|
||||
# Update style_params to use theme asset paths (not job-specific)
|
||||
style_params = self._update_asset_paths_in_style(theme_id, style_params)
|
||||
|
||||
# Upload modified style_params.json to job's style folder
|
||||
job_style_path = f"uploads/{job_id}/style/style_params.json"
|
||||
self.storage.upload_json(job_style_path, style_params)
|
||||
|
||||
logger.info(f"Prepared job {job_id} style from theme {theme_id}")
|
||||
|
||||
return job_style_path, style_assets
|
||||
|
||||
def _build_style_assets_mapping(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, str]:
|
||||
"""
|
||||
Build the style_assets mapping for a theme.
|
||||
|
||||
Maps asset keys to their GCS paths in the theme folder.
|
||||
"""
|
||||
style_assets = {}
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Check each potential asset type
|
||||
asset_checks = [
|
||||
("intro_background", "intro", "background_image"),
|
||||
("karaoke_background", "karaoke", "background_image"),
|
||||
("end_background", "end", "background_image"),
|
||||
("font", "intro", "font"), # Font is typically shared across sections
|
||||
("cdg_instrumental_background", "cdg", "instrumental_background"),
|
||||
("cdg_title_background", "cdg", "title_screen_background"),
|
||||
("cdg_outro_background", "cdg", "outro_background"),
|
||||
]
|
||||
|
||||
for asset_key, section, field in asset_checks:
|
||||
if section in style_params and field in style_params[section]:
|
||||
value = style_params[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Check if it's a theme asset path or already has the full path
|
||||
if value.startswith(f"{THEMES_PREFIX}/"):
|
||||
# Already a full theme path
|
||||
style_assets[asset_key] = value
|
||||
elif not value.startswith("/") and not value.startswith("gs://"):
|
||||
# Relative path - prepend theme assets prefix
|
||||
asset_path = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
if self.storage.file_exists(asset_path):
|
||||
style_assets[asset_key] = asset_path
|
||||
|
||||
return style_assets
|
||||
|
||||
def _update_asset_paths_in_style(
|
||||
self, theme_id: str, style_params: Dict[str, Any]
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update asset paths in style_params to use full GCS theme paths.
|
||||
|
||||
This ensures the style_loader can find assets when processing.
|
||||
"""
|
||||
result = copy.deepcopy(style_params)
|
||||
theme_assets_prefix = f"{THEMES_PREFIX}/{theme_id}/assets"
|
||||
|
||||
# Fields that contain asset paths
|
||||
path_fields = {
|
||||
"intro": ["background_image", "font"],
|
||||
"end": ["background_image", "font"],
|
||||
"karaoke": ["background_image", "font_path"],
|
||||
"cdg": [
|
||||
"font_path",
|
||||
"instrumental_background",
|
||||
"title_screen_background",
|
||||
"outro_background",
|
||||
],
|
||||
}
|
||||
|
||||
for section, fields in path_fields.items():
|
||||
if section not in result:
|
||||
continue
|
||||
for field in fields:
|
||||
if field not in result[section]:
|
||||
continue
|
||||
value = result[section][field]
|
||||
if value and isinstance(value, str):
|
||||
# Skip if already a full path or URL
|
||||
if value.startswith(f"{THEMES_PREFIX}/") or value.startswith("gs://"):
|
||||
continue
|
||||
# Skip absolute local paths (shouldn't happen but be safe)
|
||||
if value.startswith("/"):
|
||||
continue
|
||||
# Update to full theme asset path
|
||||
result[section][field] = f"{theme_assets_prefix}/{os.path.basename(value)}"
|
||||
|
||||
return result
|
||||
|
||||
def theme_exists(self, theme_id: str) -> bool:
|
||||
"""
|
||||
Check if a theme exists.
|
||||
|
||||
Args:
|
||||
theme_id: The theme identifier
|
||||
|
||||
Returns:
|
||||
True if theme exists, False otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
return any(t.id == theme_id for t in metadata.themes)
|
||||
|
||||
def get_default_theme_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the ID of the default theme.
|
||||
|
||||
Returns:
|
||||
Default theme ID if one exists, None otherwise
|
||||
"""
|
||||
metadata = self._load_metadata()
|
||||
default_theme = next((t for t in metadata.themes if t.is_default), None)
|
||||
return default_theme.id if default_theme else None
|
||||
|
||||
def invalidate_cache(self) -> None:
|
||||
"""Force invalidation of the metadata cache."""
|
||||
self._metadata_cache = None
|
||||
self._cache_time = None
|
||||
logger.info("Theme metadata cache invalidated")
|
||||
|
||||
|
||||
# Singleton instance with thread-safe initialization
|
||||
_theme_service: Optional[ThemeService] = None
|
||||
_theme_service_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_theme_service() -> ThemeService:
|
||||
"""Get or create the singleton ThemeService instance (thread-safe)."""
|
||||
global _theme_service
|
||||
if _theme_service is None:
|
||||
with _theme_service_lock:
|
||||
# Double-check after acquiring lock
|
||||
if _theme_service is None:
|
||||
_theme_service = ThemeService()
|
||||
return _theme_service
|
||||
@@ -0,0 +1,799 @@
|
||||
"""
|
||||
User service for authentication, credits, and user management.
|
||||
|
||||
Handles:
|
||||
- Magic link authentication (send, verify)
|
||||
- Session management
|
||||
- Credit operations (add, deduct, refund)
|
||||
- User CRUD operations
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, List, Tuple
|
||||
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud.firestore_v1 import Increment
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import (
|
||||
User,
|
||||
UserRole,
|
||||
UserPublic,
|
||||
MagicLinkToken,
|
||||
Session,
|
||||
CreditTransaction,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Collection names
|
||||
USERS_COLLECTION = "gen_users"
|
||||
MAGIC_LINKS_COLLECTION = "magic_links"
|
||||
SESSIONS_COLLECTION = "sessions"
|
||||
PROCESSED_STRIPE_SESSIONS_COLLECTION = "processed_stripe_sessions"
|
||||
|
||||
# Token/session configuration
|
||||
MAGIC_LINK_EXPIRY_MINUTES = 15
|
||||
SESSION_EXPIRY_DAYS = 7
|
||||
SESSION_ABSOLUTE_EXPIRY_DAYS = 30
|
||||
MAX_CREDIT_TRANSACTIONS = 100 # Keep last N transactions
|
||||
|
||||
|
||||
class UserService:
|
||||
"""Service for user management and authentication."""
|
||||
|
||||
# Number of free credits granted to new users
|
||||
NEW_USER_FREE_CREDITS = 1
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize user service with Firestore client."""
|
||||
self.settings = get_settings()
|
||||
self.db = firestore.Client(project=self.settings.google_cloud_project)
|
||||
|
||||
# =========================================================================
|
||||
# User CRUD Operations
|
||||
# =========================================================================
|
||||
|
||||
def get_user(self, email: str) -> Optional[User]:
|
||||
"""Get a user by email."""
|
||||
try:
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return User(**doc.to_dict())
|
||||
except Exception:
|
||||
logger.exception(f"Error getting user {email}")
|
||||
return None
|
||||
|
||||
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
|
||||
"""
|
||||
Get existing user or create a new one.
|
||||
|
||||
New users receive a welcome credit to try the service.
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
|
||||
Note: If user exists but has a different tenant_id, the existing user is returned.
|
||||
Users are uniquely identified by email, not email+tenant.
|
||||
"""
|
||||
email = email.lower()
|
||||
user = self.get_user(email)
|
||||
|
||||
if user:
|
||||
# If user exists but tenant_id differs, we still return the user
|
||||
# This allows users to access multiple tenants with one account
|
||||
# (though features may be restricted per-tenant)
|
||||
return user
|
||||
|
||||
# Create new user with welcome credit
|
||||
welcome_credit = self.NEW_USER_FREE_CREDITS
|
||||
welcome_transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=welcome_credit,
|
||||
reason="welcome_credit",
|
||||
)
|
||||
|
||||
user = User(
|
||||
email=email,
|
||||
credits=welcome_credit,
|
||||
credit_transactions=[welcome_transaction],
|
||||
tenant_id=tenant_id, # Associate with tenant on creation
|
||||
)
|
||||
self._save_user(user)
|
||||
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
|
||||
return user
|
||||
|
||||
def _save_user(self, user: User) -> None:
|
||||
"""Save user to Firestore."""
|
||||
try:
|
||||
user.updated_at = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(user.email.lower())
|
||||
doc_ref.set(user.model_dump(mode='json'))
|
||||
except Exception:
|
||||
logger.exception(f"Error saving user {user.email}")
|
||||
raise
|
||||
|
||||
def update_user(self, email: str, **updates) -> Optional[User]:
|
||||
"""Update user fields."""
|
||||
try:
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update(updates)
|
||||
return self.get_user(email)
|
||||
except Exception:
|
||||
logger.exception(f"Error updating user {email}")
|
||||
return None
|
||||
|
||||
def list_users(self, limit: int = 100, include_inactive: bool = False) -> List[User]:
|
||||
"""List all users (admin only)."""
|
||||
try:
|
||||
query = self.db.collection(USERS_COLLECTION)
|
||||
|
||||
if not include_inactive:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
return [User(**doc.to_dict()) for doc in docs]
|
||||
except Exception:
|
||||
logger.exception("Error listing users")
|
||||
return []
|
||||
|
||||
def get_user_public(self, email: str) -> Optional[UserPublic]:
|
||||
"""Get public user info."""
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return None
|
||||
|
||||
return UserPublic(
|
||||
email=user.email,
|
||||
role=user.role,
|
||||
credits=user.credits,
|
||||
display_name=user.display_name,
|
||||
total_jobs_created=user.total_jobs_created,
|
||||
total_jobs_completed=user.total_jobs_completed,
|
||||
tenant_id=user.tenant_id,
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# Magic Link Authentication
|
||||
# =========================================================================
|
||||
|
||||
def create_magic_link(
|
||||
self,
|
||||
email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create a magic link token for email authentication.
|
||||
|
||||
Returns the token object. The actual sending of the email
|
||||
should be handled by the caller (or an email service).
|
||||
|
||||
Args:
|
||||
email: User's email address
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
"""
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists (with tenant association)
|
||||
self.get_or_create_user(email, tenant_id=tenant_id)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
|
||||
magic_link = MagicLinkToken(
|
||||
token=token,
|
||||
email=email,
|
||||
expires_at=datetime.utcnow() + timedelta(minutes=MAGIC_LINK_EXPIRY_MINUTES),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
# Save to Firestore
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
doc_ref.set(magic_link.model_dump(mode='json'))
|
||||
|
||||
logger.info(f"Created magic link for {email}")
|
||||
return magic_link
|
||||
|
||||
def create_admin_login_token(
|
||||
self,
|
||||
email: str,
|
||||
expiry_hours: int = 24,
|
||||
) -> MagicLinkToken:
|
||||
"""
|
||||
Create an admin login token for email-embedded authentication links.
|
||||
|
||||
Similar to magic links but with configurable expiry (default 24 hours).
|
||||
Used for made-for-you order notification emails to allow admin one-click login.
|
||||
|
||||
Args:
|
||||
email: Admin's email address to authenticate as
|
||||
expiry_hours: Hours until token expires (default: 24, max: 168)
|
||||
|
||||
Returns:
|
||||
MagicLinkToken object containing the token
|
||||
|
||||
Raises:
|
||||
ValueError: If expiry_hours is out of valid range (1-168)
|
||||
"""
|
||||
# Validate expiry_hours (1 hour to 7 days)
|
||||
if not 1 <= expiry_hours <= 168:
|
||||
raise ValueError(f"expiry_hours must be between 1 and 168, got {expiry_hours}")
|
||||
|
||||
email = email.lower()
|
||||
|
||||
# Ensure user exists
|
||||
self.get_or_create_user(email)
|
||||
|
||||
# Generate secure token
|
||||
token = secrets.token_urlsafe(32)
|
||||
|
||||
admin_login = MagicLinkToken(
|
||||
token=token,
|
||||
email=email,
|
||||
expires_at=datetime.utcnow() + timedelta(hours=expiry_hours),
|
||||
)
|
||||
|
||||
# Save to Firestore (same collection as magic links for unified verification)
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
doc_ref.set(admin_login.model_dump(mode='json'))
|
||||
|
||||
# Log with redacted email (show only domain) for PII protection
|
||||
domain = email.split('@')[-1] if '@' in email else 'unknown'
|
||||
logger.info(f"Created admin login token for ***@{domain} (expires in {expiry_hours}h)")
|
||||
return admin_login
|
||||
|
||||
def verify_magic_link(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Verify a magic link token using a Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, user, message)
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
|
||||
|
||||
@firestore.transactional
|
||||
def verify_in_transaction(transaction):
|
||||
"""Atomically verify and mark magic link as used."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, None, "Invalid or expired link"
|
||||
|
||||
magic_link = MagicLinkToken(**doc.to_dict())
|
||||
|
||||
# Check if already used
|
||||
if magic_link.used:
|
||||
return False, None, "This link has already been used"
|
||||
|
||||
# Check expiry
|
||||
if datetime.utcnow() > magic_link.expires_at:
|
||||
return False, None, "This link has expired"
|
||||
|
||||
# Mark as used atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'used': True,
|
||||
'used_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, magic_link.email, "Success"
|
||||
|
||||
# Execute the transaction
|
||||
transaction = self.db.transaction()
|
||||
success, email_or_error, message = verify_in_transaction(transaction)
|
||||
|
||||
if not success:
|
||||
return False, None, message
|
||||
|
||||
# Get user and mark email as verified (outside transaction)
|
||||
user = self.get_user(email_or_error)
|
||||
if user:
|
||||
self.update_user(
|
||||
email_or_error,
|
||||
email_verified=True,
|
||||
last_login_at=datetime.utcnow()
|
||||
)
|
||||
user = self.get_user(email_or_error) # Refresh
|
||||
|
||||
logger.info(f"Magic link verified for {email_or_error}")
|
||||
return True, user, "Successfully authenticated"
|
||||
|
||||
except Exception:
|
||||
logger.exception("Error verifying magic link")
|
||||
return False, None, "An error occurred during verification"
|
||||
|
||||
# =========================================================================
|
||||
# Session Management
|
||||
# =========================================================================
|
||||
|
||||
def create_session(
|
||||
self,
|
||||
user_email: str,
|
||||
ip_address: Optional[str] = None,
|
||||
user_agent: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None
|
||||
) -> Session:
|
||||
"""
|
||||
Create a new session for an authenticated user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
ip_address: Client IP for auditing
|
||||
user_agent: Client user agent for auditing
|
||||
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
|
||||
"""
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_prefix = token[:12]
|
||||
|
||||
session = Session(
|
||||
token=token,
|
||||
user_email=user_email.lower(),
|
||||
expires_at=datetime.utcnow() + timedelta(days=SESSION_ABSOLUTE_EXPIRY_DAYS),
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
)
|
||||
|
||||
# Serialize and write to Firestore
|
||||
session_data = session.model_dump(mode='json')
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.set(session_data)
|
||||
|
||||
# Verify the write succeeded by reading back
|
||||
verify_doc = doc_ref.get()
|
||||
if not verify_doc.exists:
|
||||
logger.error(f"Session write verification FAILED for {user_email}: {token_prefix}...")
|
||||
raise RuntimeError(f"Failed to persist session for {user_email}")
|
||||
|
||||
logger.info(f"Created and verified session for {user_email}: {token_prefix}... (expires: {session.expires_at})")
|
||||
|
||||
return session
|
||||
|
||||
def validate_session(self, token: str) -> Tuple[bool, Optional[User], str]:
|
||||
"""
|
||||
Validate a session token.
|
||||
|
||||
Returns:
|
||||
(valid, user, message)
|
||||
"""
|
||||
# Log token info for debugging (only prefix for security)
|
||||
token_prefix = token[:12] if token and len(token) >= 12 else token
|
||||
logger.debug(f"Validating session token: {token_prefix}... (len={len(token) if token else 0})")
|
||||
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
logger.warning(f"Session not found in Firestore: {token_prefix}...")
|
||||
return False, None, "Invalid session"
|
||||
|
||||
raw_data = doc.to_dict()
|
||||
logger.debug(f"Session data found for {token_prefix}...: user_email={raw_data.get('user_email')}, is_active={raw_data.get('is_active')}")
|
||||
|
||||
session = Session(**raw_data)
|
||||
|
||||
# Check if active
|
||||
if not session.is_active:
|
||||
logger.warning(f"Session revoked for {token_prefix}... (user: {session.user_email})")
|
||||
return False, None, "Session has been revoked"
|
||||
|
||||
# Use timezone-aware datetime for all comparisons
|
||||
# Firestore returns timezone-aware datetimes, so we must use aware datetimes
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Normalize session datetimes to be timezone-aware for comparison
|
||||
# (handles legacy naive datetimes that may exist in Firestore)
|
||||
expires_at = session.expires_at
|
||||
if expires_at.tzinfo is None:
|
||||
expires_at = expires_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
last_activity = session.last_activity_at
|
||||
if last_activity.tzinfo is None:
|
||||
last_activity = last_activity.replace(tzinfo=timezone.utc)
|
||||
|
||||
# Check expiry
|
||||
if now > expires_at:
|
||||
logger.warning(f"Session expired for {token_prefix}... (user: {session.user_email}, expired_at: {expires_at}, now: {now})")
|
||||
return False, None, "Session has expired"
|
||||
|
||||
# Check inactivity (7 days)
|
||||
inactivity_limit = now - timedelta(days=SESSION_EXPIRY_DAYS)
|
||||
if last_activity < inactivity_limit:
|
||||
logger.warning(f"Session inactive for {token_prefix}... (user: {session.user_email}, last_activity: {last_activity}, limit: {inactivity_limit})")
|
||||
return False, None, "Session expired due to inactivity"
|
||||
|
||||
# Update last activity (use timezone-aware datetime)
|
||||
doc_ref.update({'last_activity_at': now})
|
||||
|
||||
# Get user
|
||||
user = self.get_user(session.user_email)
|
||||
if not user:
|
||||
logger.warning(f"User not found for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User not found"
|
||||
|
||||
if not user.is_active:
|
||||
logger.warning(f"User account disabled for session {token_prefix}...: {session.user_email}")
|
||||
return False, None, "User account is disabled"
|
||||
|
||||
logger.debug(f"Session valid for {token_prefix}... (user: {user.email}, credits: {user.credits})")
|
||||
return True, user, "Valid session"
|
||||
|
||||
except Exception:
|
||||
logger.exception(f"Error validating session {token_prefix}...")
|
||||
return False, None, "An error occurred during validation"
|
||||
|
||||
def revoke_session(self, token: str) -> bool:
|
||||
"""Revoke a session (logout)."""
|
||||
try:
|
||||
doc_ref = self.db.collection(SESSIONS_COLLECTION).document(token)
|
||||
doc_ref.update({'is_active': False})
|
||||
logger.info("Session revoked")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error revoking session")
|
||||
return False
|
||||
|
||||
def revoke_all_sessions(self, user_email: str) -> int:
|
||||
"""Revoke all sessions for a user."""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
).where(
|
||||
filter=FieldFilter('is_active', '==', True)
|
||||
)
|
||||
|
||||
count = 0
|
||||
for doc in query.stream():
|
||||
doc.reference.update({'is_active': False})
|
||||
count += 1
|
||||
|
||||
logger.info(f"Revoked {count} sessions for {user_email}")
|
||||
return count
|
||||
except Exception:
|
||||
logger.exception("Error revoking sessions")
|
||||
return 0
|
||||
|
||||
def list_sessions_for_user(
|
||||
self,
|
||||
user_email: str,
|
||||
include_revoked: bool = False,
|
||||
limit: int = 50
|
||||
) -> List[Session]:
|
||||
"""
|
||||
List all sessions for a user.
|
||||
|
||||
Args:
|
||||
user_email: User's email address
|
||||
include_revoked: If True, include revoked/inactive sessions
|
||||
limit: Maximum number of sessions to return
|
||||
|
||||
Returns:
|
||||
List of Session objects, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(SESSIONS_COLLECTION).where(
|
||||
filter=FieldFilter('user_email', '==', user_email.lower())
|
||||
)
|
||||
|
||||
if not include_revoked:
|
||||
query = query.where(filter=FieldFilter('is_active', '==', True))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING)
|
||||
query = query.limit(limit)
|
||||
|
||||
sessions = []
|
||||
for doc in query.stream():
|
||||
try:
|
||||
sessions.append(Session(**doc.to_dict()))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to parse session document: {e}")
|
||||
|
||||
logger.debug(f"Found {len(sessions)} sessions for {user_email}")
|
||||
return sessions
|
||||
except Exception:
|
||||
logger.exception(f"Error listing sessions for {user_email}")
|
||||
return []
|
||||
|
||||
# =========================================================================
|
||||
# Credit Operations
|
||||
# =========================================================================
|
||||
|
||||
def add_credits(
|
||||
self,
|
||||
email: str,
|
||||
amount: int,
|
||||
reason: str,
|
||||
job_id: Optional[str] = None,
|
||||
stripe_session_id: Optional[str] = None,
|
||||
admin_email: Optional[str] = None
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Add credits to a user account.
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
try:
|
||||
# For Stripe sessions, atomically check and mark as processed
|
||||
# Uses create() which fails if document exists - ensures idempotency
|
||||
if stripe_session_id:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
try:
|
||||
doc_ref.create({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email.lower(),
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Document already exists - this session was already processed
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed (idempotent skip)")
|
||||
return False, 0, "Session already processed"
|
||||
|
||||
email = email.lower()
|
||||
user = self.get_or_create_user(email)
|
||||
|
||||
# Create transaction record
|
||||
transaction = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=amount,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
stripe_session_id=stripe_session_id,
|
||||
created_by=admin_email,
|
||||
)
|
||||
|
||||
# Add to transaction history (keep last N)
|
||||
transactions = user.credit_transactions[-MAX_CREDIT_TRANSACTIONS + 1:]
|
||||
transactions.append(transaction)
|
||||
|
||||
# Update user
|
||||
new_balance = user.credits + amount
|
||||
self.update_user(
|
||||
email,
|
||||
credits=new_balance,
|
||||
credit_transactions=[t.model_dump(mode='json') for t in transactions]
|
||||
)
|
||||
|
||||
logger.info(f"Added {amount} credits to {email} ({reason}). New balance: {new_balance}")
|
||||
return True, new_balance, f"Added {amount} credits"
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error adding credits to {email}")
|
||||
return False, 0, f"Failed to add credits: {e}"
|
||||
|
||||
def deduct_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_creation"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Deduct one credit from a user account.
|
||||
|
||||
Uses Firestore transaction to prevent race conditions.
|
||||
|
||||
Returns:
|
||||
(success, remaining_credits, message)
|
||||
"""
|
||||
try:
|
||||
email = email.lower()
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email)
|
||||
|
||||
@firestore.transactional
|
||||
def deduct_in_transaction(transaction):
|
||||
"""Atomically check and deduct credit."""
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
return False, 0, "User not found"
|
||||
|
||||
user_data = doc.to_dict()
|
||||
current_credits = user_data.get('credits', 0)
|
||||
|
||||
if current_credits <= 0:
|
||||
return False, 0, "Insufficient credits"
|
||||
|
||||
# Create transaction record
|
||||
credit_txn = CreditTransaction(
|
||||
id=str(uuid.uuid4()),
|
||||
amount=-1,
|
||||
reason=reason,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
# Get existing transactions and add new one
|
||||
existing_transactions = user_data.get('credit_transactions', [])
|
||||
# Keep last N-1 transactions to make room for new one
|
||||
transactions = existing_transactions[-(MAX_CREDIT_TRANSACTIONS - 1):]
|
||||
transactions.append(credit_txn.model_dump(mode='json'))
|
||||
|
||||
# Calculate new values
|
||||
new_balance = current_credits - 1
|
||||
total_jobs = user_data.get('total_jobs_created', 0) + 1
|
||||
|
||||
# Update atomically within transaction
|
||||
transaction.update(doc_ref, {
|
||||
'credits': new_balance,
|
||||
'credit_transactions': transactions,
|
||||
'total_jobs_created': total_jobs,
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
|
||||
return True, new_balance, f"Credit deducted. {new_balance} remaining"
|
||||
|
||||
# Execute the transaction
|
||||
fs_transaction = self.db.transaction()
|
||||
success, new_balance, message = deduct_in_transaction(fs_transaction)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted 1 credit from {email} for job {job_id}. Remaining: {new_balance}")
|
||||
|
||||
return success, new_balance, message
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Error deducting credit from {email}")
|
||||
return False, 0, f"Failed to deduct credit: {e}"
|
||||
|
||||
def refund_credit(
|
||||
self,
|
||||
email: str,
|
||||
job_id: str,
|
||||
reason: str = "job_failed"
|
||||
) -> Tuple[bool, int, str]:
|
||||
"""
|
||||
Refund one credit to a user account (e.g., after job failure).
|
||||
|
||||
Returns:
|
||||
(success, new_balance, message)
|
||||
"""
|
||||
return self.add_credits(email, 1, reason, job_id=job_id)
|
||||
|
||||
def check_credits(self, email: str) -> int:
|
||||
"""Check user's credit balance."""
|
||||
user = self.get_user(email)
|
||||
return user.credits if user else 0
|
||||
|
||||
def has_credits(self, email: str) -> bool:
|
||||
"""Check if user has at least one credit."""
|
||||
return self.check_credits(email) > 0
|
||||
|
||||
def is_stripe_session_processed(self, stripe_session_id: str) -> bool:
|
||||
"""
|
||||
Check if a Stripe session ID has already been processed.
|
||||
|
||||
Used to ensure idempotency of webhook processing - prevents
|
||||
duplicate credit additions if Stripe sends the same webhook twice.
|
||||
|
||||
Uses a dedicated collection for O(1) lookup instead of scanning all users.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
|
||||
Returns:
|
||||
True if this session was already processed
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
logger.info(f"Stripe session {stripe_session_id} already processed")
|
||||
return True
|
||||
return False
|
||||
except Exception:
|
||||
logger.exception(f"Error checking if stripe session {stripe_session_id} was processed")
|
||||
# On error, return False to allow processing (better to risk duplicate than block)
|
||||
return False
|
||||
|
||||
def _mark_stripe_session_processed(self, stripe_session_id: str, email: str, amount: int) -> None:
|
||||
"""
|
||||
Mark a Stripe session as processed.
|
||||
|
||||
Called after successfully adding credits from a Stripe webhook.
|
||||
|
||||
Args:
|
||||
stripe_session_id: The Stripe checkout session ID
|
||||
email: User email who received the credits
|
||||
amount: Number of credits added
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(PROCESSED_STRIPE_SESSIONS_COLLECTION).document(stripe_session_id)
|
||||
doc_ref.set({
|
||||
'stripe_session_id': stripe_session_id,
|
||||
'email': email,
|
||||
'amount': amount,
|
||||
'processed_at': datetime.utcnow()
|
||||
})
|
||||
except Exception:
|
||||
# Log but don't fail - the credits were already added
|
||||
logger.exception(f"Error marking stripe session {stripe_session_id} as processed")
|
||||
|
||||
# =========================================================================
|
||||
# Admin Operations
|
||||
# =========================================================================
|
||||
|
||||
def set_user_role(self, email: str, role: UserRole, admin_email: str) -> bool:
|
||||
"""Set a user's role (admin only)."""
|
||||
try:
|
||||
user = self.get_user(email)
|
||||
if not user:
|
||||
return False
|
||||
|
||||
self.update_user(email, role=role.value)
|
||||
logger.info(f"Admin {admin_email} set role for {email} to {role.value}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error setting user role")
|
||||
return False
|
||||
|
||||
def disable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Disable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=False)
|
||||
self.revoke_all_sessions(email)
|
||||
logger.info(f"Admin {admin_email} disabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error disabling user")
|
||||
return False
|
||||
|
||||
def enable_user(self, email: str, admin_email: str) -> bool:
|
||||
"""Enable a user account (admin only)."""
|
||||
try:
|
||||
self.update_user(email, is_active=True)
|
||||
logger.info(f"Admin {admin_email} enabled user {email}")
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception("Error enabling user")
|
||||
return False
|
||||
|
||||
def increment_jobs_completed(self, email: str) -> bool:
|
||||
"""Increment the completed jobs counter for a user using atomic increment."""
|
||||
try:
|
||||
# Check user exists first
|
||||
if not self.get_user(email):
|
||||
return False
|
||||
|
||||
# Use atomic increment to prevent race conditions
|
||||
doc_ref = self.db.collection(USERS_COLLECTION).document(email.lower())
|
||||
doc_ref.update({
|
||||
'total_jobs_completed': Increment(1),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
return True
|
||||
except Exception:
|
||||
logger.exception(f"Error incrementing jobs completed for {email}")
|
||||
return False
|
||||
|
||||
|
||||
# Global instance
|
||||
_user_service = None
|
||||
|
||||
|
||||
def get_user_service() -> UserService:
|
||||
"""Get the global user service instance."""
|
||||
global _user_service
|
||||
if _user_service is None:
|
||||
_user_service = UserService()
|
||||
return _user_service
|
||||
@@ -20,6 +20,7 @@ import os
|
||||
import json
|
||||
from typing import Optional
|
||||
import httpx
|
||||
from google.protobuf import duration_pb2
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.services.tracing import inject_trace_context
|
||||
@@ -36,8 +37,25 @@ WORKER_QUEUES = {
|
||||
"screens": "screens-worker-queue",
|
||||
"render-video": "render-worker-queue",
|
||||
"video": "video-worker-queue",
|
||||
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
|
||||
}
|
||||
|
||||
# Dispatch deadlines for each worker type (in seconds)
|
||||
# Cloud Tasks max is 1800s (30 min). These set how long Cloud Tasks waits
|
||||
# for the HTTP handler to respond before considering it failed.
|
||||
# Must be >= actual worker timeout + buffer for startup/upload.
|
||||
WORKER_DISPATCH_DEADLINES = {
|
||||
"audio": 1800, # 30 min - Modal separation can take 15-20 min
|
||||
"lyrics": 1500, # 25 min - TRANSCRIPTION_TIMEOUT_SECONDS is 1200s (20 min)
|
||||
"screens": 600, # 10 min - Screen generation is fast
|
||||
"render-video": 1800, # 30 min - Video encoding can be slow
|
||||
"video": 1800, # 30 min - Video encoding can be slow
|
||||
"idle-reminder": 60, # 1 min - Quick check and potential email send
|
||||
}
|
||||
|
||||
# Default delay for idle reminders (seconds)
|
||||
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
|
||||
|
||||
|
||||
class WorkerService:
|
||||
"""
|
||||
@@ -234,6 +252,9 @@ class WorkerService:
|
||||
# This allows worker spans to link back to the original request trace
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Get dispatch deadline for this worker type (how long Cloud Tasks waits for response)
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get(worker_type, 600)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
@@ -247,6 +268,10 @@ class WorkerService:
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
# Set dispatch_deadline - how long Cloud Tasks waits for the handler to respond
|
||||
# Default is 10 min, but audio separation can take 30+ min
|
||||
# Max is 1800s (30 min) for Cloud Tasks
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
@@ -254,7 +279,8 @@ class WorkerService:
|
||||
# This prevents duplicate task errors on retries
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name}"
|
||||
f"[job:{job_id}] Created Cloud Task for {worker_type} worker: {response.name} "
|
||||
f"(dispatch_deadline={dispatch_deadline_seconds}s)"
|
||||
)
|
||||
return True
|
||||
|
||||
@@ -409,6 +435,99 @@ class WorkerService:
|
||||
"""Trigger render video worker (post-review)."""
|
||||
return await self.trigger_worker("render-video", job_id)
|
||||
|
||||
async def schedule_idle_reminder(
|
||||
self,
|
||||
job_id: str,
|
||||
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
|
||||
) -> bool:
|
||||
"""
|
||||
Schedule an idle reminder check for a job.
|
||||
|
||||
The reminder task will be delivered after the specified delay.
|
||||
When the task executes, it checks if the job is still in a blocking
|
||||
state and sends a reminder email if the user hasn't taken action.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
delay_seconds: Delay before executing the check (default: 5 minutes)
|
||||
|
||||
Returns:
|
||||
True if task was scheduled successfully, False otherwise
|
||||
"""
|
||||
if not self._use_cloud_tasks:
|
||||
# In development mode, log and skip (no delayed execution support)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
|
||||
f"Would have fired in {delay_seconds}s."
|
||||
)
|
||||
return True
|
||||
|
||||
try:
|
||||
from google.cloud import tasks_v2
|
||||
from google.protobuf import timestamp_pb2
|
||||
import time
|
||||
|
||||
queue_name = WORKER_QUEUES.get("idle-reminder")
|
||||
if not queue_name:
|
||||
logger.error("Idle reminder queue not configured")
|
||||
return False
|
||||
|
||||
project = self.settings.google_cloud_project
|
||||
if not project:
|
||||
logger.error("GOOGLE_CLOUD_PROJECT not set")
|
||||
return False
|
||||
|
||||
location = self.settings.gcp_region
|
||||
|
||||
# Build queue path
|
||||
parent = self.tasks_client.queue_path(project, location, queue_name)
|
||||
|
||||
# Build headers
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
if self._admin_token:
|
||||
headers["X-Admin-Token"] = self._admin_token
|
||||
|
||||
# Inject trace context
|
||||
headers = inject_trace_context(headers)
|
||||
|
||||
# Calculate schedule time
|
||||
schedule_time = timestamp_pb2.Timestamp()
|
||||
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
|
||||
|
||||
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
|
||||
|
||||
# Build task payload
|
||||
task = {
|
||||
"http_request": {
|
||||
"http_method": tasks_v2.HttpMethod.POST,
|
||||
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
|
||||
"headers": headers,
|
||||
"body": json.dumps({"job_id": job_id}).encode(),
|
||||
"oidc_token": {
|
||||
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
|
||||
},
|
||||
},
|
||||
"schedule_time": schedule_time,
|
||||
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
|
||||
}
|
||||
|
||||
# Create task
|
||||
response = self.tasks_client.create_task(parent=parent, task=task)
|
||||
logger.info(
|
||||
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
|
||||
)
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
# Global worker service instance
|
||||
_worker_service: Optional[WorkerService] = None
|
||||
|
||||
@@ -0,0 +1,445 @@
|
||||
"""
|
||||
YouTube Upload Service.
|
||||
|
||||
Provides video upload functionality for YouTube, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Video uploads with metadata (title, description, tags)
|
||||
- Thumbnail uploads
|
||||
- Duplicate video detection and replacement
|
||||
- Authentication via pre-stored credentials or client secrets file
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
|
||||
from thefuzz import fuzz
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class YouTubeUploadService:
|
||||
"""
|
||||
Service for uploading videos to YouTube.
|
||||
|
||||
Supports two authentication modes:
|
||||
1. Pre-stored credentials (for server-side/non-interactive use)
|
||||
2. Client secrets file (for interactive CLI use with browser OAuth)
|
||||
"""
|
||||
|
||||
YOUTUBE_URL_PREFIX = "https://www.youtube.com/watch?v="
|
||||
SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
non_interactive: bool = False,
|
||||
server_side_mode: bool = False,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the YouTube upload service.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict with keys:
|
||||
- token: Access token
|
||||
- refresh_token: Refresh token
|
||||
- token_uri: Token endpoint URL
|
||||
- client_id: OAuth client ID
|
||||
- client_secret: OAuth client secret
|
||||
- scopes: List of OAuth scopes
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
(for interactive authentication)
|
||||
non_interactive: If True, skip interactive prompts
|
||||
server_side_mode: If True, use exact matching for duplicate detection
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.credentials = credentials
|
||||
self.client_secrets_file = client_secrets_file
|
||||
self.non_interactive = non_interactive
|
||||
self.server_side_mode = server_side_mode
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
self._youtube_service = None
|
||||
self._channel_id = None
|
||||
|
||||
def _authenticate(self):
|
||||
"""
|
||||
Authenticate with YouTube API and return service object.
|
||||
|
||||
Returns:
|
||||
YouTube API service object
|
||||
|
||||
Raises:
|
||||
Exception: If authentication fails or is required in non-interactive mode
|
||||
"""
|
||||
from google.auth.transport.requests import Request
|
||||
from google.oauth2.credentials import Credentials
|
||||
from googleapiclient.discovery import build
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
import pickle
|
||||
|
||||
# Check if we have pre-stored credentials (for non-interactive mode)
|
||||
if self.credentials and self.non_interactive:
|
||||
try:
|
||||
creds = Credentials(
|
||||
token=self.credentials.get('token'),
|
||||
refresh_token=self.credentials.get('refresh_token'),
|
||||
token_uri=self.credentials.get('token_uri'),
|
||||
client_id=self.credentials.get('client_id'),
|
||||
client_secret=self.credentials.get('client_secret'),
|
||||
scopes=self.credentials.get('scopes', self.SCOPES)
|
||||
)
|
||||
|
||||
# Refresh token if needed
|
||||
if creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
|
||||
youtube = build('youtube', 'v3', credentials=creds)
|
||||
self.logger.info("Successfully authenticated with YouTube using pre-stored credentials")
|
||||
return youtube
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to authenticate with pre-stored credentials: {str(e)}")
|
||||
# Fall through to original authentication if pre-stored credentials fail
|
||||
|
||||
# For non-interactive mode without pre-stored credentials, raise error
|
||||
if self.non_interactive:
|
||||
raise Exception(
|
||||
"YouTube authentication required but running in non-interactive mode. "
|
||||
"Please pre-authenticate or disable YouTube upload."
|
||||
)
|
||||
|
||||
# Interactive authentication using client secrets file
|
||||
if not self.client_secrets_file:
|
||||
raise Exception("No YouTube credentials or client secrets file provided")
|
||||
|
||||
# Token file stores the user's access and refresh tokens
|
||||
youtube_token_file = "/tmp/karaoke-finalise-youtube-token.pickle"
|
||||
|
||||
creds = None
|
||||
|
||||
# Check if we have saved credentials
|
||||
if os.path.exists(youtube_token_file):
|
||||
with open(youtube_token_file, "rb") as token:
|
||||
creds = pickle.load(token)
|
||||
|
||||
# If there are no valid credentials, let the user log in
|
||||
if not creds or not creds.valid:
|
||||
if creds and creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
else:
|
||||
flow = InstalledAppFlow.from_client_secrets_file(
|
||||
self.client_secrets_file, scopes=self.SCOPES
|
||||
)
|
||||
creds = flow.run_local_server(port=0)
|
||||
|
||||
# Save the credentials for the next run
|
||||
with open(youtube_token_file, "wb") as token:
|
||||
pickle.dump(creds, token)
|
||||
|
||||
return build("youtube", "v3", credentials=creds)
|
||||
|
||||
@property
|
||||
def youtube_service(self):
|
||||
"""Lazy-load YouTube service on first access."""
|
||||
if self._youtube_service is None:
|
||||
self._youtube_service = self._authenticate()
|
||||
return self._youtube_service
|
||||
|
||||
def get_channel_id(self) -> Optional[str]:
|
||||
"""
|
||||
Get the authenticated user's YouTube channel ID.
|
||||
|
||||
Returns:
|
||||
Channel ID string, or None if not found
|
||||
"""
|
||||
if self._channel_id:
|
||||
return self._channel_id
|
||||
|
||||
request = self.youtube_service.channels().list(part="snippet", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
if "items" in response and len(response["items"]) > 0:
|
||||
self._channel_id = response["items"][0]["id"]
|
||||
return self._channel_id
|
||||
|
||||
return None
|
||||
|
||||
def check_duplicate(
|
||||
self,
|
||||
title: str,
|
||||
exact_match: Optional[bool] = None
|
||||
) -> Tuple[bool, Optional[str], Optional[str]]:
|
||||
"""
|
||||
Check if a video with the given title already exists on the channel.
|
||||
|
||||
Args:
|
||||
title: Video title to search for
|
||||
exact_match: If True, require exact title match. If None, uses
|
||||
server_side_mode setting (exact in server mode, fuzzy in CLI)
|
||||
|
||||
Returns:
|
||||
Tuple of (exists: bool, video_id: Optional[str], video_url: Optional[str])
|
||||
"""
|
||||
channel_id = self.get_channel_id()
|
||||
if not channel_id:
|
||||
self.logger.warning("Could not get channel ID, skipping duplicate check")
|
||||
return False, None, None
|
||||
|
||||
use_exact_match = exact_match if exact_match is not None else self.server_side_mode
|
||||
|
||||
self.logger.info(f"Searching YouTube channel {channel_id} for title: {title}")
|
||||
request = self.youtube_service.search().list(
|
||||
part="snippet",
|
||||
channelId=channel_id,
|
||||
q=title,
|
||||
type="video",
|
||||
maxResults=10
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
if "items" not in response or len(response["items"]) == 0:
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
for item in response["items"]:
|
||||
# Verify the video actually belongs to our channel
|
||||
result_channel_id = item["snippet"]["channelId"]
|
||||
if result_channel_id != channel_id:
|
||||
self.logger.debug(
|
||||
f"Skipping video from different channel: {item['snippet']['title']} "
|
||||
f"(channel: {result_channel_id})"
|
||||
)
|
||||
continue
|
||||
|
||||
found_title = item["snippet"]["title"]
|
||||
|
||||
# Determine if this is a match
|
||||
if use_exact_match:
|
||||
is_match = title.lower() == found_title.lower()
|
||||
similarity_score = 100 if is_match else 0
|
||||
else:
|
||||
similarity_score = fuzz.ratio(title.lower(), found_title.lower())
|
||||
is_match = similarity_score >= 70
|
||||
|
||||
if is_match:
|
||||
video_id = item["id"]["videoId"]
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(
|
||||
f"Potential match found on YouTube channel with ID: {video_id} "
|
||||
f"and title: {found_title} (similarity: {similarity_score}%)"
|
||||
)
|
||||
|
||||
# In non-interactive mode, return the match directly
|
||||
if self.non_interactive:
|
||||
self.logger.info("Non-interactive mode, found a match.")
|
||||
return True, video_id, video_url
|
||||
|
||||
# Interactive confirmation
|
||||
confirmation = input(
|
||||
f"Is '{found_title}' the video you are finalising? (y/n): "
|
||||
).strip().lower()
|
||||
if confirmation == "y":
|
||||
return True, video_id, video_url
|
||||
|
||||
self.logger.info(f"No matching video found with title: {title}")
|
||||
return False, None, None
|
||||
|
||||
def delete_video(self, video_id: str) -> bool:
|
||||
"""
|
||||
Delete a YouTube video by its ID.
|
||||
|
||||
Args:
|
||||
video_id: The YouTube video ID to delete
|
||||
|
||||
Returns:
|
||||
True if successful, False otherwise
|
||||
"""
|
||||
self.logger.info(f"Deleting YouTube video with ID: {video_id}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(f"DRY RUN: Would delete YouTube video with ID: {video_id}")
|
||||
return True
|
||||
|
||||
try:
|
||||
self.youtube_service.videos().delete(id=video_id).execute()
|
||||
self.logger.info(f"Successfully deleted YouTube video with ID: {video_id}")
|
||||
return True
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to delete YouTube video with ID {video_id}: {e}")
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def truncate_title(title: str, max_length: int = 95) -> str:
|
||||
"""
|
||||
Truncate title to the nearest whole word within max_length.
|
||||
|
||||
Args:
|
||||
title: Title to truncate
|
||||
max_length: Maximum length (default 95 for YouTube)
|
||||
|
||||
Returns:
|
||||
Truncated title with "..." if needed
|
||||
"""
|
||||
if len(title) <= max_length:
|
||||
return title
|
||||
truncated_title = title[:max_length].rsplit(" ", 1)[0]
|
||||
if len(truncated_title) < max_length:
|
||||
truncated_title += " ..."
|
||||
return truncated_title
|
||||
|
||||
def upload_video(
|
||||
self,
|
||||
video_path: str,
|
||||
title: str,
|
||||
description: str,
|
||||
thumbnail_path: Optional[str] = None,
|
||||
tags: Optional[list] = None,
|
||||
category_id: str = "10", # Music category
|
||||
privacy_status: str = "public",
|
||||
replace_existing: bool = False,
|
||||
) -> Tuple[Optional[str], Optional[str]]:
|
||||
"""
|
||||
Upload a video to YouTube with metadata and optional thumbnail.
|
||||
|
||||
Args:
|
||||
video_path: Path to the video file to upload
|
||||
title: Video title (will be truncated to 95 chars if needed)
|
||||
description: Video description
|
||||
thumbnail_path: Optional path to thumbnail image
|
||||
tags: Optional list of tags/keywords
|
||||
category_id: YouTube category ID (default "10" for Music)
|
||||
privacy_status: "public", "private", or "unlisted"
|
||||
replace_existing: If True, delete existing video with same title
|
||||
|
||||
Returns:
|
||||
Tuple of (video_id, video_url) or (None, None) if upload failed/skipped
|
||||
"""
|
||||
from googleapiclient.http import MediaFileUpload
|
||||
|
||||
# Truncate title if needed
|
||||
youtube_title = self.truncate_title(title)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube: {youtube_title}")
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would upload {video_path} to YouTube with title: {youtube_title}"
|
||||
)
|
||||
return "dry_run_video_id", f"{self.YOUTUBE_URL_PREFIX}dry_run_video_id"
|
||||
|
||||
# Check for existing video
|
||||
should_replace = True if self.server_side_mode else replace_existing
|
||||
exists, existing_id, existing_url = self.check_duplicate(youtube_title)
|
||||
|
||||
if exists:
|
||||
if should_replace:
|
||||
self.logger.info(f"Video already exists on YouTube, deleting before re-upload: {existing_url}")
|
||||
if self.delete_video(existing_id):
|
||||
self.logger.info("Successfully deleted existing video, proceeding with upload")
|
||||
else:
|
||||
self.logger.error("Failed to delete existing video, aborting upload")
|
||||
return None, None
|
||||
else:
|
||||
self.logger.warning(f"Video already exists on YouTube, skipping upload: {existing_url}")
|
||||
return existing_id, existing_url
|
||||
|
||||
# Prepare video metadata
|
||||
body = {
|
||||
"snippet": {
|
||||
"title": youtube_title,
|
||||
"description": description,
|
||||
"tags": tags or [],
|
||||
"categoryId": category_id,
|
||||
},
|
||||
"status": {"privacyStatus": privacy_status},
|
||||
}
|
||||
|
||||
# Determine MIME type based on file extension
|
||||
ext = os.path.splitext(video_path)[1].lower()
|
||||
mime_type = {
|
||||
".mkv": "video/x-matroska",
|
||||
".mp4": "video/mp4",
|
||||
".mov": "video/quicktime",
|
||||
".avi": "video/x-msvideo",
|
||||
}.get(ext, "video/*")
|
||||
|
||||
# Upload video
|
||||
self.logger.info(f"Authenticating with YouTube...")
|
||||
media_file = MediaFileUpload(video_path, mimetype=mime_type, resumable=True)
|
||||
|
||||
self.logger.info(f"Uploading video to YouTube...")
|
||||
request = self.youtube_service.videos().insert(
|
||||
part="snippet,status",
|
||||
body=body,
|
||||
media_body=media_file
|
||||
)
|
||||
response = request.execute()
|
||||
|
||||
video_id = response.get("id")
|
||||
video_url = f"{self.YOUTUBE_URL_PREFIX}{video_id}"
|
||||
self.logger.info(f"Uploaded video to YouTube: {video_url}")
|
||||
|
||||
# Upload thumbnail if provided
|
||||
if thumbnail_path and os.path.isfile(thumbnail_path):
|
||||
try:
|
||||
self.logger.info(f"Uploading thumbnail from: {thumbnail_path}")
|
||||
media_thumbnail = MediaFileUpload(thumbnail_path, mimetype="image/jpeg")
|
||||
self.youtube_service.thumbnails().set(
|
||||
videoId=video_id,
|
||||
media_body=media_thumbnail
|
||||
).execute()
|
||||
self.logger.info(f"Uploaded thumbnail for video ID {video_id}")
|
||||
except Exception as e:
|
||||
self.logger.error(f"Failed to upload thumbnail: {e}")
|
||||
self.logger.warning(
|
||||
"Video uploaded but thumbnail not set. "
|
||||
"You may need to set it manually on YouTube."
|
||||
)
|
||||
elif thumbnail_path:
|
||||
self.logger.warning(f"Thumbnail file not found, skipping: {thumbnail_path}")
|
||||
|
||||
return video_id, video_url
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_youtube_upload_service: Optional[YouTubeUploadService] = None
|
||||
|
||||
|
||||
def get_youtube_upload_service(
|
||||
credentials: Optional[Dict[str, Any]] = None,
|
||||
client_secrets_file: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> YouTubeUploadService:
|
||||
"""
|
||||
Get a YouTube upload service instance.
|
||||
|
||||
For server-side use, pass credentials from YouTubeService.
|
||||
For CLI use, pass client_secrets_file.
|
||||
|
||||
Args:
|
||||
credentials: Pre-stored OAuth credentials dict
|
||||
client_secrets_file: Path to OAuth client secrets JSON file
|
||||
**kwargs: Additional arguments passed to YouTubeUploadService
|
||||
|
||||
Returns:
|
||||
YouTubeUploadService instance
|
||||
"""
|
||||
global _youtube_upload_service
|
||||
|
||||
# Create new instance if credentials/settings changed
|
||||
if _youtube_upload_service is None or credentials or client_secrets_file:
|
||||
_youtube_upload_service = YouTubeUploadService(
|
||||
credentials=credentials,
|
||||
client_secrets_file=client_secrets_file,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _youtube_upload_service
|
||||
+23
-10
@@ -6,23 +6,29 @@ Provides common mocks and test utilities across all test modules.
|
||||
NOTE: Module-level mocks are NOT applied here to allow emulator tests to work.
|
||||
Individual tests must mock dependencies as needed.
|
||||
"""
|
||||
# IMPORTANT: Set environment variables FIRST, before ANY imports that might
|
||||
# trigger backend module loading and Settings singleton creation.
|
||||
import os
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
# This must happen before Settings is instantiated, which occurs on first import
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Set up test environment variables BEFORE importing any backend modules
|
||||
os.environ.setdefault('ADMIN_TOKENS', 'test-admin-token')
|
||||
os.environ.setdefault('GOOGLE_CLOUD_PROJECT', 'test-project')
|
||||
os.environ.setdefault('GCS_BUCKET_NAME', 'test-bucket')
|
||||
os.environ.setdefault('FIRESTORE_COLLECTION', 'jobs')
|
||||
os.environ.setdefault('ENVIRONMENT', 'test')
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, AsyncMock, patch
|
||||
from datetime import datetime, UTC
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
# Mock google.auth.default AND firestore for unit tests if not using emulator
|
||||
# This prevents DefaultCredentialsError and FirestoreClient initialization during imports
|
||||
if 'FIRESTORE_EMULATOR_HOST' not in os.environ:
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# Mock google.auth.default - prevents credential errors
|
||||
try:
|
||||
@@ -85,7 +91,14 @@ def mock_auth_dependency(request):
|
||||
if 'emulator' in test_path or 'integration' in test_path:
|
||||
yield
|
||||
return
|
||||
|
||||
|
||||
# Skip for service-only unit tests that don't need the FastAPI app
|
||||
# Also skip tests that manage their own auth mocks (e.g., push tests use AuthResult objects)
|
||||
service_only_tests = ['test_rate_limit_service', 'test_email_validation_service', 'test_rate_limits_api', 'test_gce_encoding_worker', 'test_push_routes', 'test_push_notification_service']
|
||||
if any(test_name in test_path for test_name in service_only_tests):
|
||||
yield
|
||||
return
|
||||
|
||||
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
|
||||
import os
|
||||
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
|
||||
|
||||
@@ -39,6 +39,24 @@ os.environ["ADMIN_TOKENS"] = "test-admin-token"
|
||||
|
||||
# Only import app if emulators are running
|
||||
if emulators_running():
|
||||
from unittest.mock import Mock
|
||||
|
||||
# Mock theme service BEFORE importing app to ensure all jobs get theme_id="nomad"
|
||||
_mock_theme_service = Mock()
|
||||
_mock_theme_service.get_default_theme_id.return_value = "nomad"
|
||||
_mock_theme_service.get_theme.return_value = None
|
||||
|
||||
# Apply patch at module load time (before app imports the service)
|
||||
_theme_patches = [
|
||||
patch("backend.services.theme_service.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.audio_search.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.file_upload.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.users.get_theme_service", return_value=_mock_theme_service),
|
||||
patch("backend.api.routes.jobs.get_theme_service", return_value=_mock_theme_service),
|
||||
]
|
||||
for p in _theme_patches:
|
||||
p.start()
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
from backend.main import app
|
||||
else:
|
||||
@@ -76,7 +94,10 @@ def mock_worker_service():
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def client(mock_worker_service):
|
||||
"""Create FastAPI test client with mocked workers."""
|
||||
"""Create FastAPI test client with mocked workers.
|
||||
|
||||
Theme service is mocked at module load time (see above).
|
||||
"""
|
||||
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@@ -330,7 +330,7 @@ class TestReviewEndpoints:
|
||||
|
||||
class TestRenderVideoWorker:
|
||||
"""Test render video worker endpoint."""
|
||||
|
||||
|
||||
def test_render_video_worker_endpoint_exists(self, client, auth_headers):
|
||||
"""Test that render-video worker endpoint exists."""
|
||||
# Create a job
|
||||
@@ -340,9 +340,9 @@ class TestRenderVideoWorker:
|
||||
json={"url": "https://youtube.com/watch?v=render-test"}
|
||||
)
|
||||
job_id = create_resp.json()["job_id"]
|
||||
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
|
||||
# Test render-video worker endpoint exists
|
||||
response = client.post(
|
||||
"/api/internal/workers/render-video",
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
"""
|
||||
Emulator integration tests for made-for-you order features.
|
||||
|
||||
Tests with real Firestore emulator via API endpoints to verify:
|
||||
1. Admin login token creation and verification
|
||||
2. Made-for-you job fields are properly stored
|
||||
|
||||
Run with: scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
|
||||
from .conftest import emulators_running
|
||||
|
||||
# Skip all tests if emulators not running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
|
||||
class TestAdminLoginTokenIntegration:
|
||||
"""
|
||||
Integration tests for admin login token feature via API.
|
||||
|
||||
These tests verify that admin login tokens can be verified
|
||||
through the actual API endpoint.
|
||||
"""
|
||||
|
||||
def test_verify_invalid_token_returns_401(self, client):
|
||||
"""Invalid token returns 401 unauthorized."""
|
||||
response = client.get("/api/users/auth/verify?token=invalid-token-xyz")
|
||||
assert response.status_code == 401
|
||||
|
||||
def test_verify_missing_token_returns_422(self, client):
|
||||
"""Missing token parameter returns 422 validation error."""
|
||||
response = client.get("/api/users/auth/verify")
|
||||
assert response.status_code == 422
|
||||
|
||||
def test_verify_empty_token_returns_401(self, client):
|
||||
"""Empty token returns 401."""
|
||||
response = client.get("/api/users/auth/verify?token=")
|
||||
assert response.status_code == 401
|
||||
|
||||
|
||||
class TestMadeForYouJobModel:
|
||||
"""
|
||||
Tests for made-for-you job model fields.
|
||||
|
||||
Verifies that the Job model properly supports made_for_you fields.
|
||||
"""
|
||||
|
||||
def test_job_model_has_made_for_you_fields(self):
|
||||
"""Job model includes made_for_you and customer_email fields."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime, timezone
|
||||
|
||||
# Create a made-for-you job directly
|
||||
job = Job(
|
||||
job_id="test-mfy-model",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Please make it perfect!",
|
||||
)
|
||||
|
||||
assert job.made_for_you is True
|
||||
assert job.customer_email == "customer@example.com"
|
||||
assert job.customer_notes == "Please make it perfect!"
|
||||
|
||||
def test_job_model_defaults_made_for_you_false(self):
|
||||
"""Regular jobs default made_for_you to False."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
from datetime import datetime, timezone
|
||||
|
||||
job = Job(
|
||||
job_id="test-regular-model",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Default should be False or None
|
||||
assert job.made_for_you in [False, None]
|
||||
assert job.customer_email is None
|
||||
|
||||
|
||||
class TestOwnershipTransferLogic:
|
||||
"""
|
||||
Tests for ownership transfer logic (unit-style, runs in emulator context).
|
||||
|
||||
These tests verify the conditional logic for made-for-you ownership transfer
|
||||
by importing the job_manager and testing its behavior directly.
|
||||
"""
|
||||
|
||||
def test_ownership_transfer_condition_made_for_you_true(self):
|
||||
"""
|
||||
Verify the condition: if made_for_you=True and customer_email exists,
|
||||
ownership should transfer.
|
||||
"""
|
||||
# Replicate the exact logic from _schedule_completion_email
|
||||
class MockJob:
|
||||
made_for_you = True
|
||||
customer_email = "customer@example.com"
|
||||
user_email = "admin@nomadkaraoke.com"
|
||||
job_id = "test-123"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
# The logic being tested
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is True
|
||||
assert recipient_email == "customer@example.com"
|
||||
|
||||
def test_ownership_transfer_condition_made_for_you_false(self):
|
||||
"""
|
||||
Verify: if made_for_you=False, no ownership transfer.
|
||||
"""
|
||||
class MockJob:
|
||||
made_for_you = False
|
||||
customer_email = "customer@example.com"
|
||||
user_email = "user@example.com"
|
||||
job_id = "test-456"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is False
|
||||
assert recipient_email == "user@example.com"
|
||||
|
||||
def test_ownership_transfer_condition_missing_customer_email(self):
|
||||
"""
|
||||
Verify: if made_for_you=True but no customer_email, no transfer.
|
||||
"""
|
||||
class MockJob:
|
||||
made_for_you = True
|
||||
customer_email = None
|
||||
user_email = "admin@nomadkaraoke.com"
|
||||
job_id = "test-789"
|
||||
|
||||
job = MockJob()
|
||||
|
||||
recipient_email = job.user_email
|
||||
should_transfer = False
|
||||
|
||||
if job.made_for_you and job.customer_email:
|
||||
recipient_email = job.customer_email
|
||||
should_transfer = True
|
||||
|
||||
assert should_transfer is False
|
||||
assert recipient_email == "admin@nomadkaraoke.com"
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Integration tests for worker log subcollection storage.
|
||||
|
||||
Tests the subcollection approach using real Firestore emulator to verify:
|
||||
- Logs are stored in subcollection (jobs/{job_id}/logs)
|
||||
- TTL expiry field is set correctly
|
||||
- Large log volumes work without 1MB limit issues
|
||||
- Concurrent writes are handled correctly
|
||||
- Logs can be queried efficiently
|
||||
|
||||
Run with: ./scripts/run-emulator-tests.sh
|
||||
"""
|
||||
import pytest
|
||||
import time
|
||||
import requests
|
||||
import os
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime, timezone, timedelta
|
||||
|
||||
|
||||
def emulators_running() -> bool:
|
||||
"""Check if GCP emulators are running."""
|
||||
try:
|
||||
requests.get("http://127.0.0.1:8080", timeout=1)
|
||||
return True
|
||||
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
|
||||
return False
|
||||
|
||||
|
||||
# Skip all tests in this module if emulators aren't running
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not emulators_running(),
|
||||
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
|
||||
)
|
||||
|
||||
# Set up environment for emulator
|
||||
os.environ["FIRESTORE_EMULATOR_HOST"] = "127.0.0.1:8080"
|
||||
os.environ["GOOGLE_CLOUD_PROJECT"] = "test-project"
|
||||
|
||||
|
||||
class TestWorkerLogsSubcollectionDirect:
|
||||
"""Direct Firestore tests for worker logs subcollection."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-subcollection-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self):
|
||||
"""Create a test job document."""
|
||||
job_id = f"test-sub-{int(time.time() * 1000)}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": [] # Legacy field - kept empty for new jobs
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_to_subcollection(self, job_id: str, worker: str, message: str, ttl_days: int = 30):
|
||||
"""Add log to subcollection at jobs/{job_id}/logs."""
|
||||
import uuid
|
||||
log_id = str(uuid.uuid4())
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(log_id).set({
|
||||
"id": log_id,
|
||||
"job_id": job_id,
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": worker,
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=ttl_days)
|
||||
})
|
||||
return log_id
|
||||
|
||||
def _get_logs_from_subcollection(self, job_id: str, worker: str = None, limit: int = 500):
|
||||
"""Get logs from subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
query = logs_ref.order_by("timestamp")
|
||||
|
||||
if worker:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
return [doc.to_dict() for doc in query.stream()]
|
||||
|
||||
def _count_logs_in_subcollection(self, job_id: str) -> int:
|
||||
"""Count logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
def _delete_logs_subcollection(self, job_id: str, batch_size: int = 100) -> int:
|
||||
"""Delete all logs in subcollection."""
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = list(logs_ref.limit(batch_size).stream())
|
||||
if not docs:
|
||||
break
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_count += 1
|
||||
|
||||
batch.commit()
|
||||
|
||||
return deleted_count
|
||||
|
||||
def test_subcollection_single_write(self):
|
||||
"""Test writing single log to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
log_id = self._append_log_to_subcollection(job_id, "test", "Single log message")
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
assert logs[0]["message"] == "Single log message"
|
||||
assert logs[0]["worker"] == "test"
|
||||
assert logs[0]["id"] == log_id
|
||||
print("Single write to subcollection works")
|
||||
|
||||
def test_subcollection_ttl_field_is_set(self):
|
||||
"""Test TTL expiry field is set correctly."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
self._append_log_to_subcollection(job_id, "test", "Log with TTL", ttl_days=7)
|
||||
|
||||
time.sleep(0.1)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 1
|
||||
ttl_expiry = logs[0]["ttl_expiry"]
|
||||
assert ttl_expiry is not None
|
||||
|
||||
# TTL should be ~7 days from now
|
||||
expected_ttl = datetime.now(timezone.utc) + timedelta(days=7)
|
||||
if hasattr(ttl_expiry, 'timestamp'):
|
||||
# Firestore datetime object
|
||||
diff = abs((ttl_expiry.replace(tzinfo=timezone.utc) - expected_ttl).total_seconds())
|
||||
else:
|
||||
diff = 0 # If already datetime
|
||||
assert diff < 60, f"TTL expiry should be ~7 days from now, diff was {diff}s"
|
||||
print("TTL field is set correctly")
|
||||
|
||||
def test_subcollection_sequential_writes(self):
|
||||
"""Test multiple sequential writes to subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
for i in range(20):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == 20, f"Expected 20 logs, got {len(logs)}"
|
||||
print(f"Sequential writes: {len(logs)} logs created")
|
||||
|
||||
def test_subcollection_concurrent_writes(self):
|
||||
"""Test concurrent writes to subcollection - no race conditions."""
|
||||
job_id = self._create_test_job()
|
||||
num_writes = 50
|
||||
|
||||
def write_log(index):
|
||||
self._append_log_to_subcollection(job_id, "worker", f"Concurrent Log {index}")
|
||||
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = [executor.submit(write_log, i) for i in range(num_writes)]
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(0.5)
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(logs) == num_writes, f"Expected {num_writes} logs, got {len(logs)}"
|
||||
print(f"Concurrent writes: All {num_writes} logs preserved")
|
||||
|
||||
def test_subcollection_large_volume(self):
|
||||
"""Test large volume of logs - demonstrates no 1MB limit."""
|
||||
job_id = self._create_test_job()
|
||||
num_logs = 500 # Would exceed 1MB in embedded array
|
||||
|
||||
# Write logs in batches
|
||||
batch_size = 50
|
||||
for batch_start in range(0, num_logs, batch_size):
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
futures = []
|
||||
for i in range(batch_start, min(batch_start + batch_size, num_logs)):
|
||||
futures.append(executor.submit(
|
||||
self._append_log_to_subcollection,
|
||||
job_id,
|
||||
"stress",
|
||||
f"Large volume test log {i} - " + "x" * 500 # ~500 bytes per log
|
||||
))
|
||||
for future in as_completed(futures):
|
||||
future.result()
|
||||
|
||||
time.sleep(1)
|
||||
count = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert count == num_logs, f"Expected {num_logs} logs, got {count}"
|
||||
print(f"Large volume: {count} logs created (~{count * 500 / 1024:.1f}KB would exceed 1MB if embedded)")
|
||||
|
||||
def test_subcollection_filter_by_worker(self):
|
||||
"""Test filtering logs by worker type."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs from different workers
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio log {i}")
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics log {i}")
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video log {i}")
|
||||
|
||||
time.sleep(0.3)
|
||||
|
||||
# Query by worker
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
all_logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == 10, f"Expected 10 audio logs, got {len(audio_logs)}"
|
||||
assert len(lyrics_logs) == 10, f"Expected 10 lyrics logs, got {len(lyrics_logs)}"
|
||||
assert len(all_logs) == 30, f"Expected 30 total logs, got {len(all_logs)}"
|
||||
print("Worker filtering works correctly")
|
||||
|
||||
def test_subcollection_ordered_by_timestamp(self):
|
||||
"""Test logs are returned in timestamp order."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs with slight delays to ensure different timestamps
|
||||
for i in range(5):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
time.sleep(0.02)
|
||||
|
||||
logs = self._get_logs_from_subcollection(job_id)
|
||||
|
||||
# Verify order (timestamps should be ascending)
|
||||
for i in range(len(logs) - 1):
|
||||
assert logs[i]["timestamp"] <= logs[i + 1]["timestamp"], \
|
||||
f"Logs not in order at index {i}"
|
||||
print("Logs are ordered by timestamp")
|
||||
|
||||
def test_subcollection_delete_all_logs(self):
|
||||
"""Test deleting all logs in subcollection."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(25):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
count_before = self._count_logs_in_subcollection(job_id)
|
||||
assert count_before == 25
|
||||
|
||||
# Delete all logs
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
|
||||
time.sleep(0.2)
|
||||
count_after = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert deleted == 25, f"Expected to delete 25 logs, deleted {deleted}"
|
||||
assert count_after == 0, f"Expected 0 logs after delete, got {count_after}"
|
||||
print(f"Deleted {deleted} logs from subcollection")
|
||||
|
||||
def test_subcollection_concurrent_workers_interleaved(self):
|
||||
"""Test simulating audio and lyrics workers writing concurrently."""
|
||||
job_id = self._create_test_job()
|
||||
logs_per_worker = 20
|
||||
|
||||
def audio_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "audio", f"Audio processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def lyrics_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "lyrics", f"Lyrics processing step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
def video_worker():
|
||||
for i in range(logs_per_worker):
|
||||
self._append_log_to_subcollection(job_id, "video", f"Video encoding step {i}")
|
||||
time.sleep(0.005)
|
||||
|
||||
# Start all workers
|
||||
threads = [
|
||||
threading.Thread(target=audio_worker),
|
||||
threading.Thread(target=lyrics_worker),
|
||||
threading.Thread(target=video_worker)
|
||||
]
|
||||
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
audio_logs = self._get_logs_from_subcollection(job_id, worker="audio")
|
||||
lyrics_logs = self._get_logs_from_subcollection(job_id, worker="lyrics")
|
||||
video_logs = self._get_logs_from_subcollection(job_id, worker="video")
|
||||
total = self._count_logs_in_subcollection(job_id)
|
||||
|
||||
assert len(audio_logs) == logs_per_worker
|
||||
assert len(lyrics_logs) == logs_per_worker
|
||||
assert len(video_logs) == logs_per_worker
|
||||
assert total == logs_per_worker * 3
|
||||
|
||||
print(f"Concurrent workers: {len(audio_logs)} audio + {len(lyrics_logs)} lyrics + {len(video_logs)} video = {total}")
|
||||
|
||||
def test_subcollection_job_deletion_cleans_up_logs(self):
|
||||
"""Test that deleting job document doesn't orphan logs."""
|
||||
job_id = self._create_test_job()
|
||||
|
||||
# Add logs
|
||||
for i in range(10):
|
||||
self._append_log_to_subcollection(job_id, "test", f"Log {i}")
|
||||
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 10
|
||||
|
||||
# Delete logs first (must happen before parent doc deletion in real code)
|
||||
deleted = self._delete_logs_subcollection(job_id)
|
||||
assert deleted == 10
|
||||
|
||||
# Now delete job document
|
||||
self.db.collection(self.collection).document(job_id).delete()
|
||||
|
||||
# Verify subcollection is empty (Firestore doesn't cascade delete subcollections)
|
||||
time.sleep(0.2)
|
||||
assert self._count_logs_in_subcollection(job_id) == 0
|
||||
print("Job deletion with log cleanup works correctly")
|
||||
|
||||
|
||||
class TestSubcollectionVsEmbeddedArray:
|
||||
"""Compare subcollection approach with embedded array."""
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def setup_firestore(self):
|
||||
"""Set up Firestore client for each test."""
|
||||
from google.cloud import firestore
|
||||
self.db = firestore.Client(project="test-project")
|
||||
self.collection = "test-comparison-jobs"
|
||||
yield
|
||||
|
||||
def _create_test_job(self, use_array: bool):
|
||||
"""Create test job."""
|
||||
job_id = f"test-cmp-{int(time.time() * 1000)}-{'arr' if use_array else 'sub'}"
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.set({
|
||||
"job_id": job_id,
|
||||
"status": "pending",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"worker_logs": []
|
||||
})
|
||||
return job_id
|
||||
|
||||
def _append_log_array(self, job_id: str, message: str):
|
||||
"""Add log to embedded array."""
|
||||
from google.cloud import firestore
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
"worker_logs": firestore.ArrayUnion([{
|
||||
"timestamp": datetime.now(timezone.utc).isoformat() + "Z",
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message
|
||||
}])
|
||||
})
|
||||
|
||||
def _append_log_subcollection(self, job_id: str, message: str):
|
||||
"""Add log to subcollection."""
|
||||
import uuid
|
||||
now = datetime.now(timezone.utc)
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
logs_ref.document(str(uuid.uuid4())).set({
|
||||
"timestamp": now,
|
||||
"level": "INFO",
|
||||
"worker": "test",
|
||||
"message": message,
|
||||
"ttl_expiry": now + timedelta(days=30)
|
||||
})
|
||||
|
||||
def test_document_size_comparison(self):
|
||||
"""Compare document sizes between approaches."""
|
||||
array_job_id = self._create_test_job(use_array=True)
|
||||
sub_job_id = self._create_test_job(use_array=False)
|
||||
|
||||
num_logs = 100
|
||||
large_message = "x" * 5000 # 5KB per log
|
||||
|
||||
# Add logs to both
|
||||
print("\nAdding logs to both approaches...")
|
||||
for i in range(num_logs):
|
||||
self._append_log_array(array_job_id, f"Array log {i}: {large_message}")
|
||||
self._append_log_subcollection(sub_job_id, f"Sub log {i}: {large_message}")
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# Get document sizes (approximate via raw data)
|
||||
array_doc = self.db.collection(self.collection).document(array_job_id).get()
|
||||
sub_doc = self.db.collection(self.collection).document(sub_job_id).get()
|
||||
|
||||
array_data = array_doc.to_dict()
|
||||
sub_data = sub_doc.to_dict()
|
||||
|
||||
array_logs = array_data.get("worker_logs", [])
|
||||
sub_logs = sub_data.get("worker_logs", [])
|
||||
|
||||
print(f"\nEmbedded array: {len(array_logs)} logs in document")
|
||||
print(f"Subcollection: {len(sub_logs)} logs in document (0 expected)")
|
||||
|
||||
# Embedded array should have all logs in document
|
||||
assert len(array_logs) == num_logs
|
||||
|
||||
# Subcollection should have empty array in main document
|
||||
assert len(sub_logs) == 0
|
||||
|
||||
# Subcollection logs are in subcollection
|
||||
sub_count = self.db.collection(self.collection).document(sub_job_id).collection("logs").count().get()[0][0].value
|
||||
assert sub_count == num_logs
|
||||
|
||||
print(f"Subcollection logs stored separately: {sub_count}")
|
||||
print("Subcollection approach keeps main document small")
|
||||
|
||||
|
||||
print("Worker logs subcollection integration tests ready")
|
||||
@@ -0,0 +1,352 @@
|
||||
"""
|
||||
Unit tests for admin delete job outputs endpoint.
|
||||
|
||||
Tests the POST /api/admin/jobs/{job_id}/delete-outputs endpoint that allows admins
|
||||
to delete distributed outputs (YouTube, Dropbox, Google Drive) while preserving
|
||||
the job record.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
from datetime import datetime
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_complete_job():
|
||||
"""Create a mock job in COMPLETE status with distribution data."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = "complete"
|
||||
job.dropbox_path = "/Karaoke/Organized"
|
||||
job.outputs_deleted_at = None
|
||||
job.outputs_deleted_by = None
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtu.be/abc123",
|
||||
"brand_code": "NOMAD-1234",
|
||||
"dropbox_link": "https://dropbox.com/...",
|
||||
"gdrive_files": {"mp4": "file_id_1", "mp4_720p": "file_id_2"},
|
||||
}
|
||||
job.timeline = []
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_no_outputs():
|
||||
"""Create a mock job in COMPLETE status without distribution data."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-456"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = "complete"
|
||||
job.dropbox_path = None
|
||||
job.outputs_deleted_at = None
|
||||
job.outputs_deleted_by = None
|
||||
job.state_data = {}
|
||||
job.timeline = []
|
||||
return job
|
||||
|
||||
|
||||
class TestDeleteOutputsSuccess:
|
||||
"""Tests for successful output deletion."""
|
||||
|
||||
def test_delete_outputs_success(self, client, mock_complete_job):
|
||||
"""Test successfully deleting outputs from a complete job."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert "outputs_deleted_at" in data
|
||||
|
||||
def test_delete_outputs_clears_state_data_keys(self, client, mock_complete_job):
|
||||
"""Test that output-related state_data keys are listed as cleared."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Should list cleared keys
|
||||
assert "youtube_url" in data["cleared_state_data"]
|
||||
assert "brand_code" in data["cleared_state_data"]
|
||||
assert "dropbox_link" in data["cleared_state_data"]
|
||||
assert "gdrive_files" in data["cleared_state_data"]
|
||||
|
||||
def test_delete_outputs_job_without_distribution(self, client, mock_job_no_outputs):
|
||||
"""Test deleting outputs from job that has no distribution data."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_no_outputs
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Mock Firestore
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-456/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
# All services should be skipped
|
||||
assert data["deleted_services"]["youtube"]["status"] == "skipped"
|
||||
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
|
||||
assert data["deleted_services"]["gdrive"]["status"] == "skipped"
|
||||
# No keys to clear
|
||||
assert data["cleared_state_data"] == []
|
||||
|
||||
|
||||
class TestDeleteOutputsValidation:
|
||||
"""Tests for validation on delete outputs endpoint."""
|
||||
|
||||
def test_rejects_non_terminal_status(self, client, mock_complete_job):
|
||||
"""Test that jobs not in terminal states are rejected."""
|
||||
mock_complete_job.status = "encoding"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "terminal" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_awaiting_review_status(self, client, mock_complete_job):
|
||||
"""Test that AWAITING_REVIEW is rejected (not terminal)."""
|
||||
mock_complete_job.status = "awaiting_review"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_accepts_prep_complete_status(self, client, mock_complete_job):
|
||||
"""Test that prep_complete is accepted as terminal."""
|
||||
mock_complete_job.status = "prep_complete"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_failed_status(self, client, mock_complete_job):
|
||||
"""Test that failed is accepted as terminal."""
|
||||
mock_complete_job.status = "failed"
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_rejects_already_deleted(self, client, mock_complete_job):
|
||||
"""Test that already-deleted outputs cannot be deleted again."""
|
||||
mock_complete_job.outputs_deleted_at = datetime(2026, 1, 9, 12, 0, 0)
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "already deleted" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_404_for_missing_job(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post("/api/admin/jobs/nonexistent/delete-outputs")
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
|
||||
class TestDeleteOutputsServices:
|
||||
"""Tests for service deletion behavior."""
|
||||
|
||||
def test_handles_missing_youtube(self, client, mock_complete_job):
|
||||
"""Test handling when job has no YouTube URL."""
|
||||
mock_complete_job.state_data = {"brand_code": "NOMAD-1234"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["deleted_services"]["youtube"]["status"] == "skipped"
|
||||
|
||||
def test_handles_missing_dropbox_path(self, client, mock_complete_job):
|
||||
"""Test handling when job has no dropbox_path."""
|
||||
mock_complete_job.dropbox_path = None
|
||||
mock_complete_job.state_data = {"youtube_url": "https://youtu.be/abc123"}
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
|
||||
|
||||
|
||||
class TestDeleteOutputsLogging:
|
||||
"""Tests for logging on delete outputs endpoint."""
|
||||
|
||||
def test_logs_admin_action(self, client, mock_complete_job):
|
||||
"""Test that admin delete action is logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_complete_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_db = Mock()
|
||||
mock_job_ref = Mock()
|
||||
mock_db.collection.return_value.document.return_value = mock_job_ref
|
||||
mock_user_service.return_value.db = mock_db
|
||||
|
||||
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
|
||||
|
||||
assert response.status_code == 200
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin" in log_message.lower() or "deleted" in log_message.lower()
|
||||
|
||||
|
||||
class TestDeleteOutputsAuthorization:
|
||||
"""Tests for authorization on the delete outputs endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_complete_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/delete-outputs",
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -0,0 +1,411 @@
|
||||
"""
|
||||
Unit tests for admin email endpoints.
|
||||
|
||||
Tests the completion message and send email API endpoints.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
# The router already has prefix="/admin", so we add /api prefix
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
return ("admin@example.com", "admin", 1)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.audio_hash = "hash123"
|
||||
job.review_token = "review123"
|
||||
job.instrumental_token = "inst123"
|
||||
job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=test123",
|
||||
"dropbox_link": "https://dropbox.com/folder/test",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
|
||||
|
||||
def test_returns_completion_message(self, client, mock_job):
|
||||
"""Test successful completion message retrieval."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
# Setup mocks
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["message"] == "Your video is ready!"
|
||||
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
|
||||
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
|
||||
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["youtube_url"] is None
|
||||
assert data["dropbox_url"] is None
|
||||
|
||||
def test_default_subject_without_song_info(self, client, mock_job):
|
||||
"""Test default subject when no artist/title."""
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/completion-message",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["subject"] == "Your karaoke video is ready!"
|
||||
|
||||
|
||||
class TestSendCompletionEmail:
|
||||
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
|
||||
|
||||
def test_sends_email_successfully(self, client, mock_job):
|
||||
"""Test successful email sending."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Your video is ready!"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": True},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["success"] is True
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["to_email"] == "customer@example.com"
|
||||
|
||||
# Verify email was sent with correct params
|
||||
mock_es.send_job_completion.assert_called_once_with(
|
||||
to_email="customer@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
brand_code="NOMAD-1234",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
def test_sends_email_without_cc(self, client, mock_job):
|
||||
"""Test email sending without CC."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com", "cc_admin": False},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify CC was not included
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["cc_admin"] is False
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
|
||||
def test_returns_500_when_email_fails(self, client, mock_job):
|
||||
"""Test 500 when email sending fails."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = False # Email failed
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 500
|
||||
assert "Failed to send email" in response.json()["detail"]
|
||||
|
||||
def test_handles_none_state_data(self, client, mock_job):
|
||||
"""Test handling of job with None state_data."""
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "customer@example.com"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_accepts_any_email_string(self, client, mock_job):
|
||||
"""Test that any string is currently accepted as email (no validation).
|
||||
|
||||
Documents current behavior: FastAPI/Pydantic doesn't validate email
|
||||
format by default. If validation is added later, this test should
|
||||
be updated to expect 422 for invalid emails.
|
||||
"""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
|
||||
patch('backend.services.email_service.get_email_service') as mock_get_es:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.get_completion_message.return_value = "Message"
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
mock_es = Mock()
|
||||
mock_es.send_job_completion.return_value = True
|
||||
mock_get_es.return_value = mock_es
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/send-completion-email",
|
||||
json={"to_email": "invalid-email"}, # Not a valid email format
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Currently accepts any string - endpoint succeeds
|
||||
assert response.status_code == 200
|
||||
# Verify the invalid email was passed through
|
||||
mock_es.send_job_completion.assert_called_once()
|
||||
call_kwargs = mock_es.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs["to_email"] == "invalid-email"
|
||||
|
||||
|
||||
class TestIdleReminderEndpoint:
|
||||
"""Tests for the internal idle reminder endpoint."""
|
||||
|
||||
def test_sends_reminder_when_still_idle(self, mock_job):
|
||||
"""Test that reminder is sent when user is still idle."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": False,
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
|
||||
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.firestore = Mock() # Mock firestore for update_job call
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_ns = Mock()
|
||||
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
|
||||
mock_get_ns.return_value = mock_ns
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed and return "sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
|
||||
def test_skips_reminder_when_already_sent(self, mock_job):
|
||||
"""Test that reminder is skipped if already sent."""
|
||||
from backend.api.routes.internal import router as internal_router
|
||||
from backend.api.dependencies import require_admin
|
||||
|
||||
# Create test app for internal router
|
||||
# The router already has prefix="/internal", so we add /api prefix
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(internal_router, prefix="/api")
|
||||
test_app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
|
||||
mock_job.state_data = {
|
||||
"blocking_state_entered_at": "2024-01-01T00:00:00",
|
||||
"blocking_action_type": "lyrics",
|
||||
"reminder_sent": True, # Already sent
|
||||
}
|
||||
|
||||
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
test_client = TestClient(test_app)
|
||||
response = test_client.post(
|
||||
"/api/internal/jobs/test-job-123/check-idle-reminder",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
# Should succeed with "already_sent" status
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "already_sent"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
@@ -0,0 +1,337 @@
|
||||
"""
|
||||
Unit tests for admin job files endpoint.
|
||||
|
||||
Tests the GET /api/admin/jobs/{job_id}/files endpoint that returns
|
||||
all files for a job with signed download URLs.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_with_files():
|
||||
"""Create a mock job with comprehensive file_urls."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Song"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/jobs/test-job-123/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test-job-123/stems/instrumental_clean.flac",
|
||||
"vocals": "gs://bucket/jobs/test-job-123/stems/vocals.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/test-job-123/lyrics/corrections.json",
|
||||
"lrc": "gs://bucket/jobs/test-job-123/lyrics/output.lrc",
|
||||
},
|
||||
"finals": {
|
||||
"lossy_720p_mp4": "gs://bucket/jobs/test-job-123/finals/video_720p.mp4",
|
||||
},
|
||||
"youtube": {
|
||||
"url": "https://youtube.com/watch?v=xyz789",
|
||||
"video_id": "xyz789",
|
||||
},
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_no_files():
|
||||
"""Create a mock job with empty file_urls."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-empty"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Empty Artist"
|
||||
job.title = "Empty Song"
|
||||
job.status = JobStatus.PENDING
|
||||
job.file_urls = {}
|
||||
return job
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job_partial_files():
|
||||
"""Create a mock job with only some files (partial processing)."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-partial"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Partial Artist"
|
||||
job.title = "Partial Song"
|
||||
job.status = JobStatus.SEPARATING_STAGE1
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/jobs/test-job-partial/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/jobs/test-job-partial/stems/instrumental_clean.flac",
|
||||
},
|
||||
}
|
||||
return job
|
||||
|
||||
|
||||
class TestGetJobFiles:
|
||||
"""Tests for GET /api/admin/jobs/{job_id}/files endpoint."""
|
||||
|
||||
def test_returns_files_with_signed_urls(self, client, mock_job_with_files):
|
||||
"""Test that endpoint returns all files with signed URLs."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
# Setup JobManager mock
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
# Setup StorageService mock to return predictable signed URLs
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/files",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["artist"] == "Test Artist"
|
||||
assert data["title"] == "Test Song"
|
||||
assert "files" in data
|
||||
assert data["total_files"] > 0
|
||||
|
||||
def test_returns_correct_file_count(self, client, mock_job_with_files):
|
||||
"""Test that file count matches actual GCS files (excluding non-GCS entries)."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
# Should have: input, stems.instrumental_clean, stems.vocals,
|
||||
# lyrics.corrections, lyrics.lrc, finals.lossy_720p_mp4
|
||||
# NOT youtube.url or youtube.video_id (not GCS paths)
|
||||
assert data["total_files"] == 6
|
||||
|
||||
def test_handles_nested_file_structure(self, client, mock_job_with_files):
|
||||
"""Test that nested file_urls structure is properly traversed."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Check that files from different categories are present
|
||||
categories = {f["category"] for f in files}
|
||||
assert "input" in categories or any(f["category"] == "" and f["file_key"] == "input" for f in files)
|
||||
assert "stems" in categories
|
||||
assert "lyrics" in categories
|
||||
assert "finals" in categories
|
||||
|
||||
def test_file_info_structure(self, client, mock_job_with_files):
|
||||
"""Test that each file has correct info structure."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Every file should have these fields
|
||||
for file_info in files:
|
||||
assert "name" in file_info
|
||||
assert "path" in file_info
|
||||
assert "download_url" in file_info
|
||||
assert "category" in file_info
|
||||
assert "file_key" in file_info
|
||||
# download_url should be a signed URL
|
||||
assert file_info["download_url"].startswith("https://signed.url/")
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.get(
|
||||
"/api/admin/jobs/nonexistent-job/files",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_empty_files_for_new_job(self, client, mock_job_no_files):
|
||||
"""Test that new jobs with no files return empty list."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_no_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-empty/files")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["job_id"] == "test-job-empty"
|
||||
assert data["files"] == []
|
||||
assert data["total_files"] == 0
|
||||
|
||||
def test_handles_partial_processing(self, client, mock_job_partial_files):
|
||||
"""Test job with only some files processed."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_partial_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-partial/files")
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Should have input + stems.instrumental_clean = 2 files
|
||||
assert data["total_files"] == 2
|
||||
|
||||
def test_skips_non_gcs_urls(self, client, mock_job_with_files):
|
||||
"""Test that non-GCS URLs (like youtube URLs) are skipped."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
response = client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
data = response.json()
|
||||
files = data["files"]
|
||||
|
||||
# Ensure no youtube URLs or video IDs are in the files
|
||||
for file_info in files:
|
||||
assert "youtube.com" not in file_info["path"]
|
||||
assert file_info["path"].startswith("gs://")
|
||||
|
||||
def test_signed_url_expiration(self, client, mock_job_with_files):
|
||||
"""Test that signed URLs are generated with appropriate expiration."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
|
||||
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job_with_files
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
mock_storage = Mock()
|
||||
mock_storage.generate_signed_url.return_value = "https://signed.url/test"
|
||||
mock_storage_class.return_value = mock_storage
|
||||
|
||||
client.get("/api/admin/jobs/test-job-123/files")
|
||||
|
||||
# Verify signed URLs were requested with expiration (default 120 minutes)
|
||||
assert mock_storage.generate_signed_url.called
|
||||
# Check that expiration_minutes was passed (could be any reasonable value)
|
||||
call_kwargs = mock_storage.generate_signed_url.call_args_list[0][1]
|
||||
if "expiration_minutes" in call_kwargs:
|
||||
assert call_kwargs["expiration_minutes"] >= 60
|
||||
|
||||
|
||||
class TestGetJobFilesAuthorization:
|
||||
"""Tests for authorization on the files endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job_with_files):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
# Reset the dependency override to test auth
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.get(
|
||||
"/api/admin/jobs/test-job-123/files",
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
# Restore the original override
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -0,0 +1,384 @@
|
||||
"""
|
||||
Unit tests for admin job reset endpoint.
|
||||
|
||||
Tests the POST /api/admin/jobs/{job_id}/reset endpoint that allows admins
|
||||
to reset a job to a specific state for re-processing.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
from datetime import datetime
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job in COMPLETE status for testing."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Test Artist"
|
||||
job.title = "Test Title"
|
||||
job.status = JobStatus.COMPLETE
|
||||
job.theme_id = "nomad"
|
||||
job.file_urls = {
|
||||
"input": "gs://bucket/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://bucket/stems/clean.flac",
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/lyrics/corrections.json",
|
||||
},
|
||||
}
|
||||
job.state_data = {
|
||||
"audio_search_results": [{"provider": "test"}],
|
||||
"audio_selection": {"index": 0},
|
||||
"review_complete": True,
|
||||
"corrected_lyrics": {"lines": []},
|
||||
"instrumental_selection": "clean",
|
||||
"brand_code": "NOMAD-1234",
|
||||
}
|
||||
job.timeline = []
|
||||
job.created_at = datetime(2026, 1, 9, 10, 0, 0)
|
||||
job.updated_at = datetime(2026, 1, 9, 12, 0, 0)
|
||||
return job
|
||||
|
||||
|
||||
class TestResetJobToPending:
|
||||
"""Tests for resetting job to PENDING state."""
|
||||
|
||||
def test_reset_to_pending_success(self, client, mock_job):
|
||||
"""Test resetting a job to PENDING state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert data["new_status"] == "pending"
|
||||
assert "pending" in data["message"].lower()
|
||||
|
||||
def test_reset_to_pending_clears_state_data(self, client, mock_job):
|
||||
"""Test that resetting to PENDING clears appropriate state_data keys."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify update_job was called with cleared state_data keys
|
||||
call_args = mock_jm.update_job.call_args
|
||||
update_data = call_args[0][1]
|
||||
# Verify status was set
|
||||
assert update_data["status"] == "pending"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingAudioSelection:
|
||||
"""Tests for resetting job to AWAITING_AUDIO_SELECTION state."""
|
||||
|
||||
def test_reset_to_awaiting_audio_selection(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_AUDIO_SELECTION state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_audio_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_audio_selection"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingReview:
|
||||
"""Tests for resetting job to AWAITING_REVIEW state."""
|
||||
|
||||
def test_reset_to_awaiting_review(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_REVIEW state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_review"
|
||||
|
||||
|
||||
class TestResetJobToAwaitingInstrumentalSelection:
|
||||
"""Tests for resetting job to AWAITING_INSTRUMENTAL_SELECTION state."""
|
||||
|
||||
def test_reset_to_awaiting_instrumental_selection(self, client, mock_job):
|
||||
"""Test resetting a job to AWAITING_INSTRUMENTAL_SELECTION state."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_instrumental_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["new_status"] == "awaiting_instrumental_selection"
|
||||
|
||||
|
||||
class TestResetJobValidation:
|
||||
"""Tests for reset endpoint validation."""
|
||||
|
||||
def test_rejects_invalid_target_state(self, client, mock_job):
|
||||
"""Test that invalid target states are rejected."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "encoding"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "invalid" in response.json()["detail"].lower() or "not allowed" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_complete_as_target(self, client, mock_job):
|
||||
"""Test that COMPLETE is not a valid reset target."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "complete"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_rejects_failed_as_target(self, client, mock_job):
|
||||
"""Test that FAILED is not a valid reset target."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "failed"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/nonexistent-job/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_missing_target_state(self, client, mock_job):
|
||||
"""Test that target_state is required."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={},
|
||||
)
|
||||
|
||||
assert response.status_code == 422 # Validation error
|
||||
|
||||
|
||||
class TestResetJobLogging:
|
||||
"""Tests for logging on the reset endpoint."""
|
||||
|
||||
def test_logs_admin_reset_action(self, client, mock_job):
|
||||
"""Test that admin reset actions are logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify logging was called
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin" in log_message.lower() or "reset" in log_message.lower()
|
||||
|
||||
|
||||
class TestResetJobTimeline:
|
||||
"""Tests for timeline event on reset."""
|
||||
|
||||
def test_adds_timeline_event_on_reset(self, client, mock_job):
|
||||
"""Test that a timeline event is added when job is reset."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
# Timeline event should be included in response
|
||||
assert "timeline_event" in data or data.get("message")
|
||||
|
||||
|
||||
class TestResetJobAuthorization:
|
||||
"""Tests for authorization on the reset endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
class TestResetJobClearsStateData:
|
||||
"""Tests for state_data clearing based on target state."""
|
||||
|
||||
def test_reset_to_pending_clears_all_processing_data(self, client, mock_job):
|
||||
"""Test that resetting to PENDING clears audio search results."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "pending"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# The cleared_data field should list what was cleared
|
||||
data = response.json()
|
||||
assert "cleared_data" in data or "state_data" in str(data)
|
||||
|
||||
def test_reset_to_awaiting_review_preserves_audio_data(self, client, mock_job):
|
||||
"""Test that resetting to AWAITING_REVIEW preserves audio stems."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_review"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_reset_to_instrumental_preserves_review_data(self, client, mock_job):
|
||||
"""Test that resetting to AWAITING_INSTRUMENTAL_SELECTION preserves review data."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/jobs/test-job-123/reset",
|
||||
json={"target_state": "awaiting_instrumental_selection"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
@@ -0,0 +1,326 @@
|
||||
"""
|
||||
Unit tests for admin job update endpoint.
|
||||
|
||||
Tests the PATCH /api/admin/jobs/{job_id} endpoint that allows admins
|
||||
to update editable job fields.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency."""
|
||||
from backend.api.dependencies import AuthResult, UserType
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=999,
|
||||
message="Admin authenticated",
|
||||
user_email="admin@example.com",
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
|
||||
# Override the require_admin dependency
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client."""
|
||||
return TestClient(app)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_job():
|
||||
"""Create a mock job for testing."""
|
||||
job = Mock(spec=Job)
|
||||
job.job_id = "test-job-123"
|
||||
job.user_email = "user@example.com"
|
||||
job.artist = "Original Artist"
|
||||
job.title = "Original Title"
|
||||
job.status = JobStatus.AWAITING_REVIEW
|
||||
job.theme_id = "nomad"
|
||||
job.enable_cdg = False
|
||||
job.enable_txt = False
|
||||
job.enable_youtube_upload = True
|
||||
job.customer_email = None
|
||||
job.customer_notes = None
|
||||
job.brand_prefix = None
|
||||
job.non_interactive = False
|
||||
job.prep_only = False
|
||||
job.discord_webhook_url = None
|
||||
job.youtube_description = None
|
||||
job.youtube_description_template = None
|
||||
job.file_urls = {}
|
||||
job.state_data = {}
|
||||
job.created_at = "2026-01-09T10:00:00Z"
|
||||
job.updated_at = "2026-01-09T10:30:00Z"
|
||||
return job
|
||||
|
||||
|
||||
class TestUpdateJob:
|
||||
"""Tests for PATCH /api/admin/jobs/{job_id} endpoint."""
|
||||
|
||||
def test_update_artist_field(self, client, mock_job):
|
||||
"""Test updating the artist field."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["status"] == "success"
|
||||
assert data["job_id"] == "test-job-123"
|
||||
assert "artist" in data["updated_fields"]
|
||||
|
||||
# Verify update_job was called with correct args
|
||||
mock_jm.update_job.assert_called_once()
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][0] == "test-job-123"
|
||||
assert call_args[0][1]["artist"] == "New Artist"
|
||||
|
||||
def test_update_multiple_fields(self, client, mock_job):
|
||||
"""Test updating multiple fields at once."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"artist": "New Artist",
|
||||
"title": "New Title",
|
||||
"theme_id": "default",
|
||||
"enable_cdg": True,
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert len(data["updated_fields"]) == 4
|
||||
assert "artist" in data["updated_fields"]
|
||||
assert "title" in data["updated_fields"]
|
||||
assert "theme_id" in data["updated_fields"]
|
||||
assert "enable_cdg" in data["updated_fields"]
|
||||
|
||||
def test_update_user_email(self, client, mock_job):
|
||||
"""Test updating user_email field."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"user_email": "newuser@example.com"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["user_email"] == "newuser@example.com"
|
||||
|
||||
def test_update_boolean_fields(self, client, mock_job):
|
||||
"""Test updating boolean fields."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"enable_cdg": True,
|
||||
"enable_txt": True,
|
||||
"enable_youtube_upload": False,
|
||||
"non_interactive": True,
|
||||
"prep_only": True,
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["enable_cdg"] is True
|
||||
assert call_args[0][1]["enable_txt"] is True
|
||||
assert call_args[0][1]["enable_youtube_upload"] is False
|
||||
|
||||
def test_rejects_non_editable_job_id(self, client, mock_job):
|
||||
"""Test that job_id cannot be updated."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"job_id": "new-job-id"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_non_editable_created_at(self, client, mock_job):
|
||||
"""Test that created_at cannot be updated."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"created_at": "2025-01-01T00:00:00Z"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_status_update(self, client, mock_job):
|
||||
"""Test that status cannot be updated via PATCH (use reset endpoint)."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"status": "complete"},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_rejects_state_data_update(self, client, mock_job):
|
||||
"""Test that state_data cannot be updated directly."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"state_data": {"foo": "bar"}},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "not editable" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_404_when_job_not_found(self, client):
|
||||
"""Test 404 when job doesn't exist."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = None
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/nonexistent-job",
|
||||
json={"artist": "New Artist"},
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_returns_400_for_empty_update(self, client, mock_job):
|
||||
"""Test 400 when no valid fields are provided."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "no valid" in response.json()["detail"].lower()
|
||||
|
||||
def test_logs_admin_changes(self, client, mock_job):
|
||||
"""Test that admin changes are logged."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
# Verify logging was called
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "admin@example.com" in log_message.lower() or "admin" in log_message.lower()
|
||||
|
||||
def test_update_customer_fields(self, client, mock_job):
|
||||
"""Test updating made-for-you customer fields."""
|
||||
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
|
||||
mock_jm = Mock()
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_jm.update_job.return_value = True
|
||||
mock_jm_class.return_value = mock_jm
|
||||
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={
|
||||
"customer_email": "customer@example.com",
|
||||
"customer_notes": "Special request: add extra countdown",
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
call_args = mock_jm.update_job.call_args
|
||||
assert call_args[0][1]["customer_email"] == "customer@example.com"
|
||||
assert call_args[0][1]["customer_notes"] == "Special request: add extra countdown"
|
||||
|
||||
|
||||
class TestUpdateJobAuthorization:
|
||||
"""Tests for authorization on the update endpoint."""
|
||||
|
||||
def test_requires_admin_access(self, client, mock_job):
|
||||
"""Test that non-admin users cannot access the endpoint."""
|
||||
original_override = app.dependency_overrides.get(require_admin)
|
||||
|
||||
def get_non_admin():
|
||||
from fastapi import HTTPException
|
||||
raise HTTPException(status_code=403, detail="Admin access required")
|
||||
|
||||
app.dependency_overrides[require_admin] = get_non_admin
|
||||
|
||||
try:
|
||||
response = client.patch(
|
||||
"/api/admin/jobs/test-job-123",
|
||||
json={"artist": "New Artist"},
|
||||
headers={"Authorization": "Bearer user-token"}
|
||||
)
|
||||
assert response.status_code == 403
|
||||
finally:
|
||||
if original_override:
|
||||
app.dependency_overrides[require_admin] = original_override
|
||||
else:
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
@@ -460,16 +460,18 @@ class TestAudioSearchServiceDownload:
|
||||
url="https://youtube.com/watch?v=abc123",
|
||||
index=0,
|
||||
)
|
||||
|
||||
|
||||
service = AudioSearchService(red_api_key=None, red_api_url=None, ops_api_key=None, ops_api_url=None)
|
||||
service._fetcher = Mock()
|
||||
service._fetcher.search.return_value = [mock_result]
|
||||
|
||||
# Ensure no remote client so we test the local cache path
|
||||
service._remote_client = None
|
||||
|
||||
service.search("ABBA", "Waterloo")
|
||||
|
||||
|
||||
with pytest.raises(DownloadError) as exc_info:
|
||||
service.download(99, "/tmp")
|
||||
|
||||
|
||||
assert "No cached result for index 99" in str(exc_info.value)
|
||||
|
||||
|
||||
@@ -798,6 +800,258 @@ class TestRemoteDownloadPath:
|
||||
assert result.filepath == "/tmp/test.flac"
|
||||
|
||||
|
||||
class TestAudioSearchThemeSupport:
|
||||
"""Test theme support in audio search requests.
|
||||
|
||||
These tests verify that theme_id and color_overrides are properly:
|
||||
1. Accepted in AudioSearchRequest model
|
||||
2. Passed through to JobCreate
|
||||
3. Result in correct CDG/TXT defaults when theme is set
|
||||
4. Theme style is PREPARED (copied to job folder) when theme_id is set
|
||||
"""
|
||||
|
||||
def test_audio_search_request_accepts_theme_id(self):
|
||||
"""Test AudioSearchRequest model has theme_id field."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad"
|
||||
)
|
||||
|
||||
assert request.theme_id == "nomad"
|
||||
|
||||
def test_audio_search_request_accepts_color_overrides(self):
|
||||
"""Test AudioSearchRequest model has color_overrides field."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000",
|
||||
"title_color": "#00ff00",
|
||||
}
|
||||
)
|
||||
|
||||
assert request.color_overrides["artist_color"] == "#ff0000"
|
||||
assert request.color_overrides["title_color"] == "#00ff00"
|
||||
|
||||
def test_audio_search_request_theme_defaults_cdg_txt(self):
|
||||
"""Test that when theme_id is set, CDG/TXT defaults to enabled.
|
||||
|
||||
This is the key behavior: selecting a theme should automatically
|
||||
enable CDG and TXT output formats.
|
||||
"""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# When theme_id is set, enable_cdg/enable_txt should default to True
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad"
|
||||
# enable_cdg and enable_txt are None (not specified)
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is True, "CDG should be enabled by default when theme is set"
|
||||
assert resolved_txt is True, "TXT should be enabled by default when theme is set"
|
||||
|
||||
def test_audio_search_request_no_theme_no_cdg_txt(self):
|
||||
"""Test that without theme_id, CDG/TXT defaults to disabled."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
# No theme_id, no enable_cdg, no enable_txt
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is False, "CDG should be disabled by default without theme"
|
||||
assert resolved_txt is False, "TXT should be disabled by default without theme"
|
||||
|
||||
def test_explicit_cdg_txt_overrides_theme_default(self):
|
||||
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# Theme set (would default to True), but explicitly disabled
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
enable_cdg=False,
|
||||
enable_txt=False,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is False, "Explicit False should override theme default"
|
||||
assert resolved_txt is False, "Explicit False should override theme default"
|
||||
|
||||
def test_explicit_cdg_txt_enables_without_theme(self):
|
||||
"""Test that explicit True enables CDG/TXT even without theme."""
|
||||
from backend.api.routes.audio_search import AudioSearchRequest
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
|
||||
# No theme (would default to False), but explicitly enabled
|
||||
request = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
request.theme_id, request.enable_cdg, request.enable_txt
|
||||
)
|
||||
|
||||
assert resolved_cdg is True, "Explicit True should enable CDG without theme"
|
||||
assert resolved_txt is True, "Explicit True should enable TXT without theme"
|
||||
|
||||
def test_job_create_receives_theme_from_audio_search(self):
|
||||
"""Test that JobCreate model can receive theme_id and color_overrides."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000",
|
||||
"sung_lyrics_color": "#00ff00"
|
||||
},
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
assert job_create.theme_id == "nomad"
|
||||
assert job_create.color_overrides["artist_color"] == "#ff0000"
|
||||
assert job_create.color_overrides["sung_lyrics_color"] == "#00ff00"
|
||||
assert job_create.enable_cdg is True
|
||||
assert job_create.enable_txt is True
|
||||
|
||||
def test_job_model_stores_theme_configuration(self):
|
||||
"""Test Job model stores theme configuration correctly."""
|
||||
job = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.utcnow(),
|
||||
updated_at=datetime.utcnow(),
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
color_overrides={
|
||||
"artist_color": "#ff0000"
|
||||
},
|
||||
enable_cdg=True,
|
||||
enable_txt=True,
|
||||
)
|
||||
|
||||
assert job.theme_id == "nomad"
|
||||
assert job.color_overrides["artist_color"] == "#ff0000"
|
||||
assert job.enable_cdg is True
|
||||
assert job.enable_txt is True
|
||||
|
||||
|
||||
class TestAudioSearchThemePreparation:
|
||||
"""Test that audio search endpoint properly prepares theme styles.
|
||||
|
||||
CRITICAL: This test class was added to catch the bug where audio_search.py
|
||||
accepted theme_id but never called prepare_job_style(), resulting in jobs
|
||||
with theme_id set but no style_params_gcs_path or style_assets.
|
||||
|
||||
The symptom: Preview videos show black background instead of themed background.
|
||||
Job ID ffb0b8fa in production demonstrated this issue.
|
||||
"""
|
||||
|
||||
def test_audio_search_with_theme_calls_prepare_job_style(self):
|
||||
"""Test that creating a job via audio search with theme_id prepares the style.
|
||||
|
||||
CRITICAL: This test verifies that when a job is created via the audio-search
|
||||
endpoint with a theme_id (and no custom style files), the code calls
|
||||
prepare_job_style() to set:
|
||||
1. style_params_gcs_path (pointing to the copied style_params.json)
|
||||
2. style_assets (populated with asset mappings)
|
||||
|
||||
Without this, LyricsTranscriber won't have access to the theme's styles
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
# Import the endpoint module to access internal functions
|
||||
from backend.api.routes import audio_search as audio_search_module
|
||||
|
||||
# Check if there's an import of prepare_job_style from theme_service
|
||||
source_code_path = audio_search_module.__file__
|
||||
with open(source_code_path, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
# Check for either:
|
||||
# 1. Import of _prepare_theme_for_job from file_upload
|
||||
# 2. Import of prepare_job_style from theme_service
|
||||
# 3. Inline call to theme_service.prepare_job_style
|
||||
has_theme_prep_import = (
|
||||
'_prepare_theme_for_job' in source_code or
|
||||
'prepare_job_style' in source_code
|
||||
)
|
||||
|
||||
assert has_theme_prep_import, (
|
||||
"audio_search.py does not import or call prepare_job_style() or _prepare_theme_for_job(). "
|
||||
"When theme_id is provided without custom style files, the endpoint MUST call "
|
||||
"prepare_job_style() to copy the theme's style_params.json to the job folder. "
|
||||
"Without this, LyricsTranscriber won't have style configuration and preview "
|
||||
"videos will have black backgrounds instead of themed ones."
|
||||
)
|
||||
|
||||
def test_audio_search_code_handles_theme_preparation(self):
|
||||
"""Verify audio_search.py has theme preparation logic similar to file_upload.py.
|
||||
|
||||
This test compares audio_search.py with file_upload.py to ensure they have
|
||||
similar theme preparation patterns. file_upload.py correctly prepares themes,
|
||||
audio_search.py should do the same.
|
||||
"""
|
||||
from backend.api.routes import audio_search as audio_search_module
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
# Read source code of both modules
|
||||
with open(audio_search_module.__file__, 'r') as f:
|
||||
audio_search_code = f.read()
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
file_upload_code = f.read()
|
||||
|
||||
# file_upload.py has this pattern for theme preparation:
|
||||
# if body.theme_id and not has_style_params_upload:
|
||||
# style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(...)
|
||||
assert '_prepare_theme_for_job' in file_upload_code, \
|
||||
"file_upload.py should have _prepare_theme_for_job function"
|
||||
|
||||
# audio_search.py should have similar pattern
|
||||
# Look for either the function or a similar conditional check
|
||||
# STRICT CHECK: Look for actual theme preparation logic being CALLED, not just mentioned
|
||||
has_theme_preparation_call = (
|
||||
# Check for import or call of theme preparation function
|
||||
'_prepare_theme_for_job(' in audio_search_code or
|
||||
'prepare_job_style(' in audio_search_code
|
||||
)
|
||||
|
||||
assert has_theme_preparation_call, (
|
||||
"audio_search.py does not CALL theme preparation logic. "
|
||||
"It accepts theme_id but never prepares the theme style like file_upload.py does. "
|
||||
"Add a call to _prepare_theme_for_job() when theme_id is set and no style files are uploaded."
|
||||
)
|
||||
|
||||
|
||||
class TestFlacfetchIntegration:
|
||||
"""
|
||||
Integration tests that verify flacfetch imports work correctly.
|
||||
|
||||
@@ -114,9 +114,97 @@ class TestWorkerServiceBasics:
|
||||
assert 'http' in url
|
||||
|
||||
|
||||
class TestSessionTokenValidation:
|
||||
"""Tests for session token validation in AuthService.
|
||||
|
||||
PR #120: Fixed authentication to recognize session tokens from magic link auth.
|
||||
Previously, validate_token() only checked admin tokens and auth_tokens collection.
|
||||
Session tokens (created via magic link) are stored in the sessions collection,
|
||||
so they were incorrectly rejected as "Invalid token".
|
||||
"""
|
||||
|
||||
def test_validate_token_checks_session_on_fallback(self):
|
||||
"""Test that validate_token falls back to checking session tokens."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 5
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
# Clear the cached instance to get fresh state
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token-xyz")
|
||||
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 5 # User's credits
|
||||
mock_user_service.validate_session.assert_called_once_with("session-token-xyz")
|
||||
|
||||
def test_validate_token_returns_invalid_when_session_invalid(self):
|
||||
"""Test that validate_token returns invalid when session validation fails."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None # Token not in auth_tokens
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (False, None, "Invalid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("invalid-session")
|
||||
|
||||
assert is_valid is False
|
||||
assert remaining == 0
|
||||
assert "Invalid token" in msg
|
||||
|
||||
def test_validate_token_returns_zero_credits_when_user_has_none(self):
|
||||
"""Test that validate_token handles users with 0 credits correctly."""
|
||||
mock_firestore = MagicMock()
|
||||
mock_firestore.get_token.return_value = None
|
||||
|
||||
mock_user = MagicMock()
|
||||
mock_user.email = "test@example.com"
|
||||
mock_user.credits = 0
|
||||
|
||||
mock_user_service = MagicMock()
|
||||
mock_user_service.validate_session.return_value = (True, mock_user, "Valid session")
|
||||
|
||||
with patch('backend.services.auth_service.FirestoreService', return_value=mock_firestore), \
|
||||
patch('backend.services.user_service.get_user_service', return_value=mock_user_service), \
|
||||
patch.dict('os.environ', {'ADMIN_TOKENS': 'admin-token-123'}):
|
||||
import backend.services.auth_service
|
||||
backend.services.auth_service._auth_service = None
|
||||
from backend.services.auth_service import AuthService, UserType
|
||||
|
||||
service = AuthService()
|
||||
is_valid, user_type, remaining, msg = service.validate_token("session-token")
|
||||
|
||||
# User is authenticated but has no credits
|
||||
assert is_valid is True
|
||||
assert user_type == UserType.STRIPE
|
||||
assert remaining == 0
|
||||
assert "no credits" in msg.lower()
|
||||
|
||||
|
||||
class TestJobManagerBasics:
|
||||
"""Basic tests for JobManager."""
|
||||
|
||||
|
||||
def test_job_manager_can_import(self):
|
||||
"""Test JobManager can be imported."""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
"""
|
||||
Tests for DiscordNotificationService.
|
||||
|
||||
Tests cover:
|
||||
- Service initialization
|
||||
- Webhook URL validation
|
||||
- Message posting
|
||||
- Video notification posting
|
||||
- Dry run mode
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from backend.services.discord_service import (
|
||||
DiscordNotificationService,
|
||||
get_discord_notification_service,
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceInit:
|
||||
"""Test service initialization."""
|
||||
|
||||
def test_init_with_webhook_url(self):
|
||||
"""Test initialization with valid webhook URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef123456"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
assert service.webhook_url == url
|
||||
assert service.is_enabled() is True
|
||||
|
||||
def test_init_without_webhook_url(self):
|
||||
"""Test initialization without webhook URL."""
|
||||
service = DiscordNotificationService()
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run mode."""
|
||||
service = DiscordNotificationService(dry_run=True)
|
||||
assert service.dry_run is True
|
||||
|
||||
def test_init_with_invalid_webhook_url_raises(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
DiscordNotificationService(webhook_url="https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_init_with_non_discord_url_raises(self):
|
||||
"""Test that non-Discord URL raises ValueError."""
|
||||
with pytest.raises(ValueError):
|
||||
DiscordNotificationService(
|
||||
webhook_url="https://slack.com/api/webhooks/123"
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceValidation:
|
||||
"""Test webhook URL validation."""
|
||||
|
||||
def test_validate_valid_webhook_url(self):
|
||||
"""Test that valid webhook URL passes validation."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise
|
||||
service._validate_webhook_url(
|
||||
"https://discord.com/api/webhooks/123456789/abcdef"
|
||||
)
|
||||
|
||||
def test_validate_invalid_webhook_url(self):
|
||||
"""Test that invalid webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service._validate_webhook_url("https://example.com/webhook")
|
||||
assert "Invalid Discord webhook URL" in str(exc_info.value)
|
||||
|
||||
def test_validate_strips_whitespace(self):
|
||||
"""Test that whitespace is stripped from URL."""
|
||||
service = DiscordNotificationService()
|
||||
# Should not raise - whitespace is stripped
|
||||
service._validate_webhook_url(
|
||||
" https://discord.com/api/webhooks/123456789/abcdef "
|
||||
)
|
||||
|
||||
|
||||
class TestDiscordNotificationServicePostMessage:
|
||||
"""Test message posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_success(self, mock_post):
|
||||
"""Test successful message posting."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_with_custom_webhook(self, mock_post):
|
||||
"""Test message posting with custom webhook URL."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
service = DiscordNotificationService()
|
||||
custom_url = "https://discord.com/api/webhooks/987654321/fedcba"
|
||||
|
||||
result = service.post_message("Test message", webhook_url=custom_url)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once_with(
|
||||
custom_url,
|
||||
json={"content": "Test message"},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
def test_post_message_no_webhook_raises(self):
|
||||
"""Test that posting without webhook URL raises ValueError."""
|
||||
service = DiscordNotificationService()
|
||||
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
service.post_message("Test message")
|
||||
assert "No Discord webhook URL provided" in str(exc_info.value)
|
||||
|
||||
def test_post_message_dry_run(self):
|
||||
"""Test message posting in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_message("Test message")
|
||||
|
||||
assert result is True
|
||||
# In dry run mode, no actual request should be made
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_message_http_error(self, mock_post):
|
||||
"""Test handling of HTTP errors."""
|
||||
import requests as req
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status.side_effect = req.HTTPError("404 Not Found")
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
with pytest.raises(req.HTTPError):
|
||||
service.post_message("Test message")
|
||||
|
||||
|
||||
class TestDiscordNotificationServiceVideoNotification:
|
||||
"""Test video notification posting."""
|
||||
|
||||
@patch("backend.services.discord_service.requests.post")
|
||||
def test_post_video_notification_success(self, mock_post):
|
||||
"""Test successful video notification."""
|
||||
mock_response = MagicMock()
|
||||
mock_response.raise_for_status = MagicMock()
|
||||
mock_post.return_value = mock_response
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_post.assert_called_once()
|
||||
call_args = mock_post.call_args
|
||||
assert "New upload:" in call_args[1]["json"]["content"]
|
||||
assert "abc123" in call_args[1]["json"]["content"]
|
||||
|
||||
def test_post_video_notification_no_youtube_url(self):
|
||||
"""Test video notification with empty YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification("")
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_none_youtube_url(self):
|
||||
"""Test video notification with None YouTube URL."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url)
|
||||
|
||||
result = service.post_video_notification(None)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_post_video_notification_dry_run(self):
|
||||
"""Test video notification in dry run mode."""
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = DiscordNotificationService(webhook_url=url, dry_run=True)
|
||||
|
||||
result = service.post_video_notification(
|
||||
"https://www.youtube.com/watch?v=abc123"
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestGetDiscordNotificationService:
|
||||
"""Test factory function."""
|
||||
|
||||
def test_get_service_creates_instance(self):
|
||||
"""Test that factory function creates a new instance."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url)
|
||||
|
||||
assert service is not None
|
||||
assert isinstance(service, DiscordNotificationService)
|
||||
assert service.webhook_url == url
|
||||
|
||||
def test_get_service_without_webhook(self):
|
||||
"""Test factory function without webhook URL."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
service = get_discord_notification_service()
|
||||
|
||||
assert service is not None
|
||||
assert service.webhook_url is None
|
||||
assert service.is_enabled() is False
|
||||
|
||||
def test_get_service_with_dry_run(self):
|
||||
"""Test factory function with dry run mode."""
|
||||
import backend.services.discord_service as module
|
||||
module._discord_notification_service = None
|
||||
|
||||
url = "https://discord.com/api/webhooks/123456789/abcdef"
|
||||
service = get_discord_notification_service(webhook_url=url, dry_run=True)
|
||||
|
||||
assert service.dry_run is True
|
||||
@@ -321,31 +321,95 @@ class TestUploadFile:
|
||||
|
||||
class TestUploadFolder:
|
||||
"""Test upload_folder method."""
|
||||
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder with multiple files."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files
|
||||
|
||||
# Create test folder with files (no subdirs for this test)
|
||||
(tmp_path / "file1.txt").write_text("content1")
|
||||
(tmp_path / "file2.txt").write_text("content2")
|
||||
(tmp_path / "subdir").mkdir() # Should be skipped
|
||||
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
|
||||
with patch.object(service, "upload_file") as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 2 files (not subdir)
|
||||
|
||||
# Should upload 2 files
|
||||
assert mock_upload.call_count == 2
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_recursive(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading a folder recursively includes subdirectories."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create test folder with files and subdirectories
|
||||
(tmp_path / "root_file.txt").write_text("root content")
|
||||
(tmp_path / "stems").mkdir()
|
||||
(tmp_path / "stems" / "vocals.flac").write_text("vocals")
|
||||
(tmp_path / "stems" / "instrumental.flac").write_text("instrumental")
|
||||
(tmp_path / "lyrics").mkdir()
|
||||
(tmp_path / "lyrics" / "song.lrc").write_text("lyrics")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload) as mock_upload:
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Should upload 4 files (1 root + 2 stems + 1 lyrics)
|
||||
assert mock_upload.call_count == 4
|
||||
|
||||
# Check that subdirectory structure is preserved
|
||||
assert "/Uploads/folder/root_file.txt" in uploaded_files
|
||||
assert "/Uploads/folder/stems/vocals.flac" in uploaded_files
|
||||
assert "/Uploads/folder/stems/instrumental.flac" in uploaded_files
|
||||
assert "/Uploads/folder/lyrics/song.lrc" in uploaded_files
|
||||
|
||||
@patch("backend.services.dropbox_service.secretmanager.SecretManagerServiceClient")
|
||||
def test_upload_folder_deeply_nested(self, mock_sm_client_class, tmp_path):
|
||||
"""Test uploading deeply nested folder structure."""
|
||||
from backend.services.dropbox_service import DropboxService
|
||||
|
||||
mock_sm_client = Mock()
|
||||
mock_response = Mock()
|
||||
mock_response.payload.data = json.dumps({"access_token": "token"}).encode()
|
||||
mock_sm_client.access_secret_version.return_value = mock_response
|
||||
mock_sm_client_class.return_value = mock_sm_client
|
||||
|
||||
# Create deeply nested structure
|
||||
(tmp_path / "level1").mkdir()
|
||||
(tmp_path / "level1" / "level2").mkdir()
|
||||
(tmp_path / "level1" / "level2" / "deep_file.txt").write_text("deep")
|
||||
|
||||
service = DropboxService()
|
||||
|
||||
uploaded_files = []
|
||||
def capture_upload(local_path, remote_path):
|
||||
uploaded_files.append(remote_path)
|
||||
|
||||
with patch.object(service, "upload_file", side_effect=capture_upload):
|
||||
service.upload_folder(str(tmp_path), "/Uploads/folder")
|
||||
|
||||
# Check deeply nested file is uploaded with correct path
|
||||
assert "/Uploads/folder/level1/level2/deep_file.txt" in uploaded_files
|
||||
|
||||
|
||||
class TestCreateSharedLink:
|
||||
"""Test create_shared_link method."""
|
||||
|
||||
@@ -0,0 +1,725 @@
|
||||
"""
|
||||
Unit tests for email service.
|
||||
|
||||
Tests the new job completion and action reminder email methods,
|
||||
as well as CC support.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock
|
||||
|
||||
from backend.services.email_service import (
|
||||
EmailService,
|
||||
EmailProvider,
|
||||
ConsoleEmailProvider,
|
||||
SendGridEmailProvider,
|
||||
get_email_service,
|
||||
)
|
||||
|
||||
|
||||
class TestConsoleEmailProvider:
|
||||
"""Tests for console email provider."""
|
||||
|
||||
def test_send_email_basic(self):
|
||||
"""Test basic email logging."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test content</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_text_content(self):
|
||||
"""Test email with text content."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
text_content="Test plain text",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email with CC recipients."""
|
||||
provider = ConsoleEmailProvider()
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
|
||||
class TestSendGridEmailProvider:
|
||||
"""Tests for SendGrid email provider."""
|
||||
|
||||
def test_send_email_success(self):
|
||||
"""Test successful email sending via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
from_name="Test Sender",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at the sendgrid module level since it's imported inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
mock_client.send.assert_called_once()
|
||||
|
||||
def test_send_email_with_cc(self):
|
||||
"""Test email sending with CC via SendGrid."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
assert result is True
|
||||
|
||||
def test_send_email_failure_status(self):
|
||||
"""Test email sending failure due to bad status."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 400
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
def test_send_email_exception(self):
|
||||
"""Test email sending exception handling."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_sg.return_value.send.side_effect = Exception("API error")
|
||||
|
||||
result = provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test Subject",
|
||||
html_content="<p>Test</p>",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestEmailServiceJobCompletion:
|
||||
"""Tests for job completion email method."""
|
||||
|
||||
def test_send_job_completion_basic(self):
|
||||
"""Test basic job completion email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_job_completion_with_song_info(self):
|
||||
"""Test job completion email includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_job_completion_with_brand_code(self):
|
||||
"""Test job completion email includes brand code in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
brand_code="NOMAD-1178",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
|
||||
|
||||
def test_send_job_completion_default_subject(self):
|
||||
"""Test job completion email default subject without song info."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
)
|
||||
|
||||
call_args = service.provider.send_email.call_args
|
||||
subject = call_args.kwargs.get('subject') or call_args[0][1]
|
||||
assert "karaoke video is ready" in subject.lower()
|
||||
|
||||
def test_send_job_completion_with_cc(self):
|
||||
"""Test job completion email with CC to admin."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=True,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
|
||||
|
||||
def test_send_job_completion_without_cc(self):
|
||||
"""Test job completion email without CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Your video is ready!",
|
||||
cc_admin=False,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
def test_send_job_completion_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_job_completion_includes_plain_text(self):
|
||||
"""Test that plain text content is included."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
message = "Your video is ready!"
|
||||
service.send_job_completion(
|
||||
to_email="user@example.com",
|
||||
message_content=message,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('text_content') == message
|
||||
|
||||
|
||||
class TestEmailServiceActionReminder:
|
||||
"""Tests for action reminder email method."""
|
||||
|
||||
def test_send_action_reminder_lyrics(self):
|
||||
"""Test lyrics action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review your lyrics",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "lyrics" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_instrumental(self):
|
||||
"""Test instrumental action reminder email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please select instrumental",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "instrumental" in subject.lower()
|
||||
|
||||
def test_send_action_reminder_with_song_info(self):
|
||||
"""Test action reminder includes song in subject."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Test Artist" in subject
|
||||
assert "Test Song" in subject
|
||||
|
||||
def test_send_action_reminder_unknown_type(self):
|
||||
"""Test action reminder with unknown type still sends."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please take action",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Action needed" in subject
|
||||
|
||||
def test_send_action_reminder_escapes_html(self):
|
||||
"""Test that message content is HTML-escaped."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Test <script>alert('xss')</script>",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "<script>" not in html_content
|
||||
assert "<script>" in html_content
|
||||
|
||||
def test_send_action_reminder_no_cc(self):
|
||||
"""Test that action reminders don't have CC."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_action_reminder(
|
||||
to_email="user@example.com",
|
||||
message_content="Please review",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
# Action reminders should not have CC
|
||||
assert call_kwargs.get('cc_emails') is None
|
||||
|
||||
|
||||
class TestEmailServiceConfiguration:
|
||||
"""Tests for email service configuration."""
|
||||
|
||||
def test_uses_sendgrid_when_configured(self):
|
||||
"""Test that SendGrid is used when API key is set."""
|
||||
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, SendGridEmailProvider)
|
||||
|
||||
def test_uses_console_when_not_configured(self):
|
||||
"""Test that console is used when no API key."""
|
||||
with patch.dict('os.environ', {}, clear=True):
|
||||
# Remove SENDGRID_API_KEY if it exists
|
||||
import os
|
||||
original = os.environ.pop('SENDGRID_API_KEY', None)
|
||||
try:
|
||||
service = EmailService()
|
||||
assert isinstance(service.provider, ConsoleEmailProvider)
|
||||
finally:
|
||||
if original:
|
||||
os.environ['SENDGRID_API_KEY'] = original
|
||||
|
||||
def test_is_configured_true_for_sendgrid(self):
|
||||
"""Test is_configured returns True for SendGrid."""
|
||||
service = EmailService()
|
||||
service.provider = SendGridEmailProvider("key", "from@example.com")
|
||||
assert service.is_configured() is True
|
||||
|
||||
def test_is_configured_false_for_console(self):
|
||||
"""Test is_configured returns False for console."""
|
||||
service = EmailService()
|
||||
service.provider = ConsoleEmailProvider()
|
||||
assert service.is_configured() is False
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_email_service_returns_same_instance(self):
|
||||
"""Test that get_email_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.email_service as es
|
||||
es._email_service = None
|
||||
|
||||
service1 = get_email_service()
|
||||
service2 = get_email_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
|
||||
class TestCCFunctionality:
|
||||
"""Integration tests for CC functionality across providers."""
|
||||
|
||||
def test_console_provider_logs_cc(self, caplog):
|
||||
"""Test that console provider logs CC recipients."""
|
||||
import logging
|
||||
caplog.set_level(logging.INFO)
|
||||
|
||||
provider = ConsoleEmailProvider()
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc1@example.com", "cc2@example.com"],
|
||||
)
|
||||
|
||||
# Check that CC was logged
|
||||
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
|
||||
|
||||
def test_sendgrid_provider_adds_cc_recipients(self):
|
||||
"""Test that SendGrid provider adds CC recipients to message."""
|
||||
provider = SendGridEmailProvider(
|
||||
api_key="test-api-key",
|
||||
from_email="from@example.com",
|
||||
)
|
||||
|
||||
mock_response = Mock()
|
||||
mock_response.status_code = 202
|
||||
|
||||
# Patch at sendgrid module level since imports are inline
|
||||
with patch('sendgrid.SendGridAPIClient') as mock_sg:
|
||||
mock_client = Mock()
|
||||
mock_client.send.return_value = mock_response
|
||||
mock_sg.return_value = mock_client
|
||||
|
||||
# We need to capture the Mail object
|
||||
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
|
||||
mock_mail = MagicMock()
|
||||
mock_mail_class.return_value = mock_mail
|
||||
|
||||
provider.send_email(
|
||||
to_email="user@example.com",
|
||||
subject="Test",
|
||||
html_content="<p>Test</p>",
|
||||
cc_emails=["cc@example.com"],
|
||||
)
|
||||
|
||||
# Verify add_cc was called
|
||||
mock_mail.add_cc.assert_called()
|
||||
|
||||
|
||||
class TestMadeForYouOrderConfirmation:
|
||||
"""Tests for send_made_for_you_order_confirmation method."""
|
||||
|
||||
def test_send_order_confirmation_success(self):
|
||||
"""Test successful order confirmation email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-123",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
# Verify BCC is set
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
assert call_kwargs.get('bcc_emails') == ["done@nomadkaraoke.com"]
|
||||
|
||||
def test_send_order_confirmation_subject_format(self):
|
||||
"""Test order confirmation has correct subject format."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
job_id="test-job-456",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
assert "Order Confirmed" in subject
|
||||
assert "Seether" in subject
|
||||
assert "Tonight" in subject
|
||||
|
||||
def test_send_order_confirmation_with_notes(self):
|
||||
"""Test order confirmation includes notes."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-789",
|
||||
notes="Wedding anniversary!",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "Wedding anniversary!" in html_content
|
||||
|
||||
def test_send_order_confirmation_includes_delivery_promise(self):
|
||||
"""Test order confirmation mentions delivery timeframe."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content', '')
|
||||
text_content = call_kwargs.get('text_content', '')
|
||||
# Should mention delivery timeline
|
||||
assert "24" in html_content or "24" in text_content
|
||||
|
||||
def test_send_order_confirmation_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = False
|
||||
|
||||
result = service.send_made_for_you_order_confirmation(
|
||||
to_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="test-job-def",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
|
||||
class TestMadeForYouAdminNotification:
|
||||
"""Tests for send_made_for_you_admin_notification method."""
|
||||
|
||||
def test_send_admin_notification_success(self):
|
||||
"""Test successful admin notification email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
result = service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.provider.send_email.assert_called_once()
|
||||
|
||||
def test_send_admin_notification_subject_format(self):
|
||||
"""Test admin notification has correct subject format."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Seether",
|
||||
title="Tonight",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
subject = call_kwargs.get('subject')
|
||||
# Subject format: "Karaoke Order: {artist} - {title} [ID: {job_id}]"
|
||||
assert "Karaoke Order" in subject
|
||||
assert "Seether" in subject
|
||||
assert "Tonight" in subject
|
||||
|
||||
def test_send_admin_notification_includes_customer_email(self):
|
||||
"""Test admin notification includes customer email."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="vip@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "vip@example.com" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_job_link(self):
|
||||
"""Test admin notification includes job link."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="abc-def-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "abc-def-123" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_audio_count(self):
|
||||
"""Test admin notification includes audio source count."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
audio_source_count=5,
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "5" in html_content
|
||||
|
||||
def test_send_admin_notification_includes_notes(self):
|
||||
"""Test admin notification includes customer notes."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = True
|
||||
|
||||
service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
notes="Rush order please!",
|
||||
)
|
||||
|
||||
call_kwargs = service.provider.send_email.call_args.kwargs
|
||||
html_content = call_kwargs.get('html_content')
|
||||
assert "Rush order please!" in html_content
|
||||
|
||||
def test_send_admin_notification_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = EmailService()
|
||||
service.provider = Mock()
|
||||
service.provider.send_email.return_value = False
|
||||
|
||||
result = service.send_made_for_you_admin_notification(
|
||||
to_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
job_id="job-123",
|
||||
admin_login_token="test-token-abc",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
@@ -0,0 +1,298 @@
|
||||
"""
|
||||
Unit tests for EmailValidationService.
|
||||
|
||||
Tests email normalization, disposable domain detection, and blocklist management.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import Mock, MagicMock, patch
|
||||
|
||||
# Mock Google Cloud before imports
|
||||
import sys
|
||||
sys.modules['google.cloud.firestore'] = MagicMock()
|
||||
sys.modules['google.cloud.storage'] = MagicMock()
|
||||
|
||||
|
||||
class TestEmailNormalization:
|
||||
"""Test email normalization logic."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
# Default: empty blocklist config
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
# =========================================================================
|
||||
# Gmail Normalization Tests
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_gmail_removes_dots(self, email_service):
|
||||
"""Test that dots are removed from Gmail local part."""
|
||||
result = email_service.normalize_email("j.o.h.n@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_gmail_removes_plus_suffix(self, email_service):
|
||||
"""Test that +tag is removed from Gmail local part."""
|
||||
result = email_service.normalize_email("john+spam@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_gmail_removes_dots_and_plus(self, email_service):
|
||||
"""Test both dots and +tag are removed from Gmail."""
|
||||
result = email_service.normalize_email("j.o.h.n+newsletter@gmail.com")
|
||||
assert result == "john@gmail.com"
|
||||
|
||||
def test_normalize_googlemail_treated_like_gmail(self, email_service):
|
||||
"""Test that googlemail.com is normalized like gmail.com."""
|
||||
result = email_service.normalize_email("j.o.h.n+test@googlemail.com")
|
||||
assert result == "john@googlemail.com"
|
||||
|
||||
def test_normalize_gmail_to_lowercase(self, email_service):
|
||||
"""Test Gmail addresses are lowercased."""
|
||||
result = email_service.normalize_email("JOHN.DOE+Test@GMAIL.COM")
|
||||
assert result == "johndoe@gmail.com"
|
||||
|
||||
# =========================================================================
|
||||
# Non-Gmail Normalization Tests
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_non_gmail_preserves_dots(self, email_service):
|
||||
"""Test dots are preserved for non-Gmail domains."""
|
||||
result = email_service.normalize_email("j.o.h.n@example.com")
|
||||
assert result == "j.o.h.n@example.com"
|
||||
|
||||
def test_normalize_non_gmail_preserves_plus(self, email_service):
|
||||
"""Test +tag is preserved for non-Gmail domains."""
|
||||
result = email_service.normalize_email("john+tag@company.com")
|
||||
assert result == "john+tag@company.com"
|
||||
|
||||
def test_normalize_non_gmail_to_lowercase(self, email_service):
|
||||
"""Test non-Gmail addresses are lowercased."""
|
||||
result = email_service.normalize_email("John.Doe@Example.Com")
|
||||
assert result == "john.doe@example.com"
|
||||
|
||||
def test_normalize_strips_whitespace(self, email_service):
|
||||
"""Test whitespace is stripped."""
|
||||
result = email_service.normalize_email(" john@example.com ")
|
||||
assert result == "john@example.com"
|
||||
|
||||
# =========================================================================
|
||||
# Edge Cases
|
||||
# =========================================================================
|
||||
|
||||
def test_normalize_empty_string(self, email_service):
|
||||
"""Test empty string returns empty."""
|
||||
result = email_service.normalize_email("")
|
||||
assert result == ""
|
||||
|
||||
def test_normalize_none_returns_empty(self, email_service):
|
||||
"""Test None returns empty string."""
|
||||
result = email_service.normalize_email(None)
|
||||
assert result == ""
|
||||
|
||||
def test_normalize_invalid_no_at_sign(self, email_service):
|
||||
"""Test email without @ is returned as-is (lowercase)."""
|
||||
result = email_service.normalize_email("notanemail")
|
||||
assert result == "notanemail"
|
||||
|
||||
|
||||
class TestDisposableDomainDetection:
|
||||
"""Test disposable domain detection."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
# Clear any cached blocklist
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_detect_known_disposable_domain(self, email_service):
|
||||
"""Test known disposable domains are detected."""
|
||||
# These are in DEFAULT_DISPOSABLE_DOMAINS
|
||||
assert email_service.is_disposable_domain("user@tempmail.com") is True
|
||||
assert email_service.is_disposable_domain("user@mailinator.com") is True
|
||||
assert email_service.is_disposable_domain("user@guerrillamail.com") is True
|
||||
|
||||
def test_detect_legitimate_domain(self, email_service):
|
||||
"""Test legitimate domains are not flagged."""
|
||||
assert email_service.is_disposable_domain("user@gmail.com") is False
|
||||
assert email_service.is_disposable_domain("user@yahoo.com") is False
|
||||
assert email_service.is_disposable_domain("user@company.com") is False
|
||||
|
||||
def test_detect_case_insensitive(self, email_service):
|
||||
"""Test domain detection is case-insensitive."""
|
||||
assert email_service.is_disposable_domain("user@TEMPMAIL.COM") is True
|
||||
assert email_service.is_disposable_domain("user@TempMail.Com") is True
|
||||
|
||||
def test_detect_invalid_email_returns_false(self, email_service):
|
||||
"""Test invalid email returns False."""
|
||||
assert email_service.is_disposable_domain("notanemail") is False
|
||||
assert email_service.is_disposable_domain("") is False
|
||||
|
||||
|
||||
class TestBlocklistManagement:
|
||||
"""Test blocklist management functionality."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client with blocklist."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = True
|
||||
mock_doc.to_dict.return_value = {
|
||||
"disposable_domains": ["custom-temp.com"],
|
||||
"blocked_emails": ["spammer@example.com"],
|
||||
"blocked_ips": ["192.168.1.100"],
|
||||
}
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
# Clear any cached blocklist
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_is_email_blocked(self, email_service):
|
||||
"""Test blocked email detection."""
|
||||
assert email_service.is_email_blocked("spammer@example.com") is True
|
||||
assert email_service.is_email_blocked("legitimate@example.com") is False
|
||||
|
||||
def test_is_email_blocked_case_insensitive(self, email_service):
|
||||
"""Test blocked email detection is case-insensitive."""
|
||||
assert email_service.is_email_blocked("SPAMMER@example.com") is True
|
||||
assert email_service.is_email_blocked("Spammer@Example.Com") is True
|
||||
|
||||
def test_is_ip_blocked(self, email_service):
|
||||
"""Test blocked IP detection."""
|
||||
assert email_service.is_ip_blocked("192.168.1.100") is True
|
||||
assert email_service.is_ip_blocked("10.0.0.1") is False
|
||||
|
||||
def test_custom_disposable_domain_added(self, email_service):
|
||||
"""Test custom disposable domains from Firestore are included."""
|
||||
assert email_service.is_disposable_domain("user@custom-temp.com") is True
|
||||
|
||||
def test_blocklist_caching(self, email_service, mock_db):
|
||||
"""Test blocklist is cached."""
|
||||
# First call
|
||||
email_service.is_disposable_domain("user@test.com")
|
||||
# Second call should use cache
|
||||
email_service.is_disposable_domain("user@test2.com")
|
||||
|
||||
# Should only call Firestore once due to caching
|
||||
assert mock_db.collection.return_value.document.return_value.get.call_count == 1
|
||||
|
||||
|
||||
class TestBetaEnrollmentValidation:
|
||||
"""Test beta enrollment validation."""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db(self):
|
||||
"""Create a mock Firestore client."""
|
||||
mock = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = True
|
||||
mock_doc.to_dict.return_value = {
|
||||
"disposable_domains": [],
|
||||
"blocked_emails": ["blocked@example.com"],
|
||||
"blocked_ips": [],
|
||||
}
|
||||
mock.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
return mock
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self, mock_db):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
EmailValidationService._blocklist_cache = None
|
||||
EmailValidationService._blocklist_cache_time = None
|
||||
service = EmailValidationService(db=mock_db)
|
||||
return service
|
||||
|
||||
def test_validate_legitimate_email(self, email_service):
|
||||
"""Test legitimate email passes validation."""
|
||||
is_valid, error = email_service.validate_email_for_beta("user@gmail.com")
|
||||
assert is_valid is True
|
||||
assert error == ""
|
||||
|
||||
def test_validate_disposable_email_rejected(self, email_service):
|
||||
"""Test disposable email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("user@tempmail.com")
|
||||
assert is_valid is False
|
||||
assert "Disposable email" in error
|
||||
|
||||
def test_validate_blocked_email_rejected(self, email_service):
|
||||
"""Test blocked email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("blocked@example.com")
|
||||
assert is_valid is False
|
||||
assert "not allowed" in error
|
||||
|
||||
def test_validate_invalid_format_rejected(self, email_service):
|
||||
"""Test invalid email format is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("notanemail")
|
||||
assert is_valid is False
|
||||
assert "Invalid email format" in error
|
||||
|
||||
def test_validate_empty_email_rejected(self, email_service):
|
||||
"""Test empty email is rejected."""
|
||||
is_valid, error = email_service.validate_email_for_beta("")
|
||||
assert is_valid is False
|
||||
|
||||
|
||||
class TestIPHashing:
|
||||
"""Test IP address hashing."""
|
||||
|
||||
@pytest.fixture
|
||||
def email_service(self):
|
||||
"""Create EmailValidationService instance with mocks."""
|
||||
mock_db = MagicMock()
|
||||
mock_doc = Mock()
|
||||
mock_doc.exists = False
|
||||
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
|
||||
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
return EmailValidationService(db=mock_db)
|
||||
|
||||
def test_hash_ip_consistent(self, email_service):
|
||||
"""Test IP hashing produces consistent results."""
|
||||
hash1 = email_service.hash_ip("192.168.1.1")
|
||||
hash2 = email_service.hash_ip("192.168.1.1")
|
||||
assert hash1 == hash2
|
||||
|
||||
def test_hash_ip_different_for_different_ips(self, email_service):
|
||||
"""Test different IPs produce different hashes."""
|
||||
hash1 = email_service.hash_ip("192.168.1.1")
|
||||
hash2 = email_service.hash_ip("192.168.1.2")
|
||||
assert hash1 != hash2
|
||||
|
||||
def test_hash_ip_is_sha256(self, email_service):
|
||||
"""Test IP hash is SHA-256 (64 hex chars)."""
|
||||
hash_result = email_service.hash_ip("192.168.1.1")
|
||||
assert len(hash_result) == 64
|
||||
assert all(c in "0123456789abcdef" for c in hash_result)
|
||||
@@ -0,0 +1,412 @@
|
||||
"""
|
||||
Tests for Encoding Interface.
|
||||
|
||||
Tests cover:
|
||||
- EncodingInput and EncodingOutput dataclasses
|
||||
- LocalEncodingBackend implementation
|
||||
- Backend factory function
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch, AsyncMock
|
||||
|
||||
from backend.services.encoding_interface import (
|
||||
EncodingInput,
|
||||
EncodingOutput,
|
||||
EncodingBackend,
|
||||
LocalEncodingBackend,
|
||||
GCEEncodingBackend,
|
||||
get_encoding_backend,
|
||||
)
|
||||
|
||||
|
||||
class TestEncodingInput:
|
||||
"""Test EncodingInput dataclass."""
|
||||
|
||||
def test_required_fields(self):
|
||||
"""Test that required fields must be provided."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
)
|
||||
assert input_config.title_video_path == "/path/title.mov"
|
||||
assert input_config.end_video_path is None
|
||||
|
||||
def test_all_fields(self):
|
||||
"""Test all fields including optional ones."""
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/path/title.mov",
|
||||
karaoke_video_path="/path/karaoke.mov",
|
||||
instrumental_audio_path="/path/audio.flac",
|
||||
end_video_path="/path/end.mov",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
brand_code="NOMAD-1234",
|
||||
output_dir="/output",
|
||||
options={"quality": "high"}
|
||||
)
|
||||
assert input_config.artist == "Test Artist"
|
||||
assert input_config.brand_code == "NOMAD-1234"
|
||||
assert input_config.options["quality"] == "high"
|
||||
|
||||
|
||||
class TestEncodingOutput:
|
||||
"""Test EncodingOutput dataclass."""
|
||||
|
||||
def test_success_output(self):
|
||||
"""Test successful output."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path="/output/video.mp4",
|
||||
encoding_backend="local"
|
||||
)
|
||||
assert output.success is True
|
||||
assert output.error_message is None
|
||||
|
||||
def test_failure_output(self):
|
||||
"""Test failure output."""
|
||||
output = EncodingOutput(
|
||||
success=False,
|
||||
error_message="Encoding failed",
|
||||
encoding_backend="gce"
|
||||
)
|
||||
assert output.success is False
|
||||
assert output.error_message == "Encoding failed"
|
||||
|
||||
def test_output_files_dict(self):
|
||||
"""Test output_files dictionary."""
|
||||
output = EncodingOutput(
|
||||
success=True,
|
||||
output_files={
|
||||
"lossless_4k_mp4": "/output/lossless.mp4",
|
||||
"720p_mp4": "/output/720p.mp4"
|
||||
}
|
||||
)
|
||||
assert "lossless_4k_mp4" in output.output_files
|
||||
assert output.output_files["720p_mp4"] == "/output/720p.mp4"
|
||||
|
||||
|
||||
class TestLocalEncodingBackend:
|
||||
"""Test LocalEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = LocalEncodingBackend()
|
||||
assert backend.name == "local"
|
||||
|
||||
def test_init_with_dry_run(self):
|
||||
"""Test initialization with dry run."""
|
||||
backend = LocalEncodingBackend(dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_success(self, mock_run):
|
||||
"""Test availability check when FFmpeg is installed."""
|
||||
mock_run.return_value = MagicMock(returncode=0)
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("subprocess.run")
|
||||
async def test_is_available_not_found(self, mock_run):
|
||||
"""Test availability check when FFmpeg is not installed."""
|
||||
mock_run.side_effect = FileNotFoundError()
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch.object(LocalEncodingBackend, "is_available")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_get_status(self, mock_get_service, mock_is_available):
|
||||
"""Test status retrieval."""
|
||||
mock_is_available.return_value = True
|
||||
mock_service = MagicMock()
|
||||
mock_service.hwaccel_available = True
|
||||
mock_service.video_encoder = "h264_nvenc"
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
status = await backend.get_status()
|
||||
|
||||
assert status["backend"] == "local"
|
||||
assert status["available"] is True
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_success(self, mock_get_service, mock_to_thread):
|
||||
"""Test successful encoding."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=True,
|
||||
output_files={"key": "/path/file.mp4"}
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
output_dir="/output"
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "local"
|
||||
assert output.encoding_time_seconds is not None
|
||||
|
||||
@pytest.mark.asyncio
|
||||
@patch("asyncio.to_thread")
|
||||
@patch.object(LocalEncodingBackend, "_get_service")
|
||||
async def test_encode_failure(self, mock_get_service, mock_to_thread):
|
||||
"""Test encoding failure."""
|
||||
from backend.services.local_encoding_service import EncodingResult
|
||||
|
||||
mock_result = EncodingResult(
|
||||
success=False,
|
||||
output_files={},
|
||||
error="FFmpeg failed"
|
||||
)
|
||||
mock_to_thread.return_value = mock_result
|
||||
|
||||
mock_service = MagicMock()
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = LocalEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert output.error_message == "FFmpeg failed"
|
||||
|
||||
|
||||
class TestGCEEncodingBackend:
|
||||
"""Test GCEEncodingBackend implementation."""
|
||||
|
||||
def test_name(self):
|
||||
"""Test backend name."""
|
||||
backend = GCEEncodingBackend()
|
||||
assert backend.name == "gce"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_enabled(self, mock_get_service):
|
||||
"""Test availability when GCE is enabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = True
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_is_available_disabled(self, mock_get_service):
|
||||
"""Test availability when GCE is disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
available = await backend.is_available()
|
||||
|
||||
assert available is False
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_missing_gcs_paths(self, mock_get_service):
|
||||
"""Test encoding fails without GCS paths."""
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
# Missing GCS paths in options
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "gcs_path" in output.error_message.lower()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_success(self, mock_get_service):
|
||||
"""Test successful GCE encoding."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(return_value={
|
||||
"status": "complete",
|
||||
"output_files": {
|
||||
"mp4_4k_lossless": "gs://bucket/output/lossless.mp4",
|
||||
"mp4_720p": "gs://bucket/output/720p.mp4",
|
||||
}
|
||||
})
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
artist="Test Artist",
|
||||
title="Test Title",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is True
|
||||
assert output.encoding_backend == "gce"
|
||||
mock_service.encode_videos.assert_called_once()
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_failure(self, mock_get_service):
|
||||
"""Test GCE encoding failure handling."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.encode_videos = AsyncMock(side_effect=Exception("GCE worker error"))
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
assert output.success is False
|
||||
assert "GCE worker error" in output.error_message
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
@pytest.mark.asyncio
|
||||
async def test_encode_handles_list_result(self, mock_get_service):
|
||||
"""Test GCE encoding handles list response gracefully.
|
||||
|
||||
This would have caught: 'list' object has no attribute 'get' error
|
||||
when GCE worker returns a list instead of a dict.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
# Simulate GCE worker returning a list instead of dict
|
||||
mock_service.encode_videos = AsyncMock(return_value=[
|
||||
{"output_files": {"mp4_4k_lossless": "gs://bucket/output/lossless.mp4"}}
|
||||
])
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = GCEEncodingBackend()
|
||||
input_config = EncodingInput(
|
||||
title_video_path="/input/title.mov",
|
||||
karaoke_video_path="/input/karaoke.mov",
|
||||
instrumental_audio_path="/input/audio.flac",
|
||||
options={
|
||||
"job_id": "test-job",
|
||||
"input_gcs_path": "gs://bucket/input/",
|
||||
"output_gcs_path": "gs://bucket/output/",
|
||||
}
|
||||
)
|
||||
|
||||
# This should not raise an error
|
||||
output = await backend.encode(input_config)
|
||||
|
||||
# Should still succeed by extracting from the list
|
||||
assert output.success is True
|
||||
assert output.lossless_4k_mp4_path == "gs://bucket/output/lossless.mp4"
|
||||
|
||||
|
||||
class TestGetEncodingBackend:
|
||||
"""Test encoding backend factory function."""
|
||||
|
||||
def test_get_local_backend(self):
|
||||
"""Test getting local backend."""
|
||||
backend = get_encoding_backend("local")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
assert backend.name == "local"
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_get_auto_backend_gce_disabled(self, mock_get_service):
|
||||
"""Test getting auto backend falls back to local when GCE disabled."""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
backend = get_encoding_backend("auto")
|
||||
assert isinstance(backend, LocalEncodingBackend)
|
||||
|
||||
def test_get_local_backend_with_options(self):
|
||||
"""Test getting local backend with options."""
|
||||
backend = get_encoding_backend("local", dry_run=True)
|
||||
assert backend.dry_run is True
|
||||
|
||||
def test_get_gce_backend(self):
|
||||
"""Test getting GCE backend."""
|
||||
backend = get_encoding_backend("gce")
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.name == "gce"
|
||||
|
||||
def test_get_unknown_backend_raises(self):
|
||||
"""Test that unknown backend raises ValueError."""
|
||||
with pytest.raises(ValueError) as exc_info:
|
||||
get_encoding_backend("unknown")
|
||||
assert "Unknown encoding backend type" in str(exc_info.value)
|
||||
|
||||
def test_get_gce_backend_with_options(self):
|
||||
"""Test getting GCE backend with common options like dry_run.
|
||||
|
||||
This ensures all backends accept the same kwargs, preventing
|
||||
errors when get_encoding_backend() passes **kwargs to different backends.
|
||||
"""
|
||||
# This would have caught: GCEEncodingBackend.__init__() got an unexpected keyword argument 'dry_run'
|
||||
backend = get_encoding_backend("gce", dry_run=True)
|
||||
assert isinstance(backend, GCEEncodingBackend)
|
||||
assert backend.dry_run is True
|
||||
|
||||
@patch.object(GCEEncodingBackend, "_get_service")
|
||||
def test_all_backends_accept_dry_run(self, mock_get_service):
|
||||
"""Test that all backend types accept dry_run parameter.
|
||||
|
||||
This is an integration test to ensure the factory function
|
||||
can pass dry_run to any backend without TypeError.
|
||||
"""
|
||||
mock_service = MagicMock()
|
||||
mock_service.is_enabled = False
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
for backend_type in ["local", "gce", "auto"]:
|
||||
# This should not raise TypeError
|
||||
backend = get_encoding_backend(backend_type, dry_run=True)
|
||||
assert backend is not None
|
||||
@@ -1123,12 +1123,13 @@ class TestFinaliseOnlyModels:
|
||||
]
|
||||
)
|
||||
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
# CDG/TXT default to None (server resolves based on theme_id)
|
||||
assert request.enable_cdg is None
|
||||
assert request.enable_txt is None
|
||||
assert request.brand_prefix is None
|
||||
assert request.keep_brand_code is None
|
||||
assert request.enable_youtube_upload is False
|
||||
# YouTube upload default is None (server applies default_enable_youtube_upload)
|
||||
assert request.enable_youtube_upload is None
|
||||
assert request.dropbox_path is None
|
||||
assert request.gdrive_folder_id is None
|
||||
|
||||
@@ -1275,9 +1276,9 @@ class TestCreateJobFromUrlRequest:
|
||||
assert request.url == "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
|
||||
assert request.artist is None
|
||||
assert request.title is None
|
||||
# CDG/TXT disabled by default (requires style config)
|
||||
assert request.enable_cdg is False
|
||||
assert request.enable_txt is False
|
||||
# CDG/TXT default to None (server resolves based on theme_id)
|
||||
assert request.enable_cdg is None
|
||||
assert request.enable_txt is None
|
||||
|
||||
def test_create_with_artist_and_title(self):
|
||||
"""Test creating request with URL, artist, and title."""
|
||||
@@ -1606,5 +1607,135 @@ class TestIsUrlFunction:
|
||||
assert is_url("") is False
|
||||
|
||||
|
||||
class TestUploadEndpointThemeSupport:
|
||||
"""Test that /jobs/upload endpoint supports theme configuration.
|
||||
|
||||
CRITICAL: These tests verify that the /api/jobs/upload endpoint correctly handles
|
||||
theme_id and color_overrides parameters, ensuring preview videos have themed
|
||||
backgrounds instead of black backgrounds.
|
||||
|
||||
This addresses a bug where:
|
||||
- The frontend sends theme_id and color_overrides when uploading files
|
||||
- But the backend /api/jobs/upload endpoint was ignoring these parameters
|
||||
- Result: Jobs created via file upload had black backgrounds instead of themed ones
|
||||
"""
|
||||
|
||||
def test_upload_endpoint_accepts_theme_id_parameter(self):
|
||||
"""Verify the upload endpoint has theme_id as a form parameter.
|
||||
|
||||
CRITICAL: The frontend sends theme_id when uploading files with a theme.
|
||||
If this parameter is missing from the endpoint, the theme is silently ignored
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_theme_id_param = 'theme_id: Optional[str] = Form(' in source_code
|
||||
|
||||
assert has_theme_id_param, (
|
||||
"file_upload.py /jobs/upload endpoint does not have theme_id as a Form parameter. "
|
||||
"The frontend sends theme_id when uploading files with a theme, but the backend "
|
||||
"ignores it. Add: theme_id: Optional[str] = Form(None, description='Theme ID...')"
|
||||
)
|
||||
|
||||
def test_upload_endpoint_accepts_color_overrides_parameter(self):
|
||||
"""Verify the upload endpoint has color_overrides as a form parameter."""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_color_overrides_param = 'color_overrides: Optional[str] = Form(' in source_code
|
||||
|
||||
assert has_color_overrides_param, (
|
||||
"file_upload.py /jobs/upload endpoint does not have color_overrides as a Form parameter. "
|
||||
"The frontend sends color_overrides when customizing theme colors."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_calls_prepare_theme_for_job(self):
|
||||
"""Verify the upload endpoint calls _prepare_theme_for_job when theme_id is set.
|
||||
|
||||
CRITICAL: When a job is created via the upload endpoint with a theme_id
|
||||
(and no custom style files), the code must call _prepare_theme_for_job() to set:
|
||||
1. style_params_gcs_path (pointing to the copied style_params.json)
|
||||
2. style_assets (populated with asset mappings)
|
||||
|
||||
Without this, LyricsTranscriber won't have access to the theme's styles
|
||||
and preview videos will have black backgrounds instead of themed ones.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
# The function should be called in upload_and_create_job
|
||||
has_theme_prep_call = '_prepare_theme_for_job(' in source_code
|
||||
|
||||
assert has_theme_prep_call, (
|
||||
"file_upload.py does not call _prepare_theme_for_job(). "
|
||||
"When theme_id is provided to /jobs/upload without custom style files, "
|
||||
"the endpoint MUST call _prepare_theme_for_job() to copy the theme's "
|
||||
"style_params.json to the job folder."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_uses_resolve_cdg_txt_defaults(self):
|
||||
"""Verify the upload endpoint uses resolve_cdg_txt_defaults for theme-based defaults."""
|
||||
import inspect
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
source_code = inspect.getsource(file_upload_module)
|
||||
|
||||
# Check for the centralized resolve_cdg_txt_defaults function (imported from job_defaults_service)
|
||||
has_resolve_call = 'resolve_cdg_txt_defaults(' in source_code
|
||||
|
||||
assert has_resolve_call, (
|
||||
"file_upload.py does not call resolve_cdg_txt_defaults(). "
|
||||
"Theme-driven CDG/TXT defaults (controlled by DEFAULT_ENABLE_CDG/DEFAULT_ENABLE_TXT "
|
||||
"settings) must be applied via the centralized job_defaults_service."
|
||||
)
|
||||
|
||||
def test_upload_endpoint_has_optional_cdg_txt_params(self):
|
||||
"""Verify enable_cdg and enable_txt are Optional[bool] to support theme defaults.
|
||||
|
||||
CRITICAL: If enable_cdg/enable_txt are bool instead of Optional[bool],
|
||||
they will default to False and override the theme-based defaults.
|
||||
"""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_optional_cdg = 'enable_cdg: Optional[bool] = Form(' in source_code
|
||||
has_optional_txt = 'enable_txt: Optional[bool] = Form(' in source_code
|
||||
|
||||
assert has_optional_cdg, (
|
||||
"file_upload.py has enable_cdg as bool instead of Optional[bool]. "
|
||||
"This prevents theme-based defaults from working."
|
||||
)
|
||||
assert has_optional_txt, (
|
||||
"file_upload.py has enable_txt as bool instead of Optional[bool]. "
|
||||
"This prevents theme-based defaults from working."
|
||||
)
|
||||
|
||||
def test_job_create_includes_theme_id_and_color_overrides(self):
|
||||
"""Verify JobCreate is called with theme_id and color_overrides."""
|
||||
from backend.api.routes import file_upload as file_upload_module
|
||||
|
||||
with open(file_upload_module.__file__, 'r') as f:
|
||||
source_code = f.read()
|
||||
|
||||
has_theme_id_in_job_create = 'theme_id=theme_id,' in source_code
|
||||
has_color_overrides_in_job_create = 'color_overrides=parsed_color_overrides' in source_code
|
||||
|
||||
assert has_theme_id_in_job_create, (
|
||||
"file_upload.py does not pass theme_id to JobCreate."
|
||||
)
|
||||
assert has_color_overrides_in_job_create, (
|
||||
"file_upload.py does not pass color_overrides to JobCreate."
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,229 @@
|
||||
"""
|
||||
Tests for GCE Encoding Worker (gce_encoding/main.py).
|
||||
|
||||
These tests verify the file-finding logic in the GCE encoding worker,
|
||||
particularly ensuring that instrumental selection is respected.
|
||||
|
||||
NOTE: This test file is self-contained and doesn't import the full app
|
||||
to avoid dependency issues in CI/CD environments.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
# Copy the find_file function here to test it in isolation
|
||||
# This avoids importing the full gce_encoding module which has GCS dependencies
|
||||
def find_file(work_dir: Path, *patterns):
|
||||
'''Find a file matching any of the given glob patterns.'''
|
||||
for pattern in patterns:
|
||||
matches = list(work_dir.glob(f"**/{pattern}"))
|
||||
if matches:
|
||||
return matches[0]
|
||||
return None
|
||||
|
||||
|
||||
class TestFindFile:
|
||||
"""Test the find_file function used to locate input files."""
|
||||
|
||||
def test_find_file_single_match(self, tmp_path):
|
||||
"""Test finding a file with a single match."""
|
||||
# Create test file
|
||||
test_file = tmp_path / "test_instrumental.flac"
|
||||
test_file.touch()
|
||||
|
||||
result = find_file(tmp_path, "*instrumental*.flac")
|
||||
assert result is not None
|
||||
assert result.name == "test_instrumental.flac"
|
||||
|
||||
def test_find_file_no_match(self, tmp_path):
|
||||
"""Test finding a file with no matches returns None."""
|
||||
result = find_file(tmp_path, "*nonexistent*.flac")
|
||||
assert result is None
|
||||
|
||||
def test_find_file_priority_order(self, tmp_path):
|
||||
"""Test that find_file returns first matching pattern."""
|
||||
# Create multiple files that match different patterns
|
||||
first = tmp_path / "first_pattern.flac"
|
||||
second = tmp_path / "second_pattern.flac"
|
||||
first.touch()
|
||||
second.touch()
|
||||
|
||||
# First pattern should win
|
||||
result = find_file(tmp_path, "*first*.flac", "*second*.flac")
|
||||
assert result.name == "first_pattern.flac"
|
||||
|
||||
|
||||
class TestInstrumentalSelection:
|
||||
"""Test that instrumental selection is properly respected."""
|
||||
|
||||
def test_clean_instrumental_prioritized_when_selected(self, tmp_path):
|
||||
"""Test that clean instrumental is found when 'clean' is selected.
|
||||
|
||||
When both instrumentals exist and 'clean' is selected,
|
||||
the clean version should be found.
|
||||
"""
|
||||
# Create both instrumental files (as they exist in GCS)
|
||||
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
|
||||
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'clean' selection
|
||||
# (mimics the logic in run_encoding when instrumental_selection == 'clean')
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "Clean" in result.name
|
||||
|
||||
def test_backing_instrumental_prioritized_when_selected(self, tmp_path):
|
||||
"""Test that backing instrumental is found when 'with_backing' is selected.
|
||||
|
||||
When both instrumentals exist and 'with_backing' is selected,
|
||||
the backing version should be found.
|
||||
|
||||
This test would have caught the bug where the GCE worker always
|
||||
used the clean instrumental regardless of user selection.
|
||||
"""
|
||||
# Create both instrumental files (as they exist in GCS)
|
||||
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
|
||||
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'with_backing' selection
|
||||
# (mimics the logic in run_encoding when instrumental_selection == 'with_backing')
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "Backing" in result.name
|
||||
|
||||
def test_backing_instrumental_with_stem_naming(self, tmp_path):
|
||||
"""Test finding backing instrumental with GCS stem naming convention.
|
||||
|
||||
In GCS, stems are named like 'stems/instrumental_with_backing.flac'
|
||||
"""
|
||||
stems_dir = tmp_path / "stems"
|
||||
stems_dir.mkdir()
|
||||
|
||||
clean = stems_dir / "instrumental_clean.flac"
|
||||
backing = stems_dir / "instrumental_with_backing.flac"
|
||||
clean.touch()
|
||||
backing.touch()
|
||||
|
||||
# Simulate finding instrumental with 'with_backing' selection
|
||||
result = find_file(
|
||||
tmp_path,
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
assert result is not None
|
||||
assert "with_backing" in result.name
|
||||
|
||||
def test_fallback_to_generic_when_specific_not_found(self, tmp_path):
|
||||
"""Test fallback to generic pattern when specific not found."""
|
||||
# Only create a generically named instrumental
|
||||
generic = tmp_path / "instrumental.flac"
|
||||
generic.touch()
|
||||
|
||||
# Both clean and backing selections should fall back
|
||||
for selection in ["clean", "with_backing"]:
|
||||
if selection == "with_backing":
|
||||
patterns = (
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
else:
|
||||
patterns = (
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
|
||||
result = find_file(tmp_path, *patterns)
|
||||
assert result is not None
|
||||
assert result.name == "instrumental.flac"
|
||||
|
||||
|
||||
class TestInstrumentalSelectionRegression:
|
||||
"""Regression tests for instrumental selection bug.
|
||||
|
||||
Bug: User selected 'with_backing' instrumental in the UI but the final
|
||||
video used the clean instrumental instead.
|
||||
|
||||
Root cause: The GCE worker's find_file() always prioritized
|
||||
'*instrumental_clean*' patterns before generic patterns, ignoring
|
||||
the user's selection.
|
||||
"""
|
||||
|
||||
def test_bug_scenario_both_instrumentals_present(self, tmp_path):
|
||||
"""Recreate the exact bug scenario where both instrumentals exist.
|
||||
|
||||
This is the key regression test. When both instrumentals are downloaded
|
||||
from GCS (which happens with the GCE worker), the correct one must
|
||||
be selected based on the user's choice.
|
||||
"""
|
||||
# Setup: Both instrumentals in stems/ (as downloaded from GCS)
|
||||
stems = tmp_path / "stems"
|
||||
stems.mkdir()
|
||||
|
||||
(stems / "instrumental_clean.flac").touch()
|
||||
(stems / "instrumental_with_backing.flac").touch()
|
||||
|
||||
# Also have properly named versions in the root (from _setup_working_directory)
|
||||
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
|
||||
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
|
||||
|
||||
# When user selected 'with_backing', the backing version MUST be found
|
||||
backing_patterns = (
|
||||
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
|
||||
"*with_backing*.flac", "*Backing*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
result = find_file(tmp_path, *backing_patterns)
|
||||
|
||||
assert result is not None
|
||||
# Must find a backing-related file, not the clean one
|
||||
name = result.name.lower()
|
||||
assert "backing" in name or "with_backing" in name, \
|
||||
f"Expected backing instrumental but found: {result.name}"
|
||||
|
||||
def test_clean_selection_still_works(self, tmp_path):
|
||||
"""Verify clean selection still works correctly after the fix."""
|
||||
stems = tmp_path / "stems"
|
||||
stems.mkdir()
|
||||
|
||||
(stems / "instrumental_clean.flac").touch()
|
||||
(stems / "instrumental_with_backing.flac").touch()
|
||||
|
||||
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
|
||||
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
|
||||
|
||||
# When user selected 'clean', the clean version must be found
|
||||
clean_patterns = (
|
||||
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
|
||||
"*instrumental*.flac", "*Instrumental*.flac",
|
||||
"*instrumental*.wav"
|
||||
)
|
||||
result = find_file(tmp_path, *clean_patterns)
|
||||
|
||||
assert result is not None
|
||||
name = result.name.lower()
|
||||
assert "clean" in name, f"Expected clean instrumental but found: {result.name}"
|
||||
@@ -0,0 +1,223 @@
|
||||
"""
|
||||
Unit tests for user impersonation endpoint.
|
||||
|
||||
Tests the admin impersonation API that allows admins to view the app as other users.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch
|
||||
from fastapi.testclient import TestClient
|
||||
from fastapi import FastAPI
|
||||
|
||||
from backend.api.routes.admin import router
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.models.user import User, Session
|
||||
|
||||
|
||||
# Create a test app with the admin router
|
||||
app = FastAPI()
|
||||
app.include_router(router, prefix="/api")
|
||||
|
||||
|
||||
def get_mock_admin():
|
||||
"""Override for require_admin dependency - returns admin user."""
|
||||
return ("admin@nomadkaraoke.com", "admin", 1)
|
||||
|
||||
|
||||
def get_mock_regular_user():
|
||||
"""Override for require_admin dependency - returns regular user (should fail)."""
|
||||
# This simulates what happens when a non-admin tries to access
|
||||
# In reality, require_admin raises 403, but we test the logic
|
||||
return ("user@example.com", "user", 0)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client():
|
||||
"""Create a test client with admin access."""
|
||||
app.dependency_overrides[require_admin] = get_mock_admin
|
||||
yield TestClient(app)
|
||||
app.dependency_overrides.clear()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_user_service():
|
||||
"""Create a mock user service."""
|
||||
service = Mock()
|
||||
return service
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_target_user():
|
||||
"""Create a mock target user to impersonate."""
|
||||
user = Mock(spec=User)
|
||||
user.email = "target@example.com"
|
||||
user.credits = 5
|
||||
user.role = "user"
|
||||
user.is_active = True
|
||||
return user
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_session():
|
||||
"""Create a mock session."""
|
||||
session = Mock(spec=Session)
|
||||
session.token = "test-session-token-12345678901234567890"
|
||||
session.user_email = "target@example.com"
|
||||
return session
|
||||
|
||||
|
||||
class TestImpersonateUser:
|
||||
"""Tests for POST /api/admin/users/{email}/impersonate endpoint."""
|
||||
|
||||
def test_admin_can_impersonate_user(self, client, mock_target_user, mock_session):
|
||||
"""Admin gets session token for target user."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Override the dependency
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert data["session_token"] == mock_session.token
|
||||
assert data["user_email"] == "target@example.com"
|
||||
assert "impersonating" in data["message"].lower()
|
||||
|
||||
# Verify user lookup was called
|
||||
mock_service.get_user.assert_called_once_with("target@example.com")
|
||||
|
||||
# Verify session was created for target user
|
||||
mock_service.create_session.assert_called_once()
|
||||
call_args = mock_service.create_session.call_args
|
||||
assert call_args.kwargs["user_email"] == "target@example.com"
|
||||
assert "Impersonation by admin@nomadkaraoke.com" in call_args.kwargs["user_agent"]
|
||||
|
||||
def test_impersonate_nonexistent_user_returns_404(self, client):
|
||||
"""Target user must exist."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = None # User not found
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/nonexistent@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 404
|
||||
assert "not found" in response.json()["detail"].lower()
|
||||
|
||||
def test_cannot_impersonate_self(self, client, mock_target_user):
|
||||
"""Admin cannot impersonate themselves."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
# Set up user to be found (same as admin)
|
||||
mock_target_user.email = "admin@nomadkaraoke.com"
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/admin@nomadkaraoke.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
assert "yourself" in response.json()["detail"].lower()
|
||||
|
||||
def test_impersonation_creates_valid_session(self, client, mock_target_user, mock_session):
|
||||
"""Returned token is a valid session for target user."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify create_session was called with correct user
|
||||
mock_service.create_session.assert_called_once()
|
||||
call_kwargs = mock_service.create_session.call_args.kwargs
|
||||
assert call_kwargs["user_email"] == "target@example.com"
|
||||
|
||||
def test_impersonation_logs_audit_trail(self, client, mock_target_user, mock_session):
|
||||
"""Impersonation is logged for security audit."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service, \
|
||||
patch('backend.api.routes.admin.logger') as mock_logger:
|
||||
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/target@example.com/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify logging was called with impersonation info
|
||||
mock_logger.info.assert_called()
|
||||
log_message = mock_logger.info.call_args[0][0]
|
||||
assert "IMPERSONATION" in log_message
|
||||
assert "admin@nomadkaraoke.com" in log_message
|
||||
assert "target@example.com" in log_message
|
||||
|
||||
def test_email_is_normalized_to_lowercase(self, client, mock_target_user, mock_session):
|
||||
"""Email addresses are normalized to lowercase."""
|
||||
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.get_user.return_value = mock_target_user
|
||||
mock_service.create_session.return_value = mock_session
|
||||
mock_get_service.return_value = mock_service
|
||||
app.dependency_overrides[get_user_service] = lambda: mock_service
|
||||
|
||||
response = client.post(
|
||||
"/api/admin/users/TARGET@EXAMPLE.COM/impersonate",
|
||||
headers={"Authorization": "Bearer admin-token"}
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify user lookup was with lowercase
|
||||
mock_service.get_user.assert_called_once_with("target@example.com")
|
||||
|
||||
|
||||
class TestImpersonateUserNonAdmin:
|
||||
"""Tests to verify non-admins cannot impersonate."""
|
||||
|
||||
def test_non_admin_cannot_impersonate(self):
|
||||
"""
|
||||
Regular user gets 403 Forbidden.
|
||||
|
||||
Note: This is enforced by the require_admin dependency at the route level.
|
||||
We test that the dependency is correctly applied.
|
||||
"""
|
||||
# Create a fresh client without admin override
|
||||
test_app = FastAPI()
|
||||
test_app.include_router(router, prefix="/api")
|
||||
|
||||
# Don't override require_admin - it should reject non-admin requests
|
||||
# In a real test, we'd mock the auth service to return a non-admin user
|
||||
# For now, we just verify the endpoint requires admin via dependency
|
||||
|
||||
# The require_admin dependency will reject requests without proper admin auth
|
||||
# This is tested implicitly by the fact that all other tests use admin override
|
||||
pass # Actual enforcement tested via integration tests
|
||||
@@ -0,0 +1,587 @@
|
||||
"""
|
||||
Regression tests for job creation issues discovered 2024-12-31.
|
||||
|
||||
These tests ensure:
|
||||
1. user_email is properly extracted from AuthResult and set on jobs
|
||||
2. URL jobs trigger workers sequentially (audio first, then lyrics)
|
||||
3. Audio search downloads work without relying on in-memory cache
|
||||
4. Transcription has proper timeout handling
|
||||
|
||||
See docs/archive/2024-12-31-job-failure-investigation.md for details.
|
||||
"""
|
||||
import asyncio
|
||||
import pytest
|
||||
from unittest.mock import Mock, AsyncMock, patch, MagicMock
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
|
||||
|
||||
class TestUserEmailExtraction:
|
||||
"""
|
||||
Issue 1: user_email not being set on jobs.
|
||||
|
||||
Root cause: Job creation endpoints used require_auth but didn't extract
|
||||
auth_result.user_email and set it on the job.
|
||||
|
||||
These tests verify that authenticated user's email is properly set on jobs.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_with_email(self):
|
||||
"""AuthResult for a logged-in user with email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED, # Session-based users typically have UNLIMITED
|
||||
remaining_uses=-1,
|
||||
message="Valid session",
|
||||
user_email="testuser@example.com",
|
||||
is_admin=False,
|
||||
)
|
||||
|
||||
@pytest.fixture
|
||||
def mock_auth_result_admin_token(self):
|
||||
"""AuthResult for admin token without email."""
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Valid admin token",
|
||||
user_email=None,
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
def test_auth_result_has_user_email_field(self):
|
||||
"""AuthResult must have user_email field for job association."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.UNLIMITED,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
assert hasattr(result, 'user_email')
|
||||
assert result.user_email == "user@example.com"
|
||||
|
||||
def test_auth_result_user_email_can_be_none(self):
|
||||
"""AuthResult user_email can be None for token-based auth."""
|
||||
result = AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="test",
|
||||
user_email=None,
|
||||
)
|
||||
assert result.user_email is None
|
||||
|
||||
def test_upload_endpoint_extracts_user_email_from_auth(self):
|
||||
"""
|
||||
/jobs/upload endpoint must set user_email from authenticated user.
|
||||
|
||||
This verifies the code pattern by inspecting the source.
|
||||
Integration tests verify the full behavior.
|
||||
"""
|
||||
from backend.api.routes import file_upload
|
||||
import inspect
|
||||
|
||||
# Get the source code of upload_and_create_job
|
||||
source = inspect.getsource(file_upload.upload_and_create_job)
|
||||
|
||||
# Verify the endpoint extracts user_email from auth_result
|
||||
assert 'auth_result.user_email' in source, \
|
||||
"upload_and_create_job must extract user_email from auth_result"
|
||||
assert 'effective_user_email' in source, \
|
||||
"upload_and_create_job must use effective_user_email pattern"
|
||||
assert 'user_email=effective_user_email' in source or 'user_email=' in source, \
|
||||
"upload_and_create_job must pass user_email to JobCreate"
|
||||
|
||||
def test_all_job_creation_endpoints_extract_user_email(self):
|
||||
"""
|
||||
All job creation endpoints must extract user_email from AuthResult.
|
||||
|
||||
This is a regression guard for Issue 1 (user_email not being set).
|
||||
"""
|
||||
from backend.api.routes import file_upload, audio_search
|
||||
import inspect
|
||||
|
||||
endpoints_to_check = [
|
||||
(file_upload, 'upload_and_create_job'),
|
||||
(file_upload, 'create_job_from_url'),
|
||||
(audio_search, 'search_audio'),
|
||||
]
|
||||
|
||||
for module, func_name in endpoints_to_check:
|
||||
func = getattr(module, func_name)
|
||||
source = inspect.getsource(func)
|
||||
|
||||
assert 'auth_result.user_email' in source, \
|
||||
f"{func_name} must extract user_email from auth_result"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_from_url_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/jobs/create-from-url endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.file_upload import create_job_from_url, CreateJobFromUrlRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/jobs/create-from-url")
|
||||
# Mock tenant state (no tenant = default Nomad Karaoke)
|
||||
mock_request.state.tenant_config = None
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = CreateJobFromUrlRequest(
|
||||
url="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
||||
artist="Rick Astley",
|
||||
title="Never Gonna Give You Up",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.file_upload.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.file_upload.worker_service') as mock_worker, \
|
||||
patch('backend.api.routes.file_upload.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-456"
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
|
||||
await create_job_from_url(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"create-from-url must set user_email from AuthResult"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_search_endpoint_sets_user_email(self, mock_auth_result_with_email):
|
||||
"""
|
||||
/audio-search/search endpoint must set user_email from authenticated user.
|
||||
"""
|
||||
from backend.api.routes.audio_search import search_audio, AudioSearchRequest
|
||||
|
||||
mock_request = Mock()
|
||||
mock_request.headers = {}
|
||||
mock_request.client = Mock(host="127.0.0.1")
|
||||
mock_request.url = Mock(path="/api/audio-search/search")
|
||||
# Mock tenant state (no tenant = default Nomad Karaoke)
|
||||
mock_request.state.tenant_config = None
|
||||
|
||||
mock_background_tasks = Mock()
|
||||
body = AudioSearchRequest(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
with patch('backend.api.routes.audio_search.job_manager') as mock_jm, \
|
||||
patch('backend.api.routes.audio_search.get_audio_search_service') as mock_search, \
|
||||
patch('backend.api.routes.audio_search.get_credential_manager') as mock_cred:
|
||||
|
||||
mock_job = Mock()
|
||||
mock_job.job_id = "test-job-789"
|
||||
mock_job.state_data = {}
|
||||
mock_jm.create_job.return_value = mock_job
|
||||
mock_jm.get_job.return_value = mock_job
|
||||
mock_cred.return_value.check_youtube_credentials.return_value = Mock(status=Mock(value="valid"))
|
||||
mock_search.return_value.search.return_value = []
|
||||
|
||||
await search_audio(
|
||||
request=mock_request,
|
||||
background_tasks=mock_background_tasks,
|
||||
body=body,
|
||||
auth_result=mock_auth_result_with_email,
|
||||
)
|
||||
|
||||
mock_jm.create_job.assert_called_once()
|
||||
job_create_arg = mock_jm.create_job.call_args[0][0]
|
||||
assert job_create_arg.user_email == "testuser@example.com", \
|
||||
"audio-search must set user_email from AuthResult"
|
||||
|
||||
def test_effective_user_email_prefers_auth_over_form(self):
|
||||
"""
|
||||
When both auth_result.user_email and form user_email are provided,
|
||||
the authenticated user's email should take precedence.
|
||||
"""
|
||||
auth_email = "authenticated@example.com"
|
||||
form_email = "form@example.com"
|
||||
|
||||
# This is the logic used in the endpoints
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == auth_email, \
|
||||
"Authenticated user's email should take precedence over form parameter"
|
||||
|
||||
def test_effective_user_email_falls_back_to_form(self):
|
||||
"""
|
||||
When auth_result.user_email is None (e.g., admin token),
|
||||
form user_email should be used as fallback.
|
||||
"""
|
||||
auth_email = None
|
||||
form_email = "form@example.com"
|
||||
|
||||
effective_user_email = auth_email or form_email
|
||||
|
||||
assert effective_user_email == form_email, \
|
||||
"Form parameter should be used when auth has no email"
|
||||
|
||||
|
||||
class TestUrlJobWorkerSequencing:
|
||||
"""
|
||||
Issue 2: YouTube URL download race condition.
|
||||
|
||||
Root cause: Both audio and lyrics workers were triggered in parallel.
|
||||
For URL jobs, lyrics worker would timeout waiting for audio to download.
|
||||
|
||||
These tests verify that URL jobs trigger workers sequentially.
|
||||
"""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_url_job_triggers_only_audio_worker_initially(self):
|
||||
"""
|
||||
create-from-url must only trigger audio worker, not lyrics worker.
|
||||
The audio worker will trigger lyrics after download completes.
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_audio_worker_only
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_audio_worker_only("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parallel_worker_triggers_both_workers(self):
|
||||
"""
|
||||
_trigger_workers_parallel should trigger both workers (for uploaded files).
|
||||
"""
|
||||
from backend.api.routes.file_upload import _trigger_workers_parallel
|
||||
|
||||
with patch('backend.api.routes.file_upload.worker_service') as mock_ws:
|
||||
mock_ws.trigger_audio_worker = AsyncMock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
|
||||
await _trigger_workers_parallel("test-job-id")
|
||||
|
||||
mock_ws.trigger_audio_worker.assert_called_once_with("test-job-id")
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_triggers_lyrics_after_url_download(self):
|
||||
"""
|
||||
Audio worker must trigger lyrics worker after successful URL download.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock()
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
mock_ws.trigger_lyrics_worker.assert_called_once_with("test-job-id")
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_audio_worker_lyrics_trigger_handles_errors_gracefully(self):
|
||||
"""
|
||||
If lyrics worker trigger fails, audio processing should continue.
|
||||
"""
|
||||
from backend.workers.audio_worker import _trigger_lyrics_worker_after_url_download
|
||||
|
||||
# Mock at the source module where it's imported from
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_ws:
|
||||
mock_ws = Mock()
|
||||
mock_ws.trigger_lyrics_worker = AsyncMock(side_effect=Exception("Network error"))
|
||||
mock_get_ws.return_value = mock_ws
|
||||
|
||||
# Should not raise exception
|
||||
await _trigger_lyrics_worker_after_url_download("test-job-id")
|
||||
|
||||
|
||||
class TestAudioSearchCacheIndependence:
|
||||
"""
|
||||
Issue 3: Audio search cache not persisting across Cloud Run instances.
|
||||
|
||||
Root cause: AudioSearchService used in-memory cache that doesn't persist
|
||||
across horizontally scaled instances.
|
||||
|
||||
These tests verify downloads work using state_data, not in-memory cache.
|
||||
"""
|
||||
|
||||
def test_search_results_stored_in_job_state_data(self):
|
||||
"""
|
||||
Search results must be stored in job.state_data for persistence.
|
||||
This tests the Job model's ability to store search results.
|
||||
"""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
state_data={
|
||||
'audio_search_results': [
|
||||
{
|
||||
'provider': 'YouTube',
|
||||
'title': 'Test Song',
|
||||
'artist': 'Test Artist',
|
||||
'url': 'https://youtube.com/watch?v=abc123',
|
||||
'source_id': 'abc123',
|
||||
}
|
||||
]
|
||||
}
|
||||
)
|
||||
|
||||
assert 'audio_search_results' in job.state_data
|
||||
assert len(job.state_data['audio_search_results']) == 1
|
||||
assert job.state_data['audio_search_results'][0]['url'] == 'https://youtube.com/watch?v=abc123'
|
||||
|
||||
def test_audio_search_endpoint_stores_results_in_state_data(self):
|
||||
"""
|
||||
Verify that the search_audio code path stores results in job.state_data.
|
||||
|
||||
This tests that the audio_search route has the correct pattern for
|
||||
persisting search results. The actual integration is tested elsewhere.
|
||||
"""
|
||||
# Verify the import and module structure exists
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source code of search_audio
|
||||
source = inspect.getsource(audio_search.search_audio)
|
||||
|
||||
# Verify the endpoint stores results in state_data
|
||||
# The code uses job_manager.update_job with state_data dict
|
||||
assert 'audio_search_results' in source, \
|
||||
"search_audio must store results under 'audio_search_results' key"
|
||||
assert 'state_data' in source, \
|
||||
"search_audio must use state_data for persistence"
|
||||
|
||||
def test_download_code_has_youtube_direct_download_branch(self):
|
||||
"""
|
||||
Verify that _download_and_start_processing has the YouTube direct download path.
|
||||
|
||||
This checks that the code path exists for downloading YouTube audio directly
|
||||
using the URL from state_data, avoiding the in-memory cache dependency.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
# Get the source of _download_and_start_processing
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# Verify the YouTube direct download branch exists
|
||||
assert "source_name == 'YouTube'" in source, \
|
||||
"_download_and_start_processing must check for YouTube source"
|
||||
assert 'download_url' in source, \
|
||||
"_download_and_start_processing must use download_url from state_data"
|
||||
assert 'download_from_url' in source, \
|
||||
"_download_and_start_processing must call download_from_url for YouTube"
|
||||
|
||||
def test_youtube_download_branch_avoids_cache_dependency(self):
|
||||
"""
|
||||
Verify that the YouTube download path uses URL from state_data,
|
||||
not the in-memory cache that doesn't persist across Cloud Run instances.
|
||||
"""
|
||||
from backend.api.routes import audio_search
|
||||
import inspect
|
||||
|
||||
source = inspect.getsource(audio_search._download_and_start_processing)
|
||||
|
||||
# The fix specifically uses selected.get('url') which comes from state_data
|
||||
# rather than audio_search_service cache
|
||||
assert "selected.get('url')" in source or "download_url" in source, \
|
||||
"YouTube download must use URL from selected result (state_data)"
|
||||
|
||||
# Verify it doesn't rely on cache for YouTube
|
||||
# The branch is: elif source_name == 'YouTube' and download_url:
|
||||
assert "YouTube" in source and "download_url" in source, \
|
||||
"YouTube branch must check for download_url availability"
|
||||
|
||||
|
||||
class TestTranscriptionTimeout:
|
||||
"""
|
||||
Issue 4: Jobs stuck in downloading state.
|
||||
|
||||
Root cause: Lyrics worker's AudioShake transcription could hang forever
|
||||
without proper timeout, leaving jobs stuck.
|
||||
|
||||
These tests verify timeout handling for transcription.
|
||||
"""
|
||||
|
||||
def test_transcription_timeout_constant_exists(self):
|
||||
"""
|
||||
Verify transcription timeout constant is defined in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS > 0
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS >= 300, "Timeout should be at least 5 minutes for long songs"
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS <= 1500, "Timeout shouldn't be more than 25 minutes"
|
||||
|
||||
def test_transcription_timeout_value(self):
|
||||
"""
|
||||
Verify the specific timeout value (20 minutes = 1200 seconds).
|
||||
|
||||
This accounts for:
|
||||
- Cloud Run cold start / worker initialization (1-5 min)
|
||||
- AudioShake transcription (1-2 min)
|
||||
- spaCy model loading for correction (2-3 min on cold start)
|
||||
- Agentic AI correction (1-3 min)
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
assert TRANSCRIPTION_TIMEOUT_SECONDS == 1200, "Transcription timeout should be 20 minutes (1200 seconds)"
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_asyncio_wait_for_raises_timeout_error(self):
|
||||
"""
|
||||
Verify asyncio.wait_for properly raises TimeoutError.
|
||||
This is a sanity check for the timeout mechanism we use.
|
||||
"""
|
||||
async def slow_operation():
|
||||
await asyncio.sleep(10)
|
||||
return "completed"
|
||||
|
||||
with pytest.raises(asyncio.TimeoutError):
|
||||
await asyncio.wait_for(slow_operation(), timeout=0.1)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_lyrics_worker_timeout_converts_to_descriptive_error(self):
|
||||
"""
|
||||
Verify that timeout is converted to a descriptive exception message.
|
||||
Tests the actual error conversion logic pattern used in lyrics_worker.
|
||||
"""
|
||||
from backend.workers.lyrics_worker import TRANSCRIPTION_TIMEOUT_SECONDS
|
||||
|
||||
# This simulates what happens in lyrics_worker when timeout occurs
|
||||
error_message = None
|
||||
try:
|
||||
# Simulate the timeout handling pattern from lyrics_worker
|
||||
raise asyncio.TimeoutError()
|
||||
except asyncio.TimeoutError:
|
||||
error_message = f"Transcription timed out after {TRANSCRIPTION_TIMEOUT_SECONDS} seconds"
|
||||
|
||||
assert error_message is not None
|
||||
assert "timed out" in error_message.lower()
|
||||
assert "1200" in error_message
|
||||
|
||||
def test_lyrics_worker_exception_marks_job_failed(self):
|
||||
"""
|
||||
Any exception in lyrics worker should mark job as failed.
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
# Simulate the exception handling in lyrics_worker
|
||||
job_status = JobStatus.DOWNLOADING
|
||||
error_occurred = True
|
||||
|
||||
if error_occurred:
|
||||
# This is what happens in the except block
|
||||
job_status = JobStatus.FAILED
|
||||
error_message = "Lyrics transcription failed: Transcription timed out after 1200 seconds"
|
||||
|
||||
assert job_status == JobStatus.FAILED
|
||||
assert "timed out" in error_message.lower()
|
||||
|
||||
|
||||
class TestJobOwnershipFiltering:
|
||||
"""
|
||||
Additional tests for job ownership and filtering.
|
||||
|
||||
These verify that jobs are properly associated with users
|
||||
and can be filtered by user_email.
|
||||
"""
|
||||
|
||||
def test_job_model_has_user_email_field(self):
|
||||
"""Job model must have user_email field."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
job = Job(
|
||||
job_id="test-123",
|
||||
artist="Test",
|
||||
title="Test",
|
||||
status=JobStatus.PENDING,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job, 'user_email')
|
||||
assert job.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_model_has_user_email_field(self):
|
||||
"""JobCreate model must have user_email field."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert hasattr(job_create, 'user_email')
|
||||
assert job_create.user_email == "user@example.com"
|
||||
|
||||
def test_job_create_user_email_is_optional(self):
|
||||
"""JobCreate user_email should be optional for backward compatibility."""
|
||||
from backend.models.job import JobCreate
|
||||
|
||||
job_create = JobCreate(
|
||||
artist="Test",
|
||||
title="Test",
|
||||
)
|
||||
|
||||
assert job_create.user_email is None
|
||||
|
||||
def test_jobs_can_be_filtered_by_user_email(self):
|
||||
"""Jobs should be filterable by user_email."""
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
jobs = [
|
||||
Job(
|
||||
job_id="job-1",
|
||||
artist="Test",
|
||||
title="Test 1",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-2",
|
||||
artist="Test",
|
||||
title="Test 2",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user2@example.com",
|
||||
),
|
||||
Job(
|
||||
job_id="job-3",
|
||||
artist="Test",
|
||||
title="Test 3",
|
||||
status=JobStatus.COMPLETE,
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
user_email="user1@example.com",
|
||||
),
|
||||
]
|
||||
|
||||
user1_jobs = [j for j in jobs if j.user_email == "user1@example.com"]
|
||||
|
||||
assert len(user1_jobs) == 2
|
||||
assert all(j.user_email == "user1@example.com" for j in user1_jobs)
|
||||
@@ -33,13 +33,28 @@ def job_manager(mock_firestore_service):
|
||||
|
||||
class TestJobCreation:
|
||||
"""Test job creation logic."""
|
||||
|
||||
|
||||
def test_create_job_requires_theme_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that jobs without theme_id are rejected."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
# No theme_id - should fail
|
||||
)
|
||||
|
||||
with pytest.raises(ValueError, match="theme_id is required"):
|
||||
job_manager.create_job(job_create)
|
||||
|
||||
# Verify Firestore was NOT called
|
||||
mock_firestore_service.create_job.assert_not_called()
|
||||
|
||||
def test_create_job_with_url(self, job_manager, mock_firestore_service):
|
||||
"""Test creating a job with YouTube URL."""
|
||||
job_create = JobCreate(
|
||||
url="https://youtube.com/watch?v=test",
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
# The actual create_job method creates the job and returns it
|
||||
@@ -59,7 +74,8 @@ class TestJobCreation:
|
||||
"""Test creating a job without URL (for file upload)."""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song"
|
||||
title="Test Song",
|
||||
theme_id="nomad" # Required for all jobs
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
@@ -70,8 +86,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
|
||||
"""Test that each job gets a unique ID."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
# Create multiple jobs
|
||||
ids = []
|
||||
for i in range(5):
|
||||
@@ -89,8 +105,8 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
|
||||
"""Test that new jobs start with PENDING status."""
|
||||
job_create = JobCreate()
|
||||
|
||||
job_create = JobCreate(theme_id="nomad") # Required for all jobs
|
||||
|
||||
mock_firestore_service.create_job.return_value = Job(
|
||||
job_id="test123",
|
||||
status=JobStatus.PENDING,
|
||||
@@ -105,13 +121,14 @@ class TestJobCreation:
|
||||
|
||||
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
|
||||
"""Test that distribution settings are passed from JobCreate to Job.
|
||||
|
||||
|
||||
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
|
||||
discord_webhook_url were NOT being passed to the Job constructor.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad", # Required for all jobs
|
||||
brand_prefix="NOMAD",
|
||||
discord_webhook_url="https://discord.com/webhook/test",
|
||||
dropbox_path="/Karaoke/Tracks-Organized",
|
||||
@@ -296,9 +313,154 @@ class TestJobFailure:
|
||||
assert call_args[1]['error_message'] == error_message
|
||||
|
||||
|
||||
class TestMadeForYouFieldMapping:
|
||||
"""
|
||||
CRITICAL: Test that made-for-you fields are properly copied from JobCreate to Job.
|
||||
|
||||
These tests verify that JobManager.create_job() properly maps all fields from
|
||||
JobCreate to the Job object that gets persisted to Firestore.
|
||||
|
||||
Bug context (2026-01-09): Production made-for-you orders were created with
|
||||
made_for_you=False, customer_email=None, customer_notes=None because
|
||||
JobManager.create_job() wasn't copying these fields from JobCreate to Job.
|
||||
"""
|
||||
|
||||
def test_create_job_copies_made_for_you_flag(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
CRITICAL: made_for_you flag must be copied from JobCreate to Job.
|
||||
|
||||
This flag is essential for:
|
||||
- Ownership transfer on job completion
|
||||
- Email suppression for intermediate states
|
||||
- Identifying made-for-you jobs in admin UI
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True, # This MUST be copied to the Job
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# CRITICAL ASSERTION: made_for_you must be True on the created Job
|
||||
assert job.made_for_you is True, \
|
||||
"made_for_you=True on JobCreate must be copied to Job"
|
||||
|
||||
# Also verify what was saved to Firestore
|
||||
mock_firestore_service.create_job.assert_called_once()
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.made_for_you is True, \
|
||||
"made_for_you=True must be persisted to Firestore"
|
||||
|
||||
def test_create_job_copies_customer_email(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
CRITICAL: customer_email must be copied from JobCreate to Job.
|
||||
|
||||
This field is essential for:
|
||||
- Ownership transfer on job completion (transferring to this email)
|
||||
- Sending completion email with download links to customer
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
customer_email="customer@example.com", # This MUST be copied to the Job
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# CRITICAL ASSERTION: customer_email must be copied
|
||||
assert job.customer_email == "customer@example.com", \
|
||||
"customer_email on JobCreate must be copied to Job"
|
||||
|
||||
# Also verify what was saved to Firestore
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.customer_email == "customer@example.com", \
|
||||
"customer_email must be persisted to Firestore"
|
||||
|
||||
def test_create_job_copies_customer_notes(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
customer_notes must be copied from JobCreate to Job.
|
||||
|
||||
Customer notes contain special requests that admin needs to see.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
customer_notes="Please make this perfect for my wedding!",
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.customer_notes == "Please make this perfect for my wedding!", \
|
||||
"customer_notes on JobCreate must be copied to Job"
|
||||
|
||||
saved_job = mock_firestore_service.create_job.call_args[0][0]
|
||||
assert saved_job.customer_notes == "Please make this perfect for my wedding!", \
|
||||
"customer_notes must be persisted to Firestore"
|
||||
|
||||
def test_create_job_full_made_for_you_config(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
Test complete made-for-you job creation with all fields.
|
||||
|
||||
This is the realistic scenario: a made-for-you order creates a job with:
|
||||
- made_for_you=True
|
||||
- user_email=admin (owner during processing)
|
||||
- customer_email=customer (for final delivery)
|
||||
- customer_notes=notes (customer's special requests)
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Avril Lavigne",
|
||||
title="Complicated",
|
||||
theme_id="nomad",
|
||||
made_for_you=True,
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
customer_email="customer@example.com",
|
||||
customer_notes="Anniversary gift!",
|
||||
# Distribution settings that should also be applied
|
||||
enable_youtube_upload=True,
|
||||
dropbox_path="/Production/Ready",
|
||||
brand_prefix="NOMAD",
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
# Verify all made-for-you fields
|
||||
assert job.made_for_you is True
|
||||
assert job.user_email == "admin@nomadkaraoke.com"
|
||||
assert job.customer_email == "customer@example.com"
|
||||
assert job.customer_notes == "Anniversary gift!"
|
||||
|
||||
# Verify distribution settings too
|
||||
assert job.enable_youtube_upload is True
|
||||
assert job.dropbox_path == "/Production/Ready"
|
||||
assert job.brand_prefix == "NOMAD"
|
||||
|
||||
def test_create_job_made_for_you_false_by_default(self, job_manager, mock_firestore_service):
|
||||
"""
|
||||
Regular jobs should have made_for_you=False by default.
|
||||
"""
|
||||
job_create = JobCreate(
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
theme_id="nomad",
|
||||
# No made_for_you specified - should default to False
|
||||
)
|
||||
|
||||
job = job_manager.create_job(job_create)
|
||||
|
||||
assert job.made_for_you is False
|
||||
assert job.customer_email is None
|
||||
assert job.customer_notes is None
|
||||
|
||||
|
||||
class TestJobDeletion:
|
||||
"""Test job deletion logic."""
|
||||
|
||||
|
||||
def test_delete_job(self, job_manager, mock_firestore_service):
|
||||
"""Test deleting a job."""
|
||||
existing_job = Job(
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
"""
|
||||
Unit tests for job manager notification methods.
|
||||
|
||||
Tests the state transition triggers for email notifications.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, MagicMock, AsyncMock
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import Job, JobStatus
|
||||
|
||||
|
||||
class TestTriggerStateNotifications:
|
||||
"""Tests for _trigger_state_notifications method."""
|
||||
|
||||
def test_triggers_completion_email_on_complete(self):
|
||||
"""Test that completion email is triggered when job completes."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_review(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting review."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
|
||||
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
|
||||
manager._trigger_state_notifications(
|
||||
"job-123",
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
mock_schedule.assert_called_once_with(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
def test_skips_notification_when_no_job(self):
|
||||
"""Test that notification is skipped when job not found."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(return_value=None)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_skips_notification_when_no_user_email(self):
|
||||
"""Test that notification is skipped when job has no user email."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = None
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_schedule.assert_not_called()
|
||||
|
||||
def test_does_not_trigger_on_other_states(self):
|
||||
"""Test that no notification is triggered for non-notification states."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
|
||||
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
|
||||
|
||||
# Test various non-notification states
|
||||
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
|
||||
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
|
||||
manager._trigger_state_notifications("job-123", status)
|
||||
|
||||
mock_complete.assert_not_called()
|
||||
mock_idle.assert_not_called()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate from notifications."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
manager.get_job = Mock(side_effect=Exception("Database error"))
|
||||
|
||||
# Should not raise
|
||||
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
|
||||
|
||||
|
||||
class TestScheduleCompletionEmail:
|
||||
"""Tests for _schedule_completion_email method."""
|
||||
|
||||
def test_extracts_urls_from_state_data(self):
|
||||
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = "Test Artist"
|
||||
mock_job.title = "Test Song"
|
||||
mock_job.state_data = {
|
||||
"youtube_url": "https://youtube.com/watch?v=123",
|
||||
"dropbox_link": "https://dropbox.com/folder/abc",
|
||||
}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Run in sync context - this will start a daemon thread
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
# Give the thread a moment to execute
|
||||
import time
|
||||
time.sleep(0.1)
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.artist = None
|
||||
mock_job.title = None
|
||||
mock_job.state_data = None # None state_data
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.send_job_completion_email = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
|
||||
mock_get_service.side_effect = Exception("Service error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_completion_email(mock_job)
|
||||
|
||||
|
||||
class TestScheduleIdleReminder:
|
||||
"""Tests for _schedule_idle_reminder method."""
|
||||
|
||||
def test_updates_state_data_for_lyrics_review(self):
|
||||
"""Test that state_data is updated with blocking state info for lyrics."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {"existing_key": "value"}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
assert update_call is not None
|
||||
updated_data = update_call[0][1] # Second positional arg
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'lyrics'
|
||||
assert state_data.get('reminder_sent') is False
|
||||
assert 'blocking_state_entered_at' in state_data
|
||||
assert state_data.get('existing_key') == 'value' # Preserved
|
||||
|
||||
def test_updates_state_data_for_instrumental_selection(self):
|
||||
"""Test that state_data is updated with blocking state info for instrumental."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
manager._schedule_idle_reminder(
|
||||
mock_job,
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
|
||||
)
|
||||
|
||||
# Verify state_data was updated
|
||||
update_call = manager.firestore.update_job.call_args
|
||||
updated_data = update_call[0][1]
|
||||
state_data = updated_data.get('state_data', {})
|
||||
|
||||
assert state_data.get('blocking_action_type') == 'instrumental'
|
||||
|
||||
def test_handles_none_state_data(self):
|
||||
"""Test that None state_data is handled gracefully."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = None
|
||||
|
||||
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
|
||||
mock_service = Mock()
|
||||
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
|
||||
mock_get_service.return_value = mock_service
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
# Verify update was still called
|
||||
manager.firestore.update_job.assert_called_once()
|
||||
|
||||
def test_handles_exception_gracefully(self):
|
||||
"""Test that exceptions don't propagate."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
|
||||
# Make firestore raise an exception
|
||||
manager.firestore.update_job.side_effect = Exception("Database error")
|
||||
|
||||
# Should not raise
|
||||
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
|
||||
|
||||
|
||||
class TestTransitionTriggersNotifications:
|
||||
"""Integration tests for transition_to_state triggering notifications."""
|
||||
|
||||
def test_transition_to_complete_triggers_notification(self):
|
||||
"""Test that transitioning to COMPLETE triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
|
||||
mock_job.status = JobStatus.ENCODING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.COMPLETE)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
|
||||
|
||||
def test_transition_to_awaiting_review_triggers_notification(self):
|
||||
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
|
||||
manager = JobManager()
|
||||
manager.firestore = Mock()
|
||||
|
||||
mock_job = Mock(spec=Job)
|
||||
mock_job.job_id = "job-123"
|
||||
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
|
||||
mock_job.status = JobStatus.APPLYING_PADDING
|
||||
mock_job.user_email = "user@example.com"
|
||||
mock_job.state_data = {}
|
||||
manager.get_job = Mock(return_value=mock_job)
|
||||
|
||||
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
|
||||
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
|
||||
|
||||
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
Unit tests for job notification service.
|
||||
"""
|
||||
import pytest
|
||||
from unittest.mock import Mock, patch, AsyncMock
|
||||
import urllib.parse
|
||||
|
||||
from backend.services.job_notification_service import (
|
||||
JobNotificationService,
|
||||
get_job_notification_service,
|
||||
ENABLE_AUTO_EMAILS,
|
||||
FEEDBACK_FORM_URL,
|
||||
)
|
||||
|
||||
|
||||
class TestURLBuilding:
|
||||
"""Tests for URL building methods."""
|
||||
|
||||
def test_build_review_url_basic(self):
|
||||
"""Test basic review URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/lyrics/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
# The API URL should be URL-encoded
|
||||
assert urllib.parse.quote("https://api.nomadkaraoke.com/api/review/job-123", safe='') in url
|
||||
|
||||
def test_build_review_url_with_audio_hash(self):
|
||||
"""Test review URL with audio hash parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", audio_hash="abc123")
|
||||
|
||||
assert "audioHash=abc123" in url
|
||||
|
||||
def test_build_review_url_with_review_token(self):
|
||||
"""Test review URL with review token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job-123", review_token="token456")
|
||||
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_with_all_params(self):
|
||||
"""Test review URL with all parameters."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url(
|
||||
"job-123",
|
||||
audio_hash="hash789",
|
||||
review_token="token456"
|
||||
)
|
||||
|
||||
assert "baseApiUrl=" in url
|
||||
assert "audioHash=hash789" in url
|
||||
assert "reviewToken=token456" in url
|
||||
|
||||
def test_build_review_url_encodes_special_chars(self):
|
||||
"""Test that special characters in job ID are encoded."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_review_url("job/with/slashes")
|
||||
|
||||
# The baseApiUrl parameter should have encoded slashes
|
||||
assert "%2F" in url
|
||||
|
||||
def test_build_instrumental_url_basic(self):
|
||||
"""Test basic instrumental URL building."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123")
|
||||
|
||||
assert "gen.nomadkaraoke.com/instrumental/" in url
|
||||
assert "baseApiUrl=" in url
|
||||
|
||||
def test_build_instrumental_url_with_token(self):
|
||||
"""Test instrumental URL with token parameter."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
|
||||
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
|
||||
|
||||
assert "instrumentalToken=inst-token" in url
|
||||
|
||||
|
||||
class TestCompletionEmail:
|
||||
"""Tests for job completion email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_success(self):
|
||||
"""Test successful completion email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
service.email_service.send_job_completion.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_disabled(self):
|
||||
"""Test that completion email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_no_email(self):
|
||||
"""Test that completion email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_empty_email(self):
|
||||
"""Test that completion email is skipped when user email is empty."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_job_completion.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_send_failure(self):
|
||||
"""Test handling of email send failure."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = False
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_exception(self):
|
||||
"""Test handling of exceptions during email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.side_effect = Exception("Send error")
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_completion_email_passes_cc_admin(self):
|
||||
"""Test that completion email is sent with CC to admin."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_job_completion.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Test message"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_job_completion_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
)
|
||||
|
||||
# Verify cc_admin=True was passed
|
||||
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('cc_admin') is True
|
||||
|
||||
|
||||
class TestActionReminderEmail:
|
||||
"""Tests for action reminder email sending."""
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_success(self):
|
||||
"""Test successful lyrics reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_lyrics.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_success(self):
|
||||
"""Test successful instrumental reminder email sending."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
)
|
||||
|
||||
assert result is True
|
||||
service.template_service.render_action_needed_instrumental.assert_called_once()
|
||||
service.email_service.send_action_reminder.assert_called_once()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_disabled(self):
|
||||
"""Test that reminder email is skipped when auto emails are disabled."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_no_email(self):
|
||||
"""Test that reminder email is skipped when no user email."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email=None,
|
||||
action_type="lyrics",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_reminder_unknown_action_type(self):
|
||||
"""Test handling of unknown action type."""
|
||||
service = JobNotificationService()
|
||||
service.email_service = Mock()
|
||||
service.template_service = Mock()
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
result = await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="unknown",
|
||||
)
|
||||
|
||||
assert result is False
|
||||
service.email_service.send_action_reminder.assert_not_called()
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_lyrics_reminder_includes_review_url(self):
|
||||
"""Test that lyrics reminder includes correct review URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_lyrics.return_value = "Review"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="lyrics",
|
||||
audio_hash="hash123",
|
||||
review_token="token456",
|
||||
)
|
||||
|
||||
# Verify the review URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
|
||||
review_url = call_kwargs.get('review_url')
|
||||
assert "audioHash=hash123" in review_url
|
||||
assert "reviewToken=token456" in review_url
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_send_instrumental_reminder_includes_url(self):
|
||||
"""Test that instrumental reminder includes correct URL."""
|
||||
service = JobNotificationService()
|
||||
service.frontend_url = "https://gen.nomadkaraoke.com"
|
||||
service.backend_url = "https://api.nomadkaraoke.com"
|
||||
service.email_service = Mock()
|
||||
service.email_service.send_action_reminder.return_value = True
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_action_needed_instrumental.return_value = "Select"
|
||||
|
||||
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
|
||||
await service.send_action_reminder_email(
|
||||
job_id="job-123",
|
||||
user_email="user@example.com",
|
||||
action_type="instrumental",
|
||||
instrumental_token="inst-token",
|
||||
)
|
||||
|
||||
# Verify the instrumental URL was passed to template
|
||||
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
|
||||
instrumental_url = call_kwargs.get('instrumental_url')
|
||||
assert "instrumentalToken=inst-token" in instrumental_url
|
||||
|
||||
|
||||
class TestGetCompletionMessage:
|
||||
"""Tests for get_completion_message method."""
|
||||
|
||||
def test_get_completion_message_basic(self):
|
||||
"""Test basic completion message retrieval."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Your video is ready!"
|
||||
|
||||
result = service.get_completion_message(
|
||||
job_id="job-123",
|
||||
user_name="Test User",
|
||||
artist="Test Artist",
|
||||
title="Test Song",
|
||||
youtube_url="https://youtube.com/watch?v=123",
|
||||
dropbox_url="https://dropbox.com/folder/abc",
|
||||
)
|
||||
|
||||
assert result == "Your video is ready!"
|
||||
service.template_service.render_job_completion.assert_called_once()
|
||||
|
||||
def test_get_completion_message_includes_feedback_url(self):
|
||||
"""Test that completion message includes feedback URL."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
# Verify feedback URL was passed
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert 'feedback_url' in call_kwargs
|
||||
|
||||
def test_get_completion_message_with_defaults(self):
|
||||
"""Test completion message with default values."""
|
||||
service = JobNotificationService()
|
||||
service.template_service = Mock()
|
||||
service.template_service.render_job_completion.return_value = "Message"
|
||||
|
||||
service.get_completion_message(job_id="job-123")
|
||||
|
||||
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
|
||||
assert call_kwargs.get('job_id') == "job-123"
|
||||
# Other params should be None/default
|
||||
assert call_kwargs.get('name') is None
|
||||
assert call_kwargs.get('artist') is None
|
||||
|
||||
|
||||
class TestGlobalInstance:
|
||||
"""Tests for global instance management."""
|
||||
|
||||
def test_get_job_notification_service_returns_same_instance(self):
|
||||
"""Test that get_job_notification_service returns singleton."""
|
||||
# Reset global
|
||||
import backend.services.job_notification_service as jns
|
||||
jns._job_notification_service = None
|
||||
|
||||
service1 = get_job_notification_service()
|
||||
service2 = get_job_notification_service()
|
||||
|
||||
assert service1 is service2
|
||||
|
||||
def test_service_initializes_with_dependencies(self):
|
||||
"""Test that service initializes with email and template services."""
|
||||
service = JobNotificationService()
|
||||
|
||||
assert service.email_service is not None
|
||||
assert service.template_service is not None
|
||||
assert service.frontend_url is not None
|
||||
assert service.backend_url is not None
|
||||
Loaded 100 of 520 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user