Compare commits

..
Author SHA1 Message Date
Andrew Beveridge d0a657610b Added admin API script 2026-01-04 20:32:46 -05:00
977 changed files with 247425 additions and 67728 deletions

No files matched your search

-393
View File
@@ -1,393 +0,0 @@
# Plan: Admin Job Details Page
**Created:** 2026-01-09
**Branch:** feat/sess-20260109-1738-admin-job-details
**Status:** In Progress (Phase 1 Complete)
## Overview
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
## Requirements
### Must Have
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
- [ ] Download links for all existing files (generate signed URLs)
- [ ] View worker logs with filtering by level/worker
- [ ] Restart job from beginning (reset to PENDING)
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
### Should Have
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
- [ ] View/edit raw JSON fields (state_data, file_urls)
- [ ] Quick actions: send completion email, clear cache, impersonate user
### Nice to Have
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
- [ ] Diff view when editing fields
- [ ] Audit log of admin changes
## Technical Approach
### Routing Fix (COMPLETED)
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
- Uses `useSearchParams()` to detect `?id=` parameter
- Shows detail view when `selectedJobId` is present, list view otherwise
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
- Deleted old `/admin/jobs/detail/` directory
### Backend Enhancements
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
### Frontend Structure
The page will have collapsible sections:
1. **Header**: Job ID, status badge, artist/title, quick actions
2. **Overview Cards**: User, dates, progress, source info
3. **Stage Timeline**: Visual breakdown of each stage with times
4. **Files**: Downloadable links for all GCS files
5. **Logs**: Filterable worker logs
6. **Configuration**: Editable job settings
7. **Raw Data**: JSON views of state_data, file_urls, full job
### Job Reset Logic
Reset to specific states requires:
- Clearing downstream state_data keys
- Resetting status
- Optionally clearing associated files
- Adding timeline event for the reset
Valid reset targets:
| Target State | Clears | Use Case |
|--------------|--------|----------|
| PENDING | Everything except input | Full restart |
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
## Implementation Steps
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
### Phase 1: Fix Routing (COMPLETED)
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
2. [x] Combined list and detail views in single component
3. [x] Use `useSearchParams()` to detect `?id=` parameter
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
6. [x] Verified: All 9 E2E tests pass, build succeeds
### Phase 2: Backend - File Downloads with Signed URLs
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
10. [ ] Traverse job.file_urls recursively
11. [ ] Generate signed URLs for each GCS path
12. [ ] Return structured response with file metadata
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
### Phase 3: Frontend - Enhanced File Display
15. [ ] Create "Files" accordion section with grouped downloads
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
17. [ ] Show file name, type, download button
18. [ ] Handle missing/unavailable files gracefully
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
20. [ ] Verify: `npm run test:e2e` passes
### Phase 4: Backend - Job Update Endpoint
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
23. [ ] Define editable fields whitelist
24. [ ] Validate field values (e.g., status must be valid enum)
25. [ ] Log admin changes for audit trail
26. [ ] Verify: Backend tests pass
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
### Phase 5: Frontend - Editable Fields
28. [ ] Create "Configuration" section with editable fields
29. [ ] Use inline edit pattern (click to edit, enter to save)
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
31. [ ] Show validation errors inline
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
33. [ ] Verify: `npm run test:e2e` passes
### Phase 6: Backend - Job Reset Endpoint
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
36. [ ] Accept `target_state` parameter
37. [ ] Implement reset logic for each target state
38. [ ] Clear appropriate state_data keys
39. [ ] Optionally trigger workers after reset
40. [ ] Verify: Backend tests pass
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
### Phase 7: Frontend - Reset Actions
42. [ ] Add "Reset Job" dropdown/menu in header
43. [ ] Show available reset targets based on current state
44. [ ] Confirm dialog explaining what will be cleared
45. [ ] Show success/error toast after reset
46. [ ] **E2E Test**: Add reset confirmation dialog test
47. [ ] Verify: `npm run test:e2e` passes
### Phase 8: Enhanced Stage Timeline
48. [ ] Parse timeline events to calculate stage durations
49. [ ] Display stages as cards or timeline items
50. [ ] Show duration for each stage
51. [ ] Highlight current/blocking stages
52. [ ] **E2E Test**: Add timeline display test
### Phase 9: Final Verification
53. [ ] Run full backend test suite: `make test-backend`
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
55. [ ] Manual production testing checklist (see Testing Strategy)
56. [ ] Verify CI passes on PR
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
## Testing Strategy
Based on `docs/TESTING.md` guidelines, this feature requires:
### Test Types Required
| Type | Location | Purpose |
|------|----------|---------|
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
| Manual Production E2E | N/A | Final validation on staging/prod |
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
```python
def test_get_job_files_returns_signed_urls_for_all_files():
"""Test that all files in file_urls get signed URLs."""
def test_get_job_files_handles_nested_file_urls():
"""Test recursive traversal of stems.instrumental_clean, etc."""
def test_get_job_files_handles_missing_files_gracefully():
"""Test files that don't exist in GCS are omitted or marked."""
def test_get_job_files_requires_admin():
"""Test non-admin gets 403."""
def test_get_job_files_returns_404_for_missing_job():
"""Test invalid job_id returns 404."""
```
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
```python
def test_update_job_allows_editable_fields():
"""Test artist, title, user_email, theme_id can be updated."""
def test_update_job_rejects_non_editable_fields():
"""Test job_id, created_at, etc. cannot be updated."""
def test_update_job_validates_status_enum():
"""Test invalid status value returns 422."""
def test_update_job_validates_theme_id_exists():
"""Test non-existent theme_id returns 422."""
def test_update_job_requires_admin():
"""Test non-admin gets 403."""
def test_update_job_returns_updated_job():
"""Test response includes updated job data."""
```
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
```python
def test_reset_to_pending_clears_all_state_data():
"""Test full reset clears everything except input."""
def test_reset_to_audio_selection_clears_search_results():
"""Test audio selection reset clears audio_search_results."""
def test_reset_to_awaiting_review_clears_review_data():
"""Test review reset clears screens, videos, review data."""
def test_reset_to_instrumental_selection_clears_finals():
"""Test instrumental reset clears encoding results."""
def test_reset_adds_timeline_event():
"""Test reset adds admin action to timeline."""
def test_reset_invalid_target_returns_422():
"""Test invalid target_state returns validation error."""
def test_reset_requires_admin():
"""Test non-admin gets 403."""
def test_reset_from_complete_to_pending_works():
"""Test can reset completed job back to start."""
```
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
```typescript
import { test, expect } from '@playwright/test';
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
test.describe('Admin Job Detail Page', () => {
test.beforeEach(async ({ page }) => {
await setAdminAuthToken(page);
});
test('navigates from jobs list to detail page', async ({ page }) => {
// Mock jobs list and single job endpoints
await page.route('**/api/admin/jobs', route => route.fulfill({
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
}));
await page.route('**/api/jobs/test123', route => route.fulfill({
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
}));
await page.goto('/admin/jobs');
await page.click('text=test123');
await expect(page).toHaveURL('/admin/jobs/test123');
await expect(page.getByText('Job test123')).toBeVisible();
});
test('displays all job metadata sections', async ({ page }) => {
// Test overview cards, timeline, logs accordion, etc.
});
test('files section shows downloadable links', async ({ page }) => {
// Mock files endpoint, verify download buttons appear
});
test('edit field inline and save', async ({ page }) => {
// Click edit, change value, press Enter, verify PATCH called
});
test('reset job shows confirmation dialog', async ({ page }) => {
// Click reset, verify dialog, confirm, verify POST called
});
test('handles API errors gracefully', async ({ page }) => {
// Mock 500 response, verify error toast shown
});
});
```
### Test File Organization
```
backend/tests/
├── test_admin_job_detail.py # NEW - all backend tests for this feature
├── test_admin_api.py # Existing admin tests (keep separate)
└── conftest.py # Shared fixtures
frontend/e2e/
├── fixtures/
│ ├── test-helper.ts # Add setAdminAuthToken helper
│ └── mock-responses.ts # Add admin job detail mocks
└── regression/
└── admin-job-detail.spec.ts # NEW - E2E regression tests
```
### Mocking Guidelines
**Backend - What to mock:**
- `StorageService.generate_signed_url()` - return predictable test URLs
- `StorageService.file_exists()` - control which files "exist"
- Firestore operations via existing `mock_firestore` fixture
**Backend - What NOT to mock:**
- `JobManager` logic - test the real reset/update logic
- Validation logic - test real Pydantic validation
- Authorization checks - test real `require_admin` dependency
**Frontend - What to mock:**
- All API responses via Playwright route interception
- Time (if testing token expiry)
**Frontend - What NOT to mock:**
- React component behavior
- User interactions (click, type, etc.)
### CI Requirements
This feature will be validated by existing CI checks:
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
- `frontend-build` - Must pass (verifies new page compiles)
### Running Tests Locally
```bash
# Backend tests only (for this feature)
poetry run pytest backend/tests/test_admin_job_detail.py -v
# All backend tests
make test-backend
# Frontend E2E regression (includes new tests)
cd frontend && npm run test:e2e
# Single E2E test file
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
# With browser visible for debugging
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
```
### Manual Production Testing Checklist
After deployment, manually verify on production:
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
- [ ] All file download links work (valid signed URLs)
- [ ] Edit artist/title → refresh → changes persisted
- [ ] Reset job to pending → job status shows "pending"
- [ ] Logs section loads and filters work
- [ ] Error states handled (invalid job ID shows friendly error)
## Open Questions
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
- **Recommendation**: Delete it. No external links depend on it.
2. [ ] Should job reset delete GCS files or just clear state_data references?
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
3. [ ] Should edits be immediately saved or require explicit "Save" button?
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
## Rollback Plan
If issues arise:
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
2. Backend: New endpoints are additive, won't affect existing functionality
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
## Notes
- The existing detail page has good foundational code - we're enhancing rather than replacing
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
- Job reset is powerful - consider adding confirmation dialogs and logging
@@ -1,204 +0,0 @@
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
**Created:** 2026-01-19
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
**Status:** Draft
## Overview
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
This plan covers:
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
2. Archiving the GitHub repo
3. Deprecating the PyPI package (via README update published as final version)
4. Reorganizing vendored code to a permanent location in karaoke-gen
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
## Requirements
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
- [ ] python-lyrics-transcriber GitHub repo archived
- [ ] Final PyPI release with deprecation notice in README
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
- [ ] Tests pass after reorganization
- [ ] karaoke-gen pyproject.toml updated for new path
## Technical Approach
### New Path Structure
Move the vendored code to `libs/lyrics_transcriber/`:
```
karaoke-gen/
├── libs/
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
│ ├── __init__.py
│ ├── cli/
│ ├── core/
│ ├── correction/
│ ├── lyrics/
│ ├── output/
│ ├── review/
│ ├── storage/
│ ├── transcribers/
│ ├── types.py
│ └── utils/
├── karaoke_gen/
├── backend/
├── frontend/
└── ...
```
### Why `libs/lyrics_transcriber/` instead of alternatives?
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
### pyproject.toml Changes
```toml
packages = [
{ include = "karaoke_gen" },
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
{ include = "backend" }
]
include = [
"karaoke_gen/nextjs_frontend/out",
"karaoke_gen/nextjs_frontend/out/**/*",
"libs/lyrics_transcriber/output/fonts/*", # Updated path
"libs/lyrics_transcriber/output/cdgmaker/images/*",
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
]
```
### Files to Remove from Vendored Code
From current `lyrics_transcriber_temp/`:
- `pyproject.toml` - Package config for standalone release
- `poetry.lock` - Lock file for standalone package
- `README.md` - Will be replaced by deprecation notice
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
- `Dockerfile` - Standalone container build
- `.github/` - CI/CD for standalone package
- `.cursor/` - Cursor AI config
- `.specify/` - Specify config
- `.gitignore` - Not needed (karaoke-gen has its own)
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
- `scripts/` - Development scripts for standalone package
- `specs/` - Specification files
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
## Implementation Steps
### Phase 1: Update and Archive External Project (Manual Steps)
1. [ ] Clone python-lyrics-transcriber repo locally
2. [ ] Create deprecation notice README:
```markdown
# python-lyrics-transcriber (DEPRECATED)
**This project has been deprecated and archived.**
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
## Migration
For karaoke video generation with synchronized lyrics:
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
- Web app: https://gen.nomadkaraoke.com
## Historical Context
This library was originally developed as a standalone tool for creating synchronized
lyrics files. It has now been integrated into the karaoke-gen platform which provides
a complete end-to-end solution for karaoke video generation.
The last standalone version was 0.81.0. No further releases will be made to PyPI.
```
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
### Phase 2: Reorganize Vendored Code in karaoke-gen
5. [ ] Create `libs/` directory
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
8. [ ] Update `pyproject.toml` packages and include paths
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
### Phase 3: Verify and Test
10. [ ] Run `poetry install` to verify package configuration
11. [ ] Run `make test` to ensure all tests pass
12. [ ] Verify imports work:
```python
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
```
### Phase 4: Handle Tests from lyrics_transcriber
13. [ ] Evaluate lyrics_transcriber tests:
- **74 test files** in `lyrics_transcriber_temp/tests/`
- These test the internal lyrics_transcriber library functionality
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
- Skip integration/performance/manual tests that were for standalone development
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
| `pyproject.toml` | Modify | Update package paths |
| `scripts/benchmark_encoding.py` | Modify | Update import path |
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
## Testing Strategy
### Immediate Testing
- Run `make test` after reorganization - all existing tests must pass
- Verify Poetry can build/install the package correctly
### Test Migration Decision
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
- `unit/correction/` - Correction logic tests (important)
- `unit/lyrics/` - Lyrics provider tests
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
- `integration/` - Integration tests
- `performance/` - Performance benchmarks
**Recommendation:** Migrate a subset of critical unit tests, particularly:
- `test_corrector.py` - Core correction logic
- `test_anchor_sequence*.py` - Anchor sequence matching
- Handler tests in `handlers/`
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
## Open Questions
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
## Rollback Plan
If issues arise:
1. Git revert the consolidation commits
2. `lyrics_transcriber_temp/` can be restored from git history
3. The external python-lyrics-transcriber repo can be unarchived if needed
## Notes
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
- Only the internal project structure changes, not the API
- This is primarily a cleanup/organization task with some external repo admin work
@@ -1,218 +0,0 @@
# Plan: Job Email Notifications & Admin Message Tools
**Created:** 2026-01-06
**Branch:** feat/sess-20260106-0116-add-email-button
**Status:** Draft
## Overview
Add comprehensive email notification system for karaoke-gen that:
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
2. **Uses cloud-stored templates** for the job completion message
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
## Requirements
### Functional Requirements
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
- [ ] Store email templates in GCS (editable without code deploy)
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
### Non-Functional Requirements
- [ ] Email sending must not block job completion (async/background)
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
- [ ] Template changes should take effect without restart (no long-term caching)
## Technical Approach
### 1. Template Storage (GCS)
Store templates in `gs://karaoke-gen-storage/templates/`:
- `job-completion.txt` - Plain text template for job completion
- `job-completion.html` - HTML version (optional, for rich emails)
- `action-needed-lyrics.txt` - Reminder for lyrics review
- `action-needed-instrumental.txt` - Reminder for instrumental selection
**Template Variables:**
- `{name}` - User's display name or "there" if unknown
- `{youtube_url}` - YouTube video URL
- `{dropbox_url}` - Dropbox folder URL
- `{artist}` - Artist name
- `{title}` - Song title
- `{job_id}` - Job ID
- `{review_url}` - Lyrics review URL (for action-needed emails)
- `{instrumental_url}` - Instrumental selection URL
- `{feedback_url}` - Feedback form URL (for completion emails)
### 2. Email Service Extensions
Extend `backend/services/email_service.py`:
- Add `send_job_completion()` method with CC support
- Add `send_action_reminder()` method
- Add `render_template()` helper to fetch & render GCS templates
### 3. Auto-Email Triggers
**Job Completion:**
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
- Implementation: Call email service async in background task
**Action Reminders (5-min idle detection):**
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
- Record `blocking_state_entered_at` timestamp in `state_data`
- Schedule a Cloud Tasks task for 5 minutes later
- The scheduled task checks if job is still in blocking state and user hasn't started review
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
### 4. Admin UI (Frontend)
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
- "Copy Message" button - fetches rendered template, copies to clipboard
- "Send Email" button - opens dialog for email input, calls API to send
### 5. Backend API Endpoints
New endpoints in `backend/api/routes/admin.py`:
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
New internal endpoint for scheduled reminder:
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
## Implementation Steps
### Phase 1: Template Infrastructure
1. [ ] Upload initial template to GCS from user's existing template file
2. [ ] Create `template_service.py` to fetch/render templates from GCS
3. [ ] Add unit tests for template rendering
### Phase 2: Email Extensions
4. [ ] Extend `email_service.py` with CC support in `send_email()`
5. [ ] Add `send_job_completion()` method
6. [ ] Add `send_action_reminder()` method
7. [ ] Add unit tests for new email methods
### Phase 3: Auto-Email on Completion
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
9. [ ] Add integration test for completion email trigger
### Phase 4: Idle Reminder System
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
12. [ ] Schedule 5-minute delayed task when entering blocking states
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
14. [ ] Add tests for idle reminder flow
### Phase 5: Admin API
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
17. [ ] Add tests for admin endpoints
### Phase 6: Admin UI
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
19. [ ] Add "Copy Message" button (admin-only) with loading state
20. [ ] Add "Send Email" button (admin-only) with email input dialog
21. [ ] Add E2E test for admin buttons
### Phase 7: Documentation
22. [ ] Document template management in docs/DEVELOPMENT.md
23. [ ] Update docs/API.md with new endpoints
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
## Testing Strategy
**Unit Tests:**
- Template rendering with all variable combinations
- Email service CC support
- Completion/reminder email generation
- Idle detection logic
**Integration Tests:**
- Full flow: job completion → email sent
- Idle reminder: job enters blocking state → 5 min → reminder sent
- Admin endpoint authentication
**Manual Testing:**
- Verify emails render correctly in Gmail/Outlook
- Test admin buttons on completed job
- Test clipboard copy works across browsers
## Open Questions
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
## Rollback Plan
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
2. **Template fetch failures** - Fall back to hardcoded template in code
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
## Template Content
Initial template (from user's file, with feedback link added):
```
Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
```
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
## Updating Templates
To update email templates after deployment:
```bash
# Upload new template
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
# Templates are fetched fresh on each use (no caching in code)
# Changes take effect immediately
```
-317
View File
@@ -1,317 +0,0 @@
# Plan: White Label B2B Karaoke Portals
**Created:** 2026-01-08
**Branch:** feat/sess-20260108-0358-white-label-portal
**Status:** Draft
## Overview
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
**First customer: Vocal Star**
- Subdomain: `vocalstar.nomadkaraoke.com`
- Workflow: Upload original + instrumental audio (no audio search)
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
- Distribution: Download only (no Dropbox/GDrive/YouTube)
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
## Requirements
### Functional Requirements
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
- [ ] Per-tenant frontend branding (logo, colors, metadata)
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
- [ ] Per-tenant default theme (always use Vocal Star theme)
- [ ] Per-tenant allowed email domains for magic link auth
- [ ] Per-tenant fixed API token support (for automation)
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
- [ ] Users scoped to tenant
### Non-Functional Requirements
- [ ] Minimal code duplication - single codebase serves all tenants
- [ ] Easy to add new tenants via config, not code changes
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
- [ ] No performance impact on existing single-tenant flow
## Technical Approach
### Architecture Decision: Config-Driven Multi-Tenancy
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
2. **Frontend detects subdomain** and fetches tenant config from API
3. **Backend middleware** extracts tenant from request and applies config
4. **Firestore collections** get `tenant_id` field for isolation
This approach:
- Requires no database migrations for new tenants
- Configs can be managed via simple JSON files
- Easy to test locally with query param override (`?tenant=vocalstar`)
- Can evolve to admin UI later if needed
### Tenant Config Schema
```json
{
"id": "vocalstar",
"name": "Vocal Star",
"subdomain": "vocalstar.nomadkaraoke.com",
"branding": {
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
"primary_color": "#ffff00",
"secondary_color": "#006CF9",
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
"site_title": "Vocal Star Karaoke Generator"
},
"features": {
"audio_search": false,
"file_upload": true,
"youtube_upload": false,
"dropbox_upload": false,
"gdrive_upload": false,
"theme_selection": false
},
"defaults": {
"theme_id": "vocalstar",
"distribution_mode": "download_only"
},
"auth": {
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
"fixed_tokens": ["<will be in Secret Manager>"]
}
}
```
### Key Implementation Details
#### 1. Subdomain Detection (Frontend)
- Next.js middleware checks `request.headers.host`
- If subdomain matches tenant, store tenant ID in cookie/context
- Fetch `/api/tenant/config` to get branding/features
- Apply CSS variables dynamically for colors
- Swap logo component based on config
#### 2. Tenant Middleware (Backend)
- FastAPI middleware extracts tenant from:
- `X-Tenant-ID` header (set by frontend)
- Or subdomain from `Host` header
- Or query param `?tenant=` (dev only)
- Attaches `tenant_id` to request state
- Tenant config loaded from GCS (cached 5 min like themes)
#### 3. Auth Enhancement
- `AuthService.validate_token_full()` already returns `user_email`
- Add tenant validation: check if user's email domain is in tenant's allowed list
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
- Magic links include tenant context for proper redirect
#### 4. Data Isolation
- Add `tenant_id: Optional[str]` to User and Job models
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
- Query filters: `where('tenant_id', '==', tenant_id or None)`
- No schema migration needed - new field, optional
#### 5. Theme Integration
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
- Tenant config specifies `default_theme_id: "vocalstar"`
- When `theme_selection: false`, UI hides theme picker and uses default
## Implementation Steps
### Phase 1: Backend Tenant Foundation (Day 1-2)
1. [ ] **Create tenant model and config loader**
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
- Create `backend/services/tenant_service.py` with GCS config loading and caching
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
2. [ ] **Add tenant middleware**
- Create `backend/api/middleware/tenant.py`
- Extract tenant from subdomain or header
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
- Add `get_tenant()` dependency for routes
3. [ ] **Update data models**
- Add `tenant_id: Optional[str] = None` to `User` model
- Add `tenant_id: Optional[str] = None` to `Job` model
- Update Firestore indexes in Pulumi for tenant-scoped queries
4. [ ] **Update auth for tenant awareness**
- Modify `AuthService` to validate email domain against tenant config
- Add tenant context to magic link generation
- Support per-tenant fixed tokens in `auth_tokens` collection
### Phase 2: Upload Vocal Star Theme (Day 2)
5. [ ] **Prepare Vocal Star theme for GCS**
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
- Upload background images to `themes/vocalstar/assets/`
- Upload Oswald font to `themes/vocalstar/assets/`
- Upload `style_params.json` to `themes/vocalstar/`
- Create preview image and upload to `themes/vocalstar/preview.png`
- Add Vocal Star to `themes/_metadata.json`
6. [ ] **Create Vocal Star tenant config**
- Create `tenants/vocalstar/config.json` in GCS
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
- Upload favicon to `tenants/vocalstar/favicon.ico`
### Phase 3: Frontend Tenant Support (Day 3-4)
7. [ ] **Add tenant context provider**
- Create `frontend/lib/tenant.ts` with tenant config types
- Create `frontend/hooks/useTenant.ts` for tenant state
- Create `TenantProvider` component that fetches config on mount
- Detect subdomain in `_app.tsx` or middleware
8. [ ] **Dynamic branding**
- Create `TenantLogo` component that renders tenant or default logo
- Update `globals.css` to support CSS variable overrides
- Add `applyTenantColors()` function to set CSS variables from config
- Update page metadata (title, favicon) from tenant config
9. [ ] **Feature-flagged UI**
- Wrap audio search in `{tenant.features.audio_search && ...}`
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
- Wrap distribution options based on tenant features
- Show simplified form for Vocal Star (just file upload fields)
10. [ ] **Update magic link flow**
- Pass tenant context when requesting magic link
- Magic link redirect goes to tenant subdomain
- Verify endpoint extracts tenant from link context
### Phase 4: Backend Feature Enforcement (Day 4-5)
11. [ ] **Enforce tenant features in API**
- Audio search endpoint: return 403 if `audio_search: false`
- File upload: check `file_upload: true`
- Distribution settings: ignore/error if disabled features requested
- Theme selection: use default if `theme_selection: false`
12. [ ] **Job creation with tenant context**
- `JobCreate` handler sets `tenant_id` from request state
- Distribution config respects tenant defaults
- Theme defaults to tenant's default theme
13. [ ] **Query scoping**
- Update `JobManager.get_jobs()` to filter by tenant_id
- Update `UserService` queries to include tenant_id
- Admin endpoints bypass tenant filter (see all tenants)
### Phase 5: Infrastructure & Deployment (Day 5-6)
14. [ ] **DNS setup**
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
- Configure Cloudflare Pages to accept wildcard subdomain
- Add SSL certificate for subdomain
15. [ ] **Backend CORS update**
- Add `vocalstar.nomadkaraoke.com` to allowed origins
- Or use pattern matching for `*.nomadkaraoke.com`
16. [ ] **Pulumi updates**
- Add Firestore indexes for tenant_id queries
- Add any new secrets (tenant fixed tokens)
### Phase 6: Testing & Documentation (Day 6-7)
17. [ ] **Testing**
- Unit tests for TenantService config loading
- Unit tests for tenant middleware
- E2E test for Vocal Star flow (upload audio, generate, download)
- Test magic link works for @vocal-star.com email
18. [ ] **Documentation**
- Update CLAUDE.md with tenant info
- Create `docs/WHITE-LABEL.md` with tenant setup guide
- Document how to add new tenants
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/tenant.py` | Create | Tenant config models |
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
| `backend/models/user.py` | Modify | Add `tenant_id` field |
| `backend/models/job.py` | Modify | Add `tenant_id` field |
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
| `backend/main.py` | Modify | Add tenant middleware |
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
## Testing Strategy
### Unit Tests
- `TenantService`: Config loading, caching, validation
- `TenantMiddleware`: Subdomain extraction, header extraction
- `AuthService`: Tenant email domain validation
- `JobManager`: Tenant-scoped queries
### Integration Tests (Emulator)
- Create job with tenant context
- Query jobs filtered by tenant
- Auth with tenant-allowed email domain
- Auth rejection for wrong email domain
### E2E Tests
- Vocal Star happy path: Upload audio files, review lyrics, download video
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
- Feature hiding: Verify audio search not shown on Vocal Star portal
## Open Questions
- [x] **Q: Should tenant configs be in GCS or Firestore?**
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
- [x] **Q: How to handle localhost development for specific tenant?**
A: Query param override `?tenant=vocalstar` for local testing
- [x] **Q: Should admin portal be tenant-aware?**
A: No - admins see all tenants. Admin routes bypass tenant filter.
- [x] **Q: Do we need tenant-specific email sender identity?**
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
## Rollback Plan
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
1. Revert frontend to not check tenant
2. Revert backend middleware to not extract tenant
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
4. No schema migration to reverse
For DNS: Simply remove CNAME record if needed.
## Success Criteria
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
- [ ] Portal shows Vocal Star branding (logo, colors)
- [ ] Audio search is hidden; only file upload available
- [ ] Theme is locked to Vocal Star style
- [ ] Distribution options hidden; download only
- [ ] Magic link works for @vocal-star.com emails
- [ ] Fixed API token works for automation
- [ ] Generated videos use Vocal Star theme
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
@@ -1,115 +0,0 @@
# Build Encoding Worker GCE Image
#
# Rebuilds the custom Packer image when provisioning changes.
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
#
# Triggers:
# - Push to main with changes to infrastructure/packer/**
# - Manual dispatch (workflow_dispatch)
#
# After image build, the encoding-worker VM will use the new image
# on next recreation (e.g., via pulumi up or manual restart).
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
inputs:
python_version:
description: 'Python version to install'
required: false
default: '3.13.1'
concurrency:
group: build-encoding-worker-image
cancel-in-progress: false # Don't cancel in-progress builds
jobs:
build-image:
name: Build Packer Image
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # Required for Workload Identity Federation
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Packer
run: |
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y packer
- name: Initialize Packer
working-directory: infrastructure/packer
run: packer init encoding-worker.pkr.hcl
- name: Validate Packer Template
working-directory: infrastructure/packer
run: packer validate encoding-worker.pkr.hcl
- name: Build Image
working-directory: infrastructure/packer
env:
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
run: |
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
packer build \
-var "python_version=${PYTHON_VERSION}" \
encoding-worker.pkr.hcl
- name: Get Latest Image Info
id: image-info
run: |
IMAGE_NAME=$(gcloud compute images list \
--project=nomadkaraoke \
--filter="family:encoding-worker" \
--sort-by="~creationTimestamp" \
--limit=1 \
--format="value(name)")
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
echo "Built image: ${IMAGE_NAME}"
- name: Summary
run: |
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
notify-on-failure:
name: Notify on Failure
runs-on: ubuntu-latest
needs: build-image
if: failure()
steps:
- name: Log Failure
run: |
echo "::error::Encoding worker image build failed. Check the build-image job logs."
+60 -262
View File
@@ -67,29 +67,6 @@ jobs:
# FRONTEND TESTS (PRs only - main skips tests)
# ============================================
frontend-lint:
name: "Frontend - Lint (Rules of Hooks)"
runs-on: ubuntu-latest
needs: changes
if: github.ref != 'refs/heads/main' && needs.changes.outputs.frontend == 'true'
steps:
- uses: actions/checkout@v4
- name: Setup Node.js with caching
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Run ESLint
working-directory: frontend
run: npm run lint
frontend-unit-tests:
name: "Frontend - Unit Tests"
runs-on: ubuntu-latest
@@ -252,7 +229,6 @@ jobs:
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ secrets.E2E_TEST_TOKEN }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
run: |
echo "Starting E2E Happy Path Test against Production"
echo "Time: $(date)"
@@ -679,14 +655,12 @@ jobs:
runs-on: ubuntu-latest
# Note: frontend-e2e-full is optional (requires 'e2e' label) so not included here
# frontend-e2e-smoke IS required - fast smoke tests catch critical regressions
# frontend-lint IS required - catches React hooks violations that cause runtime crashes
needs: [frontend-lint, frontend-unit-tests, frontend-e2e-smoke, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
needs: [frontend-unit-tests, frontend-e2e-smoke, frontend-build, package-unit-tests, package-integration-tests, backend-unit-tests, backend-emulator-tests, backend-lint]
if: github.ref != 'refs/heads/main' && always()
steps:
- name: Check results
run: |
results=(
"${{ needs.frontend-lint.result }}"
"${{ needs.frontend-unit-tests.result }}"
"${{ needs.frontend-e2e-smoke.result }}"
"${{ needs.frontend-build.result }}"
@@ -866,23 +840,51 @@ jobs:
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
# ===== Build Unified Frontend (Next.js) =====
# NOTE: Lyrics review and instrumental review UIs were consolidated into
# the main Next.js frontend (PR #267, Jan 2025). The unified frontend
# serves all three UIs from a single codebase at:
# - / (main app)
# - /app/jobs/{id}/lyrics (lyrics review)
# - /app/jobs/{id}/instrumental (instrumental review)
- name: Install frontend dependencies
- name: Enable Yarn (for lyrics frontend)
run: corepack enable
# ===== Build Main Frontend (Next.js) =====
- name: Install main frontend dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Build frontend (Next.js static export)
- name: Build main frontend (Next.js static export)
working-directory: frontend
env:
NEXT_PUBLIC_API_URL: https://api.nomadkaraoke.com
run: npm run build
# ===== Build Lyrics Review Frontend (Vite) =====
- name: Cache Yarn dependencies (lyrics frontend)
uses: actions/cache@v4
with:
path: lyrics_transcriber_temp/lyrics_transcriber/frontend/node_modules
key: yarn-lyrics-${{ runner.os }}-${{ hashFiles('lyrics_transcriber_temp/lyrics_transcriber/frontend/yarn.lock') }}
restore-keys: |
yarn-lyrics-${{ runner.os }}-
- name: Install lyrics frontend dependencies
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
run: yarn install --frozen-lockfile
- name: Build lyrics frontend (Vite)
working-directory: lyrics_transcriber_temp/lyrics_transcriber/frontend
env:
# Set base path for subpath deployment
VITE_BASE_PATH: /lyrics/
run: yarn build --base=/lyrics/
- name: Copy lyrics frontend to main output
run: |
mkdir -p frontend/out/lyrics
cp -r lyrics_transcriber_temp/lyrics_transcriber/frontend/dist/* frontend/out/lyrics/
# ===== Copy Instrumental Review UI (Static HTML) =====
- name: Copy instrumental review UI to main output
run: |
mkdir -p frontend/out/instrumental
cp karaoke_gen/instrumental_review/static/index.html frontend/out/instrumental/
- name: Setup Pages
uses: actions/configure-pages@v4
@@ -901,14 +903,11 @@ jobs:
deploy-backend:
name: "Deploy - Backend (Cloud Run)"
needs: [package-build]
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
outputs:
base_rebuilt: ${{ steps.check-base.outputs.needs_rebuild }}
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
@@ -966,64 +965,6 @@ jobs:
with:
version: "latest"
# ============================================
# Upload wheel to GCS for GCE encoding worker
# The GCE worker installs this wheel at job start for hot code updates
# ============================================
- name: Download wheel artifact
uses: actions/download-artifact@v4
with:
name: package-dist
path: dist/
- name: Upload wheel to GCS for GCE worker
run: |
echo "📦 Uploading wheel to GCS for GCE encoding worker..."
WHEEL_FILE=$(ls dist/*.whl | head -1)
if [ -z "$WHEEL_FILE" ]; then
echo "::error::No wheel file found in dist/"
exit 1
fi
echo "Found wheel: $WHEEL_FILE"
# Upload to GCS (GCE worker downloads latest at job start)
gsutil cp "$WHEEL_FILE" gs://karaoke-gen-storage-${{ env.PROJECT_ID }}/wheels/
echo "✅ Wheel uploaded to GCS"
- name: Restart GCE Encoding Worker
run: |
echo "🔄 Restarting encoding worker to pick up new wheel..."
# Use IAP tunneling - works directly with service account credentials
# (avoids SSH key propagation delays that cause silent failures)
gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="sudo systemctl restart encoding-worker"
echo "⏳ Waiting for service to start..."
sleep 15
# Verify the restart worked by checking the wheel version
NEW_VERSION=$(gcloud compute ssh encoding-worker \
--zone=us-central1-c \
--project=nomadkaraoke \
--tunnel-through-iap \
--command="curl -s http://localhost:8080/health | grep -o '\"wheel_version\":\"[^\"]*\"' | cut -d'\"' -f4")
EXPECTED_VERSION=$(cat pyproject.toml | grep 'version = ' | head -1 | cut -d'"' -f2)
if [ "$NEW_VERSION" = "$EXPECTED_VERSION" ]; then
echo "✅ Encoding worker restarted successfully (v$NEW_VERSION)"
else
echo "⚠️ Version mismatch: encoder has v$NEW_VERSION, expected v$EXPECTED_VERSION"
echo " This may indicate the restart failed or the wheel wasn't downloaded"
# Don't fail the build - the encoder will still work, just with old version
fi
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
@@ -1049,32 +990,32 @@ jobs:
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Fetch hash label directly from registry (10KB config blob, not full image)
# This avoids issues with stale local Docker cache on GitHub runners
# Try to get the existing base image's hash label
# First check if image exists locally (fast, no network call)
# Only pull from registry if not cached locally
EXISTING_HASH=""
# Get the manifest (small JSON, ~1KB) - includes config digest
MANIFEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null || echo "")
CONFIG_DIGEST=$(echo "$MANIFEST" | jq -r '.config.digest // ""' 2>/dev/null || echo "")
if [ -n "$CONFIG_DIGEST" ] && [ "$CONFIG_DIGEST" != "null" ] && [ "$CONFIG_DIGEST" != "" ]; then
echo "📋 Found manifest, config digest: $CONFIG_DIGEST"
# Fetch just the config blob (~10KB) to get labels - much faster than pulling full image
TOKEN=$(gcloud auth print-access-token)
CONFIG_JSON=$(curl -sL -H "Authorization: Bearer $TOKEN" \
"https://${{ env.REGISTRY }}/v2/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}/blobs/$CONFIG_DIGEST" 2>/dev/null)
EXISTING_HASH=$(echo "$CONFIG_JSON" | jq -r '.config.Labels["base.content.hash"] // .config.Labels["poetry.lock.hash"] // ""' 2>/dev/null || echo "")
echo "📋 Registry base image hash: $EXISTING_HASH"
if docker image inspect $BASE_IMAGE >/dev/null 2>&1; then
echo "✅ Base image found in local cache (skipping slow registry pull)"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
elif docker pull $BASE_IMAGE 2>/dev/null; then
echo "📥 Base image pulled from registry"
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "base.content.hash"}}' 2>/dev/null || echo "")
# Fallback to old label name for backwards compatibility
if [ -z "$EXISTING_HASH" ]; then
EXISTING_HASH=$(docker inspect $BASE_IMAGE --format='{{index .Config.Labels "poetry.lock.hash"}}' 2>/dev/null || echo "")
fi
echo "Existing base image hash: $EXISTING_HASH"
else
echo "⚠️ No existing base image found in registry - will build"
echo "⚠️ No existing base image found - will build"
fi
if [ "${{ steps.base-hash.outputs.hash }}" != "$EXISTING_HASH" ]; then
echo "🔄 Base image needs rebuild (poetry.lock or Dockerfile.base changed)"
echo " Current: ${{ steps.base-hash.outputs.hash }}"
echo " Registry: $EXISTING_HASH"
echo "needs_rebuild=true" >> $GITHUB_OUTPUT
else
echo "✅ Base image is up to date"
@@ -1102,31 +1043,6 @@ jobs:
# Provenance disabled for smaller image and faster push
provenance: false
- name: Ensure local base image is current (if not rebuilt)
if: steps.check-base.outputs.needs_rebuild == 'false'
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
# Get the config digest from registry manifest (identifies the image content)
# This is the same value as Docker's Image ID for the same image
REGISTRY_CONFIG_DIGEST=$(docker manifest inspect $BASE_IMAGE 2>/dev/null | jq -r '.config.digest' || echo "")
# Get local image's ID (which is the config digest)
LOCAL_IMAGE_ID=$(docker image inspect $BASE_IMAGE --format='{{.Id}}' 2>/dev/null || echo "")
echo "Registry config digest: $REGISTRY_CONFIG_DIGEST"
echo "Local image ID: $LOCAL_IMAGE_ID"
if [ -z "$LOCAL_IMAGE_ID" ]; then
echo "📥 Base image not in local cache - pulling from registry"
docker pull $BASE_IMAGE
elif [ "$LOCAL_IMAGE_ID" != "$REGISTRY_CONFIG_DIGEST" ]; then
echo "🔄 Local cache has different image - pulling latest"
docker pull $BASE_IMAGE
else
echo "✅ Local cache matches registry - skipping pull"
fi
- name: Build and push app image
uses: docker/build-push-action@v6
with:
@@ -1164,16 +1080,13 @@ jobs:
--memory 16Gi \
--cpu 8 \
--cpu-boost \
--no-cpu-throttling \
--execution-environment gen2 \
--timeout 1800 \
--concurrency 1 \
--max-instances 50 \
--min-instances 4 \
--vpc-connector projects/nomadkaraoke/locations/us-central1/connectors/cloud-run-connector \
--vpc-egress private-ranges-only \
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true,USE_GCE_PREVIEW_ENCODING=true,ENABLE_PUSH_NOTIFICATIONS=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest,VAPID_PUBLIC_KEY=vapid-public-key:latest,VAPID_PRIVATE_KEY=vapid-private-key:latest,E2E_BYPASS_KEY=e2e-bypass-key:latest"
--set-env-vars "GOOGLE_CLOUD_PROJECT=${{ env.PROJECT_ID }},GCS_BUCKET_NAME=karaoke-gen-storage-${{ env.PROJECT_ID }},FIRESTORE_COLLECTION=jobs,ENVIRONMENT=production,AUDIO_SEPARATOR_API_URL=https://nomadkaraoke--audio-separator-api.modal.run,KARAOKE_GEN_VERSION=$VERSION,DEFAULT_DROPBOX_PATH=/MediaUnsynced/Karaoke/Tracks-Organized,DEFAULT_GDRIVE_FOLDER_ID=1laRKAyxo0v817SstfM5XkpbWiNKNAMSX,ENABLE_CLOUD_TASKS=true,CLOUD_RUN_SERVICE_URL=https://api.nomadkaraoke.com,GCP_REGION=us-central1,DEFAULT_ENABLE_YOUTUBE_UPLOAD=true,DEFAULT_BRAND_PREFIX=NOMAD,USE_GCE_ENCODING=true" \
--set-secrets "AUDIOSHAKE_API_TOKEN=audioshake-api-key:latest,GENIUS_API_TOKEN=genius-api-key:latest,SPOTIFY_COOKIE_SP_DC=spotify-cookie:latest,RAPIDAPI_KEY=rapidapi-key:latest,DEFAULT_DISCORD_WEBHOOK_URL=discord-releases-webhook:latest,ADMIN_TOKENS=admin-tokens:latest,RED_API_KEY=red-api-key:latest,RED_API_URL=red-api-url:latest,OPS_API_KEY=ops-api-key:latest,OPS_API_URL=ops-api-url:latest,FLACFETCH_API_URL=flacfetch-api-url:latest,FLACFETCH_API_KEY=flacfetch-api-key:latest,SENDGRID_API_KEY=sendgrid-api-key:latest,STRIPE_SECRET_KEY=stripe-secret-key:latest,STRIPE_WEBHOOK_SECRET=stripe-webhook-secret:latest,LANGFUSE_PUBLIC_KEY=langfuse-public-key:latest,LANGFUSE_SECRET_KEY=langfuse-secret-key:latest,LANGFUSE_HOST=langfuse-host:latest,ENCODING_WORKER_URL=encoding-worker-url:latest,ENCODING_WORKER_API_KEY=encoding-worker-api-key:latest"
- name: Verify deployment and version
run: |
@@ -1226,118 +1139,3 @@ jobs:
echo ""
echo "📊 Built on self-hosted GCP runner (200GB SSD, no disk space issues)"
echo " Runner: gcp-runner with 4 vCPU, 16GB RAM"
# ============================================
# PROPAGATE BASE IMAGE TO ALL RUNNERS
# When base image is rebuilt, pull it to all 20 runners
# so subsequent app builds don't wait 8 min for the pull
#
# NOTE: This is best-effort distribution. GitHub Actions doesn't
# support targeting specific runners by name, so we spawn 20 jobs
# and rely on the scheduler to distribute them across idle runners.
# In practice, with all runners idle after deploy-backend, distribution
# is good. Runners that miss this pull will get the image on their
# next app build (with the 8 min delay).
# ============================================
propagate-base-image:
name: "Propagate Base Image (${{ matrix.runner_index }})"
needs: deploy-backend
if: github.ref == 'refs/heads/main' && github.event_name == 'push' && needs.deploy-backend.outputs.base_rebuilt == 'true'
runs-on: [self-hosted, linux, gcp]
permissions:
contents: read
id-token: write
strategy:
matrix:
# 20 parallel jobs - GitHub distributes across available runners
runner_index: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
max-parallel: 20
fail-fast: false # Don't cancel other runners if one fails
env:
REGISTRY: us-central1-docker.pkg.dev
PROJECT_ID: nomadkaraoke
REPOSITORY: karaoke-repo
BASE_IMAGE_NAME: karaoke-backend-base
steps:
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Configure Docker for Artifact Registry
run: gcloud auth configure-docker ${{ env.REGISTRY }} --quiet
- name: Pull latest base image
run: |
BASE_IMAGE="${{ env.REGISTRY }}/${{ env.PROJECT_ID }}/${{ env.REPOSITORY }}/${{ env.BASE_IMAGE_NAME }}:latest"
RUNNER_NAME=$(hostname)
echo "📥 Pulling latest base image on runner $RUNNER_NAME..."
docker pull $BASE_IMAGE
echo "✅ Base image cached on $RUNNER_NAME"
deploy-infrastructure:
name: "Deploy - Infrastructure (Pulumi)"
runs-on: [self-hosted, linux, gcp]
if: github.ref == 'refs/heads/main' && github.event_name == 'push'
permissions:
contents: read
id-token: write
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: '3.13'
- name: Install Pulumi
run: |
curl -fsSL https://get.pulumi.com | sh
echo "$HOME/.pulumi/bin" >> $GITHUB_PATH
- name: Install Python dependencies
working-directory: infrastructure
run: |
pip install pulumi pulumi-gcp pulumi-docker
- name: Configure Pulumi
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
pulumi login
- name: Deploy Infrastructure
working-directory: infrastructure
env:
PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
run: |
echo "🚀 Deploying infrastructure changes..."
# Skip preview to avoid needing broad read permissions on all GCP resources
# Pulumi state is authoritative since infrastructure is only managed via Pulumi
pulumi up --yes --skip-preview --stack nomadkaraoke/karaoke-gen-infrastructure/prod
- name: Summary
if: success()
run: |
echo "🎉 Infrastructure deployment successful!"
echo "Commit: ${{ github.sha }}"
echo "Branch: ${{ github.ref_name }}"
-1
View File
@@ -74,7 +74,6 @@ jobs:
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
+2 -1
View File
@@ -58,7 +58,8 @@ frontend/.eslintcache
# Storybook
frontend/storybook-static/
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
# PWA files
frontend/**/public/sw.js
frontend/**/public/workbox-*.js
frontend/**/public/swe-worker-*.js
+9 -28
View File
@@ -16,11 +16,8 @@
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
## Tech Stack
@@ -37,38 +34,22 @@
- Create PR with summary, changes, testing info
- Merge only after CI passes
### Testing (Required)
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
- Which test types are needed (unit, integration, E2E)
- Where tests should be placed
- What mocking approach to use
- Coverage expectations
- **Production E2E tests for user-facing features** - see below
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
**Before committing**, run all tests:
### Testing (Required Before Commit)
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
### Infrastructure
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
## API Authentication (for Agents)
@@ -105,11 +86,11 @@ EOF
### GCE Encoding Worker
```bash
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
+12 -38
View File
@@ -1,50 +1,34 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make test - Run all tests (unit + backend + emulator-based)"
@echo " make test-unit - Run unit tests only (karaoke_gen)"
@echo " make test-backend - Run backend unit tests only"
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
@echo " make test-all - Alias for 'make test'"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@echo ""
@echo "Before committing, run: make test"
# Install dependencies (only if needed)
install-backend:
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
echo "=== Installing backend dependencies ==="; \
poetry install; \
fi
install-frontend:
@if [ ! -d "frontend/node_modules" ]; then \
echo "=== Installing frontend dependencies ==="; \
cd frontend && npm install; \
fi
install: install-backend install-frontend
# Run unit tests for karaoke_gen package
test-unit: install-backend
test-unit:
@echo "=== Running karaoke_gen unit tests ==="
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
# Run backend unit tests
test-backend:
@echo "=== Running backend unit tests ==="
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
test-e2e:
@echo "=== Running E2E integration tests with emulators ==="
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
@@ -58,18 +42,8 @@ test-e2e: install-backend
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
# Run all backend tests (unit + emulator)
test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend tests (unit + E2E)
test-frontend: install-frontend
@echo "=== Running frontend tests ==="
cd frontend && npm run test:all
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
# Run all tests (use this before committing!)
test: test-unit test-backend test-e2e
@echo ""
@echo "✅ All tests passed! Ready to commit."
-3
View File
@@ -159,9 +159,6 @@ karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"
# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
+3 -2
View File
@@ -22,9 +22,10 @@ WORKDIR /app
COPY backend /app/backend
# Copy and reinstall karaoke_gen from local source
# This ensures new modules are included without rebuilding base image
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
# This ensures new modules (like style_loader.py) are included without rebuilding base image
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY pyproject.toml README.md LICENSE /app/
RUN pip install --no-cache-dir -e .
-1
View File
@@ -54,7 +54,6 @@ WORKDIR /app
COPY pyproject.toml README.md LICENSE /app/
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
File diff suppressed because it is too large. Load diff
+45 -57
View File
@@ -15,13 +15,12 @@ import asyncio
import logging
import os
import tempfile
from typing import Optional, List, Dict, Any
from typing import Optional, List, Dict, Any, Tuple
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
from pydantic import BaseModel, Field, validator
from pydantic import BaseModel, Field
from backend.models.job import JobCreate, JobStatus
from karaoke_gen.utils import normalize_text
from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.worker_service import get_worker_service
@@ -33,13 +32,10 @@ from backend.services.audio_search_service import (
NoResultsError,
DownloadError,
)
from backend.services.theme_service import get_theme_service
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.config import get_settings
from backend.version import VERSION
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
from pathlib import Path
logger = logging.getLogger(__name__)
@@ -99,7 +95,7 @@ class AudioSearchRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -111,10 +107,6 @@ class AudioSearchRequest(BaseModel):
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
# Display overrides (optional - if empty, search values are used for display)
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
# Audio separation model configuration
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
@@ -128,13 +120,6 @@ class AudioSearchRequest(BaseModel):
# Non-interactive mode
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
def normalize_text_fields(cls, v):
"""Normalize text fields to standardize Unicode characters."""
if v is not None and isinstance(v, str):
return normalize_text(v)
return v
class AudioSearchResultResponse(BaseModel):
"""A single audio search result.
@@ -202,6 +187,35 @@ class AudioSelectResponse(BaseModel):
selected_provider: str
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Explicit values override defaults, None uses default
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
"""Extract metadata from request for job tracking."""
headers = dict(request.headers)
@@ -452,25 +466,17 @@ async def search_audio(
):
"""
Search for audio by artist and title, creating a new job.
This endpoint:
1. Creates a job in PENDING state
2. Searches for audio using flacfetch
3. Either returns search results for user selection, or
4. If auto_download=True, automatically selects best and starts processing
Use cases:
- Interactive mode (default): Returns results, user calls /select endpoint
- Auto mode (auto_download=True): Automatically selects and downloads best
"""
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.audio_search:
raise HTTPException(
status_code=403,
detail="Audio search is not available for this portal"
)
try:
# Apply default distribution settings
settings = get_settings()
@@ -479,8 +485,7 @@ async def search_audio(
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
# Apply defaults for YouTube/Dropbox distribution (for web service)
# Use explicit value if provided, otherwise fall back to server default
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
effective_enable_youtube_upload = body.enable_youtube_upload or settings.default_enable_youtube_upload
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
effective_youtube_description = body.youtube_description or settings.default_youtube_description
@@ -516,34 +521,19 @@ async def search_audio(
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="audio_search")
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
effective_theme_id = body.theme_id
if effective_theme_id is None:
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme: {effective_theme_id}")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
effective_theme_id, body.enable_cdg, body.enable_txt
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Use authenticated user's email
effective_user_email = auth_result.user_email
# Determine display values - use display_* if provided, otherwise fall back to search values
# Display values are used for title screens, filenames, YouTube, etc.
# Search values (body.artist, body.title) are used for audio search
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
effective_display_title = body.display_title.strip() if body.display_title else body.title
# Create job
job_create = JobCreate(
artist=effective_display_artist, # Display value for title screens, filenames
title=effective_display_title, # Display value for title screens, filenames
theme_id=effective_theme_id,
artist=body.artist,
title=body.title,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
@@ -566,10 +556,8 @@ async def search_audio(
auto_download=body.auto_download,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
job_id = job.job_id
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
@@ -584,11 +572,11 @@ async def search_audio(
# If theme is set and no custom style files are being uploaded, prepare theme style now
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
# can access the style configuration for preview videos
if effective_theme_id and not body.style_files:
if body.theme_id and not body.style_files:
from backend.api.routes.file_upload import _prepare_theme_for_job
try:
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, body.color_overrides
job_id, body.theme_id, body.color_overrides
)
theme_update = {
'style_params_gcs_path': style_params_path,
@@ -597,9 +585,9 @@ async def search_audio(
if youtube_desc and not effective_youtube_description:
theme_update['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, theme_update)
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
except Exception as e:
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
logger.warning(f"Failed to prepare theme '{body.theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Handle style file uploads if provided
+118 -153
View File
@@ -15,6 +15,7 @@ import json
import logging
import tempfile
import os
from dataclasses import dataclass
from fastapi import APIRouter, UploadFile, File, Form, HTTPException, BackgroundTasks, Request, Body, Depends
from pathlib import Path
from typing import Optional, List, Dict, Any, Tuple
@@ -28,17 +29,11 @@ from backend.services.storage_service import StorageService
from backend.services.worker_service import get_worker_service
from backend.services.credential_manager import get_credential_manager, CredentialStatus
from backend.services.theme_service import get_theme_service
from backend.services.job_defaults_service import (
get_effective_distribution_settings,
resolve_cdg_txt_defaults,
EffectiveDistributionSettings,
)
from backend.config import get_settings
from backend.version import VERSION
from backend.services.metrics import metrics
from backend.api.dependencies import require_auth
from backend.services.auth_service import UserType, AuthResult
from backend.middleware.tenant import get_tenant_config_from_request
logger = logging.getLogger(__name__)
router = APIRouter(tags=["jobs"])
@@ -74,7 +69,7 @@ class CreateJobFromUrlRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -128,7 +123,7 @@ class CreateJobWithUploadUrlsRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
webhook_url: Optional[str] = Field(None, description="Generic webhook URL")
@@ -268,6 +263,75 @@ async def _trigger_audio_worker_only(job_id: str) -> None:
await worker_service.trigger_audio_worker(job_id)
def _resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool],
enable_txt: Optional[bool]
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
When a theme is selected, CDG and TXT are enabled by default.
Explicit True/False values always override the default.
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
# Default based on whether theme is set
default_enabled = theme_id is not None
# Resolve with explicit override taking precedence
resolved_cdg = enable_cdg if enable_cdg is not None else default_enabled
resolved_txt = enable_txt if enable_txt is not None else default_enabled
return resolved_cdg, resolved_txt
@dataclass
class EffectiveDistributionSettings:
"""Settings with defaults applied from environment variables."""
dropbox_path: Optional[str]
gdrive_folder_id: Optional[str]
discord_webhook_url: Optional[str]
brand_prefix: Optional[str]
def _get_effective_distribution_settings(
dropbox_path: Optional[str] = None,
gdrive_folder_id: Optional[str] = None,
discord_webhook_url: Optional[str] = None,
brand_prefix: Optional[str] = None,
) -> EffectiveDistributionSettings:
"""
Get distribution settings with defaults applied from environment variables.
This ensures consistent handling of defaults across all job creation endpoints.
Each parameter, if not provided (None), falls back to the corresponding
environment variable configured in settings.
Args:
dropbox_path: Explicit Dropbox path or None for default
gdrive_folder_id: Explicit Google Drive folder ID or None for default
discord_webhook_url: Explicit Discord webhook URL or None for default
brand_prefix: Explicit brand prefix or None for default
Returns:
EffectiveDistributionSettings with defaults applied
"""
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=dropbox_path or settings.default_dropbox_path,
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
brand_prefix=brand_prefix or settings.default_brand_prefix,
)
def _prepare_theme_for_job(
job_id: str,
theme_id: str,
@@ -342,7 +406,7 @@ async def upload_and_create_job(
enable_txt: Optional[bool] = Form(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise"),
# Finalisation options
brand_prefix: Optional[str] = Form(None, description="Brand code prefix (e.g., NOMAD)"),
enable_youtube_upload: Optional[bool] = Form(None, description="Upload to YouTube. None = use server default"),
enable_youtube_upload: bool = Form(False, description="Upload to YouTube"),
youtube_description: Optional[str] = Form(None, description="YouTube video description text"),
discord_webhook_url: Optional[str] = Form(None, description="Discord webhook URL for notifications"),
webhook_url: Optional[str] = Form(None, description="Generic webhook URL"),
@@ -382,14 +446,6 @@ async def upload_and_create_job(
The style_params JSON can reference the uploaded images/fonts by their original
filenames, and the backend will update the paths to GCS locations.
"""
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.file_upload:
raise HTTPException(
status_code=403,
detail="File upload is not available for this portal"
)
try:
# Validate main audio file type
file_ext = Path(file.filename).suffix.lower()
@@ -438,7 +494,7 @@ async def upload_and_create_job(
)
# Apply default distribution settings from environment if not provided
dist = get_effective_distribution_settings(
dist = _get_effective_distribution_settings(
dropbox_path=dropbox_path,
gdrive_folder_id=gdrive_folder_id,
discord_webhook_url=discord_webhook_url,
@@ -454,19 +510,12 @@ async def upload_and_create_job(
if dist.brand_prefix and not brand_prefix:
logger.info(f"Using default brand_prefix: {dist.brand_prefix}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload
if effective_enable_youtube_upload and enable_youtube_upload is None:
logger.info("Using default enable_youtube_upload: True (from env)")
# Validate credentials for requested distribution services (including defaults)
# This prevents accepting jobs that will fail later due to missing credentials
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -506,18 +555,9 @@ async def upload_and_create_job(
detail=f"Invalid color_overrides JSON: {e}"
)
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
effective_theme_id = theme_id
if effective_theme_id is None:
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme: {effective_theme_id}")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
effective_theme_id, enable_cdg, enable_txt
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
theme_id, enable_cdg, enable_txt
)
# Check if any custom style files are being uploaded (overrides theme)
@@ -549,12 +589,12 @@ async def upload_and_create_job(
artist=artist,
title=title,
filename=file.filename,
theme_id=effective_theme_id,
theme_id=theme_id,
color_overrides=parsed_color_overrides,
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
enable_youtube_upload=enable_youtube_upload,
youtube_description=youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=webhook_url,
@@ -576,12 +616,10 @@ async def upload_and_create_job(
request_metadata=request_metadata,
# Non-interactive mode
non_interactive=non_interactive,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="upload")
@@ -593,16 +631,16 @@ async def upload_and_create_job(
theme_style_params_path = None
theme_style_assets = {}
theme_youtube_desc = None
if effective_theme_id and not has_custom_style_files:
if theme_id and not has_custom_style_files:
try:
theme_style_params_path, theme_style_assets, theme_youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, parsed_color_overrides or None
job_id, theme_id, parsed_color_overrides or None
)
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
logger.info(f"Applied theme '{theme_id}' to job {job_id}")
except HTTPException:
raise # Re-raise validation errors (e.g., theme not found)
except Exception as e:
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
logger.warning(f"Failed to prepare theme '{theme_id}' for job {job_id}: {e}")
# Continue without theme - job can still be processed with defaults
# Upload main audio file to GCS
@@ -777,12 +815,11 @@ async def upload_and_create_job(
"using_default": gdrive_folder_id is None,
}
if effective_enable_youtube_upload:
if enable_youtube_upload:
youtube_result = credential_manager.check_youtube_credentials()
distribution_services["youtube"] = {
"enabled": True,
"credentials_valid": youtube_result.status == CredentialStatus.VALID,
"using_default": enable_youtube_upload is None,
}
if dist.discord_webhook_url:
@@ -871,7 +908,7 @@ class CreateFinaliseOnlyJobRequest(BaseModel):
# Finalisation options
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
keep_brand_code: Optional[str] = Field(None, description="Preserve existing brand code from folder name")
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
enable_youtube_upload: bool = Field(False, description="Upload to YouTube")
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
@@ -975,14 +1012,6 @@ async def create_job_with_upload_urls(
- Works with any HTTP client (no HTTP/2 required)
- Resumable uploads possible with GCS
"""
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.file_upload:
raise HTTPException(
status_code=403,
detail="File upload is not available for this portal"
)
try:
# Validate files list
if not body.files:
@@ -1012,23 +1041,18 @@ async def create_job_with_upload_urls(
)
# Apply default distribution settings from environment if not provided
dist = get_effective_distribution_settings(
dist = _get_effective_distribution_settings(
dropbox_path=body.dropbox_path,
gdrive_folder_id=body.gdrive_folder_id,
discord_webhook_url=body.discord_webhook_url,
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if body.enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1060,18 +1084,9 @@ async def create_job_with_upload_urls(
# Get original audio filename
audio_file = audio_files[0]
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
effective_theme_id = body.theme_id
if effective_theme_id is None:
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme: {effective_theme_id}")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
effective_theme_id, body.enable_cdg, body.enable_txt
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Check if style_params is being uploaded (overrides theme)
@@ -1085,12 +1100,12 @@ async def create_job_with_upload_urls(
artist=body.artist,
title=body.title,
filename=audio_file.filename,
theme_id=effective_theme_id,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
youtube_description_template=body.youtube_description, # video_worker reads this field
discord_webhook_url=dist.discord_webhook_url,
@@ -1107,10 +1122,8 @@ async def create_job_with_upload_urls(
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
@@ -1119,9 +1132,9 @@ async def create_job_with_upload_urls(
logger.info(f"Created job {job_id} for {body.artist} - {body.title} (signed URL upload flow)")
# If theme is set and no style_params uploaded, prepare theme style now
if effective_theme_id and not has_style_params_upload:
if body.theme_id and not has_style_params_upload:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, body.color_overrides
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
@@ -1131,7 +1144,7 @@ async def create_job_with_upload_urls(
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
# Generate signed upload URLs for each file
upload_urls = []
@@ -1437,14 +1450,6 @@ async def create_job_from_url(
Note: YouTube rate limiting may cause occasional download failures.
The backend will retry automatically.
"""
# Check tenant feature flag
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.youtube_url:
raise HTTPException(
status_code=403,
detail="URL-based job creation is not available for this portal"
)
try:
# Validate URL
if not _validate_url(body.url):
@@ -1454,23 +1459,18 @@ async def create_job_from_url(
)
# Apply default distribution settings from environment if not provided
dist = get_effective_distribution_settings(
dist = _get_effective_distribution_settings(
dropbox_path=body.dropbox_path,
gdrive_folder_id=body.gdrive_folder_id,
discord_webhook_url=body.discord_webhook_url,
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate credentials for requested distribution services
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if body.enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1503,18 +1503,9 @@ async def create_job_from_url(
artist = body.artist
title = body.title
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
effective_theme_id = body.theme_id
if effective_theme_id is None:
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme: {effective_theme_id}")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
effective_theme_id, body.enable_cdg, body.enable_txt
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Prefer authenticated user's email over request body
@@ -1526,12 +1517,12 @@ async def create_job_from_url(
artist=artist,
title=title,
filename=None, # No file uploaded
theme_id=effective_theme_id,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
webhook_url=body.webhook_url,
@@ -1547,19 +1538,17 @@ async def create_job_from_url(
other_stems_models=body.other_stems_models,
request_metadata=request_metadata,
non_interactive=body.non_interactive,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
metrics.record_job_created(job_id, source="url")
# If theme is set, prepare theme style now
if effective_theme_id:
if body.theme_id:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, body.color_overrides
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
@@ -1569,8 +1558,8 @@ async def create_job_from_url(
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
logger.info(f"Created URL-based job {job_id} for URL: {body.url}")
if artist:
logger.info(f" Artist: {artist}")
@@ -1669,14 +1658,6 @@ async def create_finalise_only_job(
The endpoint returns signed URLs for uploading all the prep files.
"""
# Check tenant feature flag - finalise-only requires file upload capability
tenant_config = get_tenant_config_from_request(request)
if tenant_config and not tenant_config.features.file_upload:
raise HTTPException(
status_code=403,
detail="File upload is not available for this portal"
)
try:
# Validate files list
if not body.files:
@@ -1724,23 +1705,18 @@ async def create_finalise_only_job(
)
# Apply default distribution settings
dist = get_effective_distribution_settings(
dist = _get_effective_distribution_settings(
dropbox_path=body.dropbox_path,
gdrive_folder_id=body.gdrive_folder_id,
discord_webhook_url=body.discord_webhook_url,
brand_prefix=body.brand_prefix,
)
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Validate distribution credentials if services are requested
invalid_services = []
credential_manager = get_credential_manager()
if effective_enable_youtube_upload:
if body.enable_youtube_upload:
result = credential_manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append(f"youtube ({result.message})")
@@ -1769,18 +1745,9 @@ async def create_finalise_only_job(
# Extract request metadata
request_metadata = extract_request_metadata(request, created_from="finalise_only_upload")
# Apply default theme if none specified
# This ensures all karaoke videos use the Nomad theme by default
effective_theme_id = body.theme_id
if effective_theme_id is None:
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme: {effective_theme_id}")
# Resolve CDG/TXT defaults based on theme
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
effective_theme_id, body.enable_cdg, body.enable_txt
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
body.theme_id, body.enable_cdg, body.enable_txt
)
# Check if style_params is being uploaded (overrides theme)
@@ -1794,12 +1761,12 @@ async def create_finalise_only_job(
artist=body.artist,
title=body.title,
filename="finalise_only", # No single audio file - using prep outputs
theme_id=effective_theme_id,
theme_id=body.theme_id,
color_overrides=body.color_overrides or {},
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
brand_prefix=dist.brand_prefix,
enable_youtube_upload=effective_enable_youtube_upload,
enable_youtube_upload=body.enable_youtube_upload,
youtube_description=body.youtube_description,
discord_webhook_url=dist.discord_webhook_url,
dropbox_path=dist.dropbox_path,
@@ -1808,10 +1775,8 @@ async def create_finalise_only_job(
finalise_only=True,
keep_brand_code=body.keep_brand_code,
request_metadata=request_metadata,
# Tenant scoping
tenant_id=tenant_config.id if tenant_config else None,
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
job_id = job.job_id
# Record job creation metric
@@ -1820,9 +1785,9 @@ async def create_finalise_only_job(
logger.info(f"Created finalise-only job {job_id} for {body.artist} - {body.title}")
# If theme is set and no style_params uploaded, prepare theme style now
if effective_theme_id and not has_style_params_upload:
if body.theme_id and not has_style_params_upload:
style_params_path, style_assets, youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, body.color_overrides
job_id, body.theme_id, body.color_overrides
)
# Update job with theme style data
update_data = {
@@ -1832,7 +1797,7 @@ async def create_finalise_only_job(
if youtube_desc and not body.youtube_description:
update_data['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, update_data)
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
logger.info(f"Applied theme '{body.theme_id}' to job {job_id}")
# Generate signed upload URLs for each file
upload_urls = []
-99
View File
@@ -13,9 +13,6 @@ from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -230,7 +227,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
"active_jobs": health.get("active_jobs", 0),
"queue_length": health.get("queue_length", 0),
"ffmpeg_version": health.get("ffmpeg_version"),
"wheel_version": health.get("wheel_version"),
}
except Exception as e:
return {
@@ -241,39 +237,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
}
@router.get("/health/encoding-worker")
async def encoding_worker_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check encoding worker status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_encoding_worker_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("wheel_version"),
"active_jobs": status.get("active_jobs", 0),
"queue_length": status.get("queue_length", 0),
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -337,68 +300,6 @@ async def detailed_health_check() -> Dict[str, Any]:
}
@router.get("/health/preload-status")
async def preload_status() -> Dict[str, Any]:
"""
Check status of preloaded resources for performance optimization.
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
were successfully preloaded at container startup. If any show as
not preloaded, check Cloud Run startup logs for errors.
Expected state after successful deployment:
- spacy.preloaded: true
- nltk.preloaded: true
- langfuse.preloaded: true (if configured) or configured: false
"""
# SpaCy status
spacy_model = get_preloaded_model("en_core_web_sm")
spacy_status = {
"preloaded": is_model_preloaded("en_core_web_sm"),
"model": "en_core_web_sm",
}
if spacy_model:
spacy_status["vocab_size"] = len(spacy_model.vocab)
# NLTK status
cmudict = get_preloaded_cmudict()
nltk_status = {
"preloaded": is_cmudict_preloaded(),
"resource": "cmudict",
}
if cmudict:
nltk_status["entries"] = len(cmudict)
# Langfuse status
langfuse_handler = get_preloaded_langfuse_handler()
langfuse_status = {
"configured": is_langfuse_configured(),
"preloaded": is_langfuse_preloaded(),
}
if langfuse_handler:
langfuse_status["handler_type"] = type(langfuse_handler).__name__
# Overall status
all_preloaded = (
spacy_status["preloaded"]
and nltk_status["preloaded"]
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
)
return {
"status": "ok" if all_preloaded else "degraded",
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
"spacy": spacy_status,
"nltk": nltk_status,
"langfuse": langfuse_status,
"performance_impact": {
"spacy_preload": "Saves ~60s on first lyrics correction",
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
}
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+1 -112
View File
@@ -316,124 +316,13 @@ async def trigger_render_video_worker(
)
@router.post("/jobs/{job_id}/check-idle-reminder")
async def check_idle_reminder(
job_id: str,
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Check if a job needs an idle reminder email.
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
a job enters a blocking state (AWAITING_REVIEW or AWAITING_INSTRUMENTAL_SELECTION).
If the job is still in the blocking state and no reminder has been sent yet,
sends a reminder email to the user.
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
"""
from backend.models.job import JobStatus
from backend.services.job_notification_service import get_job_notification_service
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
add_span_attribute("job_id", job_id)
add_span_attribute("operation", "idle_reminder_check")
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
add_span_event("job_not_found")
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
# Check if job is still in a blocking state
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
if job.status not in [s.value for s in blocking_states]:
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
add_span_event("not_blocking", {"current_status": job.status})
return {
"status": "skipped",
"job_id": job_id,
"message": f"Job not in blocking state (current: {job.status})"
}
# Normalize state_data to prevent None errors
state_data = job.state_data or {}
# Check if reminder was already sent (idempotency)
if state_data.get('reminder_sent'):
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
add_span_event("already_sent")
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
if getattr(job, 'made_for_you', False):
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
add_span_event("made_for_you_skip")
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
# Check if user has an email
if not job.user_email:
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
add_span_event("no_email")
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
# Determine action type
action_type = state_data.get('blocking_action_type')
if not action_type:
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
# Send the reminder email
try:
notification_service = get_job_notification_service()
success = await notification_service.send_action_reminder_email(
job_id=job.job_id,
user_email=job.user_email,
action_type=action_type,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
audio_hash=job.audio_hash,
review_token=job.review_token,
instrumental_token=job.instrumental_token,
)
if success:
# Mark reminder as sent (prevents duplicate sends)
job_manager.firestore.update_job(job_id, {
'state_data': {**state_data, 'reminder_sent': True}
})
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
add_span_event("reminder_sent", {"action_type": action_type})
return {
"status": "sent",
"job_id": job_id,
"message": f"Sent {action_type} reminder to {job.user_email}"
}
else:
logger.error(f"[job:{job_id}] Failed to send reminder email")
add_span_event("send_failed")
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
except Exception as e:
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Internal health check endpoint.
Used to verify the internal API is responsive.
Requires admin authentication.
"""
+19 -41
View File
@@ -11,7 +11,7 @@ import asyncio
import logging
import httpx
from typing import List, Optional, Dict, Any, Tuple
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends
from backend.models.job import Job, JobCreate, JobResponse, JobStatus
from backend.models.requests import (
@@ -25,13 +25,10 @@ from backend.models.requests import (
from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
from backend.services.storage_service import StorageService
from backend.services.theme_service import get_theme_service
from backend.config import get_settings
from backend.api.dependencies import require_admin, require_auth, require_instrumental_auth
from backend.services.auth_service import UserType, AuthResult
from backend.services.metrics import metrics
from backend.middleware.tenant import get_tenant_from_request
from backend.utils.test_data import is_test_email
logger = logging.getLogger(__name__)
@@ -90,36 +87,20 @@ async def create_job(
user_email = request.user_email
logger.info(f"Admin {auth_result.user_email} creating job on behalf of {user_email}")
# Apply YouTube upload default from settings
# Use explicit value if provided, otherwise fall back to server default
settings = get_settings()
effective_enable_youtube_upload = request.enable_youtube_upload if request.enable_youtube_upload is not None else settings.default_enable_youtube_upload
# Apply default theme - all jobs require a theme
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if not effective_theme_id:
raise HTTPException(
status_code=422,
detail="No default theme configured. Please contact support or specify a theme_id."
)
logger.info(f"Applying default theme: {effective_theme_id}")
# Create job with all preferences
job_create = JobCreate(
url=str(request.url),
artist=request.artist,
title=request.title,
theme_id=effective_theme_id, # Required - all jobs must have a theme
enable_cdg=request.enable_cdg,
enable_txt=request.enable_txt,
enable_youtube_upload=effective_enable_youtube_upload,
enable_youtube_upload=request.enable_youtube_upload,
youtube_description=request.youtube_description,
webhook_url=request.webhook_url,
user_email=user_email
)
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
job = job_manager.create_job(job_create)
# Record job creation metric
metrics.record_job_created(job.job_id, source="url")
@@ -191,13 +172,11 @@ def _check_job_ownership(job: Job, auth_result: AuthResult) -> bool:
@router.get("", response_model=List[Job])
async def list_jobs(
request: Request,
status: Optional[JobStatus] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[str] = None,
created_before: Optional[str] = None,
exclude_test: bool = True,
limit: int = 100,
auth_result: AuthResult = Depends(require_auth)
) -> List[Job]:
@@ -205,7 +184,6 @@ async def list_jobs(
List jobs with optional filters.
Regular users only see their own jobs. Admins see all jobs.
Users on tenant portals only see jobs from their tenant.
Args:
status: Filter by job status (pending, complete, failed, etc.)
@@ -213,7 +191,6 @@ async def list_jobs(
client_id: Filter by request_metadata.client_id (customer identifier)
created_after: Filter jobs created after this ISO datetime (e.g., 2024-01-01T00:00:00Z)
created_before: Filter jobs created before this ISO datetime
exclude_test: If True (default), exclude jobs from test users (admin only)
limit: Maximum number of jobs to return (default 100)
Returns:
@@ -250,10 +227,6 @@ async def list_jobs(
logger.warning("Non-admin auth without user_email, returning empty job list")
return []
# Get tenant_id from request for portal scoping
# Tenant users only see jobs from their tenant
tenant_id = get_tenant_from_request(request)
jobs = job_manager.list_jobs(
status=status,
environment=environment,
@@ -261,14 +234,9 @@ async def list_jobs(
created_after=created_after_dt,
created_before=created_before_dt,
user_email=user_email_filter,
tenant_id=tenant_id,
limit=limit
)
# Filter out test user jobs if exclude_test is True (admin only)
if exclude_test and auth_result.is_admin:
jobs = [j for j in jobs if not is_test_email(j.user_email or "")]
logger.debug(f"Listed {len(jobs)} jobs for user={auth_result.user_email}, admin={auth_result.is_admin}")
return jobs
except HTTPException:
@@ -1191,11 +1159,21 @@ async def download_file(
content_type = content_types.get(ext, 'application/octet-stream')
# Build proper filename: "Artist - Title (Final Karaoke Lossy 4k).mp4"
# Use sanitize_filename to handle Unicode characters (curly quotes, em dashes, etc.)
# that cause HTTP header encoding issues (Content-Disposition uses latin-1)
from karaoke_gen.utils import sanitize_filename
artist_clean = sanitize_filename(job.artist) if job.artist else None
title_clean = sanitize_filename(job.title) if job.title else None
# Sanitize artist/title to remove filesystem-unsafe characters
def sanitize_filename_part(s: str) -> str:
if not s:
return ""
# Remove/replace characters unsafe for filesystems: / \ : * ? " < > |
unsafe_chars = r'[/\\:*?"<>|]'
import re
sanitized = re.sub(unsafe_chars, '', s)
# Collapse multiple spaces and trim
sanitized = ' '.join(sanitized.split())
# Limit length to prevent overly long filenames
return sanitized[:100].strip()
artist_clean = sanitize_filename_part(job.artist) if job.artist else None
title_clean = sanitize_filename_part(job.title) if job.title else None
base_name = f"{artist_clean} - {title_clean}" if artist_clean and title_clean else None
if base_name and file_key in DOWNLOAD_FILENAME_SUFFIXES:
-238
View File
@@ -1,238 +0,0 @@
"""
Push Notification API routes.
Provides endpoints for managing Web Push notification subscriptions:
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
- POST /api/push/subscribe: Register a push subscription
- POST /api/push/unsubscribe: Remove a push subscription
- GET /api/push/subscriptions: List user's subscriptions
- POST /api/push/test: Send a test notification (admin only)
"""
import logging
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/push", tags=["push"])
# Request/Response Models
class VapidPublicKeyResponse(BaseModel):
"""Response containing VAPID public key."""
enabled: bool
vapid_public_key: Optional[str] = None
class SubscribeRequest(BaseModel):
"""Request to subscribe to push notifications."""
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
class SubscribeResponse(BaseModel):
"""Response after subscribing."""
status: str
message: str
class UnsubscribeRequest(BaseModel):
"""Request to unsubscribe from push notifications."""
endpoint: str
class UnsubscribeResponse(BaseModel):
"""Response after unsubscribing."""
status: str
message: str
class SubscriptionInfo(BaseModel):
"""Information about a push subscription."""
endpoint: str
device_name: Optional[str] = None
created_at: Optional[str] = None
last_used_at: Optional[str] = None
class SubscriptionsListResponse(BaseModel):
"""Response containing user's subscriptions."""
subscriptions: List[SubscriptionInfo]
count: int
class TestNotificationRequest(BaseModel):
"""Request to send a test notification."""
title: Optional[str] = "Test Notification"
body: Optional[str] = "This is a test push notification from Karaoke Generator"
class TestNotificationResponse(BaseModel):
"""Response after sending test notification."""
status: str
sent_count: int
message: str
# Routes
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
async def get_vapid_public_key():
"""
Get the VAPID public key for push subscription.
This endpoint is public - no authentication required.
Returns the public key needed for client-side PushManager.subscribe().
"""
settings = get_settings()
push_service = get_push_notification_service()
if not settings.enable_push_notifications:
return VapidPublicKeyResponse(enabled=False)
public_key = push_service.get_public_key()
if not public_key:
return VapidPublicKeyResponse(enabled=False)
return VapidPublicKeyResponse(
enabled=True,
vapid_public_key=public_key
)
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
request: SubscribeRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
Register a push notification subscription for the current user.
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
"""
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in request.keys or "auth" not in request.keys:
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=request.endpoint,
keys=request.keys,
device_name=request.device_name
)
if not success:
raise HTTPException(status_code=500, detail="Failed to save subscription")
return SubscribeResponse(
status="success",
message="Push subscription registered successfully"
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
auth_result: AuthResult = Depends(require_auth)
):
"""
Remove a push notification subscription.
Requires authentication. Users can only remove their own subscriptions.
"""
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
success = await push_service.remove_subscription(
user_email=auth_result.user_email,
endpoint=request.endpoint
)
if not success:
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message="Subscription removed (or was not found)"
)
return UnsubscribeResponse(
status="success",
message="Push subscription removed successfully"
)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
auth_result: AuthResult = Depends(require_auth)
):
"""
List all push notification subscriptions for the current user.
Requires authentication.
"""
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
return SubscriptionsListResponse(
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
count=len(subscriptions)
)
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
auth_result: AuthResult = Depends(require_admin)
):
"""
Send a test push notification to the current user's devices.
Admin only. Useful for testing push notification setup.
"""
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
if not auth_result.user_email:
raise HTTPException(status_code=401, detail="User email not available")
push_service = get_push_notification_service()
sent_count = await push_service.send_push(
user_email=auth_result.user_email,
title=request.title or "Test Notification",
body=request.body or "This is a test push notification",
url="/app/",
tag="test"
)
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=f"Test notification sent to {sent_count} device(s)"
)
-428
View File
@@ -1,428 +0,0 @@
"""
Admin API routes for rate limit management.
Handles:
- Rate limit statistics and monitoring
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- User override management (whitelist/bypass permissions)
"""
import logging
from datetime import datetime, timezone
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel, EmailStr
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.config import settings
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# =============================================================================
# Response Models
# =============================================================================
class RateLimitStatsResponse(BaseModel):
"""Current rate limit statistics."""
# Configuration
jobs_per_day_limit: int
youtube_uploads_per_day_limit: int
beta_ip_per_day_limit: int
rate_limiting_enabled: bool
# Current usage
youtube_uploads_today: int
youtube_uploads_remaining: int
# Blocklist stats
disposable_domains_count: int
blocked_emails_count: int
blocked_ips_count: int
# User override stats
total_overrides: int
class UserRateLimitStatusResponse(BaseModel):
"""Rate limit status for a specific user."""
email: str
jobs_today: int
jobs_limit: int
jobs_remaining: int
has_bypass: bool
custom_limit: Optional[int]
bypass_reason: Optional[str]
class BlocklistsResponse(BaseModel):
"""All blocklist data."""
disposable_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
updated_at: Optional[datetime]
updated_by: Optional[str]
class DomainRequest(BaseModel):
"""Request to add/remove a domain."""
domain: str
class EmailRequest(BaseModel):
"""Request to add/remove an email."""
email: str
class IPRequest(BaseModel):
"""Request to add/remove an IP."""
ip_address: str
class UserOverride(BaseModel):
"""User override configuration."""
email: str
bypass_job_limit: bool
custom_daily_job_limit: Optional[int]
reason: str
created_by: str
created_at: datetime
class UserOverrideRequest(BaseModel):
"""Request to set/update a user override."""
bypass_job_limit: bool = False
custom_daily_job_limit: Optional[int] = None
reason: str
class UserOverridesListResponse(BaseModel):
"""List of all user overrides."""
overrides: List[UserOverride]
total: int
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Statistics Endpoints
# =============================================================================
@router.get("/stats", response_model=RateLimitStatsResponse)
async def get_rate_limit_stats(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get current rate limit statistics.
Returns configuration values, current usage counts, and blocklist stats.
"""
rate_limit_service = get_rate_limit_service()
email_validation = get_email_validation_service()
# Get YouTube uploads today
youtube_today = rate_limit_service.get_youtube_uploads_today()
youtube_limit = settings.rate_limit_youtube_uploads_per_day
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
# Get blocklist stats
blocklist_stats = email_validation.get_blocklist_stats()
# Get override count
overrides = rate_limit_service.get_all_overrides()
return RateLimitStatsResponse(
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
rate_limiting_enabled=settings.enable_rate_limiting,
youtube_uploads_today=youtube_today,
youtube_uploads_remaining=youtube_remaining,
disposable_domains_count=blocklist_stats["disposable_domains_count"],
blocked_emails_count=blocklist_stats["blocked_emails_count"],
blocked_ips_count=blocklist_stats["blocked_ips_count"],
total_overrides=len(overrides),
)
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
async def get_user_rate_limit_status(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Get rate limit status for a specific user.
Returns their current usage and any override settings.
"""
rate_limit_service = get_rate_limit_service()
jobs_today = rate_limit_service.get_user_job_count_today(email)
override = rate_limit_service.get_user_override(email)
if override:
has_bypass = override.get("bypass_job_limit", False)
custom_limit = override.get("custom_daily_job_limit")
bypass_reason = override.get("reason")
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
else:
has_bypass = False
custom_limit = None
bypass_reason = None
jobs_limit = settings.rate_limit_jobs_per_day
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
return UserRateLimitStatusResponse(
email=email,
jobs_today=jobs_today,
jobs_limit=jobs_limit,
jobs_remaining=jobs_remaining,
has_bypass=has_bypass,
custom_limit=custom_limit,
bypass_reason=bypass_reason,
)
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@router.get("/blocklists", response_model=BlocklistsResponse)
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get all blocklist data.
Returns disposable domains, blocked emails, and blocked IPs.
"""
email_validation = get_email_validation_service()
# Force refresh to get latest data
config = email_validation.get_blocklist_config(force_refresh=True)
# Get the raw document for metadata
from google.cloud import firestore
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
db = get_firestore_client()
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
updated_at = None
updated_by = None
if doc.exists:
data = doc.to_dict()
updated_at = data.get("updated_at")
updated_by = data.get("updated_by")
return BlocklistsResponse(
disposable_domains=sorted(list(config["disposable_domains"])),
blocked_emails=sorted(list(config["blocked_emails"])),
blocked_ips=sorted(list(config["blocked_ips"])),
updated_at=updated_at,
updated_by=updated_by,
)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
async def add_disposable_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail="Invalid domain format")
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"Domain '{domain}' added to disposable domains blocklist"
)
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
async def remove_disposable_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
return SuccessResponse(
success=True,
message=f"Domain '{domain}' removed from disposable domains blocklist"
)
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
async def add_blocked_email(
request: EmailRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an email to the blocked emails list."""
email_validation = get_email_validation_service()
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"Email '{email}' added to blocked emails list"
)
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
async def remove_blocked_email(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an email from the blocked emails list."""
email_validation = get_email_validation_service()
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Email not found in blocklist")
return SuccessResponse(
success=True,
message=f"Email '{email}' removed from blocked emails list"
)
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
async def add_blocked_ip(
request: IPRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an IP address to the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail="Invalid IP address")
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=f"IP '{ip_address}' added to blocked IPs list"
)
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
async def remove_blocked_ip(
ip_address: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an IP address from the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail="IP not found in blocklist")
return SuccessResponse(
success=True,
message=f"IP '{ip_address}' removed from blocked IPs list"
)
# =============================================================================
# User Override Management Endpoints
# =============================================================================
@router.get("/overrides", response_model=UserOverridesListResponse)
async def get_all_overrides(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all user overrides."""
rate_limit_service = get_rate_limit_service()
overrides_data = rate_limit_service.get_all_overrides()
overrides = [
UserOverride(
email=email,
bypass_job_limit=data.get("bypass_job_limit", False),
custom_daily_job_limit=data.get("custom_daily_job_limit"),
reason=data.get("reason", ""),
created_by=data.get("created_by", "unknown"),
created_at=data.get("created_at", datetime.now(timezone.utc)),
)
for email, data in overrides_data.items()
]
return UserOverridesListResponse(
overrides=overrides,
total=len(overrides),
)
@router.put("/overrides/{email}", response_model=SuccessResponse)
async def set_user_override(
email: str,
request: UserOverrideRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Set or update a user override."""
rate_limit_service = get_rate_limit_service()
email = email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail="Invalid email format")
if not request.reason or len(request.reason.strip()) < 3:
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
rate_limit_service.set_user_override(
user_email=email,
bypass_job_limit=request.bypass_job_limit,
custom_daily_job_limit=request.custom_daily_job_limit,
reason=request.reason,
admin_email=auth_result.user_email,
)
return SuccessResponse(
success=True,
message=f"Override set for user '{email}'"
)
@router.delete("/overrides/{email}", response_model=SuccessResponse)
async def remove_user_override(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a user override."""
rate_limit_service = get_rate_limit_service()
email = email.lower().strip()
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
raise HTTPException(status_code=404, detail="Override not found for user")
return SuccessResponse(
success=True,
message=f"Override removed for user '{email}'"
)
+43 -132
View File
@@ -27,15 +27,13 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.services.job_logging import job_log_context, JobLogger
from backend.services.tracing import create_span, add_span_attribute, add_span_event
from backend.services.encoding_service import get_encoding_service
from backend.api.dependencies import require_auth, require_review_auth
from backend.services.auth_service import UserType
from backend.config import get_settings
# LyricsTranscriber imports for preview generation
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from lyrics_transcriber.types import CorrectionResult
from lyrics_transcriber.core.config import OutputConfig
from lyrics_transcriber.correction.operations import CorrectionOperations
# Import from the unified style loader
from karaoke_gen.style_loader import load_styles_from_gcs
@@ -387,38 +385,28 @@ async def generate_preview_video(
):
"""
Generate a preview video with the current corrections.
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
video with the user's current corrections applied.
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
the high-performance GCE worker for faster generation (15-20s vs 60+s).
"""
job_manager = JobManager()
storage = StorageService()
settings = get_settings()
encoding_service = get_encoding_service()
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail="Job not found")
# Job must be in review state to generate preview
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
raise HTTPException(
status_code=400,
detail=f"Job not in review state (current status: {job.status})"
)
# Check if GCE preview encoding is enabled
use_gce_preview = encoding_service.is_preview_enabled
# Check if user wants theme background image (slower) or black background (faster, default)
use_background_image = updated_data.get("use_background_image", False)
logger.info(f"Job {job_id}: Generating preview video (GCE: {use_gce_preview}, background image: {use_background_image})")
logger.info(f"Job {job_id}: Generating preview video")
# Use tracing and job_log_context for full observability
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
with create_span("generate-preview-video", {"job_id": job_id}) as span:
with job_log_context(job_id, worker="preview"):
try:
# Create temp directory for this preview operation
@@ -428,19 +416,19 @@ async def generate_preview_video(
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
corrections_path = os.path.join(temp_dir, "corrections.json")
storage.download_file(corrections_gcs, corrections_path)
with open(corrections_path, 'r', encoding='utf-8') as f:
original_data = json.load(f)
# 2. Download input audio
audio_path = os.path.join(temp_dir, "audio.flac")
storage.download_file(job.input_media_gcs_path, audio_path)
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
# 3. Load original as CorrectionResult
correction_result = CorrectionResult.from_dict(original_data)
add_span_event("corrections_loaded")
# 4. Get or create styles file for preview using unified style loader
with create_span("load-styles") as styles_span:
styles_path, _ = load_styles_from_gcs(
@@ -451,131 +439,54 @@ async def generate_preview_video(
logger=logger,
)
styles_span.set_attribute("styles_path", styles_path)
# 5. Set up output config for preview
output_dir = os.path.join(temp_dir, "output")
cache_dir = os.path.join(temp_dir, "cache")
os.makedirs(output_dir, exist_ok=True)
os.makedirs(cache_dir, exist_ok=True)
output_config = OutputConfig(
output_styles_json=styles_path,
output_dir=output_dir,
cache_dir=cache_dir,
video_resolution="360p",
)
# 6. Generate preview (ASS-only if using GCE, or full video if local)
preview_gcs_path = None
if use_gce_preview:
# GCE path: Generate ASS only, then offload encoding to GCE
try:
with create_span("generate-ass-subtitles") as ass_span:
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=True, # Only generate ASS, skip video encoding
)
preview_hash = result["preview_hash"]
ass_path = result["ass_path"]
ass_span.set_attribute("ass_path", ass_path)
add_span_event("ass_generated")
# Upload ASS to GCS
with create_span("upload-ass-to-gcs") as upload_ass_span:
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
storage.upload_file(ass_path, ass_gcs_path)
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
# Call GCE encoding service
with create_span("gce-preview-encoding") as gce_span:
bucket_name = settings.gcs_bucket_name
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
# Get background image and font from style assets if available
style_assets = job.style_assets or {}
# Only use background image if user explicitly requested it
# Default is black background for faster preview generation (~10s vs ~30-60s)
background_image_gcs_path = None
if use_background_image:
for key in ["karaoke_background", "style_karaoke_background"]:
if key in style_assets:
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("background_image", background_image_gcs_path)
break
gce_span.set_attribute("use_background_image", use_background_image)
font_gcs_path = None
for key in ["font", "style_font"]:
if key in style_assets:
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
gce_span.set_attribute("font", font_gcs_path)
break
gce_result = await encoding_service.encode_preview_video(
job_id=f"preview_{job_id}_{preview_hash}",
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
background_color="black",
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
gce_span.set_attribute("gce_status", gce_result.get("status"))
add_span_event("gce_encoding_complete")
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
except Exception as gce_error:
# Fall back to local encoding if GCE fails
logger.warning(
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
)
span.set_attribute("gce_fallback", True)
use_gce_preview = False # Fall through to local encoding below
if not use_gce_preview:
# Local path: Generate full preview video locally
with create_span("render-preview-video-local") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger,
ass_only=False, # Generate full video locally
)
preview_hash = result["preview_hash"]
video_path = result["video_path"]
add_span_event("render_complete")
# Upload preview video to GCS
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# 6. Generate preview video using CorrectionOperations (heavy operation)
with create_span("render-preview-video") as render_span:
render_span.set_attribute("resolution", "360p")
result = CorrectionOperations.generate_preview_video(
correction_result=correction_result,
updated_data=updated_data,
output_config=output_config,
audio_filepath=audio_path,
artist=job.artist,
title=job.title,
logger=logger
)
add_span_event("render_complete")
# 7. Upload preview video to GCS for serving
preview_hash = result["preview_hash"]
video_path = result["video_path"]
with create_span("upload-preview-video") as upload_span:
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
storage.upload_file(video_path, preview_gcs_path)
upload_span.set_attribute("gcs_path", preview_gcs_path)
# Store the GCS path for serving
if job_id not in _preview_videos:
_preview_videos[job_id] = {}
_preview_videos[job_id][preview_hash] = preview_gcs_path
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
span.set_attribute("preview_hash", preview_hash)
span.set_attribute("success", True)
return {"status": "success", "preview_hash": preview_hash}
except Exception as e:
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
span.set_attribute("error", str(e))
-120
View File
@@ -1,120 +0,0 @@
"""
Tenant API routes for white-label portal configuration.
These endpoints are public (no auth required) since the frontend needs
to fetch tenant branding before the user logs in.
"""
import logging
from typing import Optional
from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
@router.get("/config", response_model=TenantConfigResponse)
async def get_tenant_config(
request: Request,
tenant: Optional[str] = Query(
None,
description="Tenant ID override (for development). If not provided, detected from subdomain.",
),
):
"""
Get tenant configuration for the current portal.
This endpoint detects the tenant from:
1. Query parameter `tenant` (for development/testing)
2. X-Tenant-ID header (set by frontend)
3. Host header subdomain
Returns the public tenant configuration (branding, features, defaults)
or indicates this is the default Nomad Karaoke portal.
"""
tenant_service = get_tenant_service()
# Priority 1: Query parameter (for dev/testing)
tenant_id = tenant
# Priority 2: X-Tenant-ID header
if not tenant_id:
tenant_id = request.headers.get("X-Tenant-ID")
# Priority 3: Detect from Host header
if not tenant_id:
host = request.headers.get("Host", "")
# Check if this is a tenant subdomain
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
if host and "nomadkaraoke.com" in host.lower():
parts = host.lower().split(".")
# Skip known non-tenant subdomains
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
potential_tenant = parts[0]
if tenant_service.tenant_exists(potential_tenant):
tenant_id = potential_tenant
# If no tenant detected, return default config
if not tenant_id:
logger.debug("No tenant detected, returning default config")
return TenantConfigResponse(tenant=None, is_default=True)
# Load tenant config
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
logger.warning(f"Tenant not found: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
logger.info(f"Returning config for tenant: {tenant_id}")
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
async def get_tenant_config_by_id(tenant_id: str):
"""
Get tenant configuration by explicit tenant ID.
This is useful for admin tools or debugging.
"""
tenant_service = get_tenant_service()
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
return TenantConfigResponse(tenant=None, is_default=True)
# Check if tenant is active
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/asset/{tenant_id}/{asset_name}")
async def get_tenant_asset(tenant_id: str, asset_name: str):
"""
Get a signed URL for a tenant asset (logo, favicon, etc.).
This redirects to the signed GCS URL for the asset.
"""
from fastapi.responses import RedirectResponse
tenant_service = get_tenant_service()
url = tenant_service.get_asset_url(tenant_id, asset_name)
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail="Asset not found")
return RedirectResponse(url=url)
+51 -543
View File
@@ -8,28 +8,8 @@ Handles:
- Stripe checkout and webhooks
- Admin user management
"""
import hashlib
import logging
from typing import Optional, Tuple
def _mask_email(email: str) -> str:
"""Mask email for logging to avoid PII exposure.
Example: test@example.com -> te***@ex***.com
"""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
masked_local = local[:2] + "***" if len(local) > 2 else "***"
domain_parts = domain.split(".")
if len(domain_parts) >= 2:
masked_domain = domain_parts[0][:2] + "***." + domain_parts[-1]
else:
masked_domain = "***"
return f"{masked_local}@{masked_domain}"
from fastapi import APIRouter, HTTPException, Depends, Request, Header
from pydantic import BaseModel, EmailStr
@@ -52,15 +32,8 @@ from backend.models.user import (
from backend.services.user_service import get_user_service, UserService, USERS_COLLECTION
from backend.services.email_service import get_email_service, EmailService
from backend.services.stripe_service import get_stripe_service, StripeService, CREDIT_PACKAGES
from backend.services.theme_service import get_theme_service
from backend.services.job_defaults_service import (
get_effective_distribution_settings,
resolve_cdg_txt_defaults,
)
from backend.api.dependencies import require_admin
from backend.api.routes.file_upload import _prepare_theme_for_job
from backend.services.auth_service import UserType
from backend.utils.test_data import is_test_email
logger = logging.getLogger(__name__)
@@ -84,16 +57,6 @@ class CreateCheckoutResponse(BaseModel):
message: str
class MadeForYouCheckoutRequest(BaseModel):
"""Request to create a made-for-you karaoke video order."""
email: EmailStr
artist: str
title: str
source_type: str = "search" # search, youtube, or upload
youtube_url: Optional[str] = None
notes: Optional[str] = None
class CreditPackage(BaseModel):
"""Credit package information."""
id: str
@@ -136,11 +99,7 @@ async def send_magic_link(
The user will receive an email with a link that logs them in.
Links expire after 15 minutes and can only be used once.
For white-label tenants, validates email domain against tenant config.
"""
from backend.middleware.tenant import get_tenant_from_request, get_tenant_config_from_request
# Check if email service is configured
if not email_service.is_configured():
logger.error("Email service not configured - cannot send magic links")
@@ -151,39 +110,19 @@ async def send_magic_link(
email = request.email.lower()
# Get tenant context from middleware
tenant_id = get_tenant_from_request(http_request)
tenant_config = get_tenant_config_from_request(http_request)
# Validate email domain for tenant if configured
if tenant_config and tenant_config.auth.allowed_email_domains:
if not tenant_config.is_email_allowed(email):
logger.warning(f"Email domain not allowed for tenant {tenant_id}: {_mask_email(email)}")
# Return success anyway to prevent email enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
# Get client info for security logging
ip_address = http_request.client.host if http_request.client else None
user_agent = http_request.headers.get("user-agent")
# Create magic link token with tenant context
# Create magic link token
magic_link = user_service.create_magic_link(
email,
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id
user_agent=user_agent
)
# Get tenant-specific sender email
sender_email = None
if tenant_config:
sender_email = tenant_config.get_sender_email()
# Send email (with tenant-specific sender if configured)
sent = email_service.send_magic_link(email, magic_link.token, sender_email=sender_email)
# Send email
sent = email_service.send_magic_link(email, magic_link.token)
if not sent:
logger.error(f"Failed to send magic link email to {email}")
@@ -207,26 +146,18 @@ async def verify_magic_link(
Verify a magic link token and create a session.
Returns a session token that should be stored and used for subsequent requests.
The session will be associated with the tenant from the magic link.
"""
# Reject empty tokens early to avoid invalid Firestore document paths
if not token or not token.strip():
raise HTTPException(status_code=401, detail="Invalid token")
# Get client info
ip_address = http_request.client.host if http_request.client else None
user_agent = http_request.headers.get("user-agent")
# Check if this is a first login BEFORE verification (which sets last_login_at)
# We need to get the user's state before verify_magic_link updates it
# Also extract tenant_id from the magic link for session creation
from backend.services.user_service import MAGIC_LINKS_COLLECTION
magic_link_doc = user_service.db.collection(MAGIC_LINKS_COLLECTION).document(token).get()
is_first_login = False
magic_link_tenant_id = None
if magic_link_doc.exists:
magic_link_data = magic_link_doc.to_dict()
magic_link_tenant_id = magic_link_data.get('tenant_id')
pre_verify_user = user_service.get_user(magic_link_data.get('email', ''))
if pre_verify_user:
is_first_login = pre_verify_user.total_jobs_created == 0 and not pre_verify_user.last_login_at
@@ -237,19 +168,18 @@ async def verify_magic_link(
if not success or not user:
raise HTTPException(status_code=401, detail=message)
# Create session with tenant context from the magic link
# Create session
session = user_service.create_session(
user.email,
ip_address=ip_address,
user_agent=user_agent,
tenant_id=magic_link_tenant_id
user_agent=user_agent
)
# Send welcome email to first-time users
if is_first_login:
email_service.send_welcome_email(user.email, user.credits)
# Return user info with tenant_id
# Return user info
user_public = UserPublic(
email=user.email,
role=user.role,
@@ -257,7 +187,6 @@ async def verify_magic_link(
display_name=user.display_name,
total_jobs_created=user.total_jobs_created,
total_jobs_completed=user.total_jobs_completed,
tenant_id=user.tenant_id,
)
return VerifyMagicLinkResponse(
@@ -398,328 +327,10 @@ async def create_checkout(
)
@router.post("/made-for-you/checkout", response_model=CreateCheckoutResponse)
async def create_made_for_you_checkout(
request: MadeForYouCheckoutRequest,
stripe_service: StripeService = Depends(get_stripe_service),
):
"""
Create a Stripe checkout session for a made-for-you karaoke video order.
This is the full-service option where Nomad Karaoke handles everything:
- Finding or processing the audio
- Reviewing and correcting lyrics
- Selecting the best instrumental
- Generating the final video
$15 with 24-hour delivery guarantee.
No authentication required - customer email is provided in the request.
"""
if not stripe_service.is_configured():
raise HTTPException(status_code=503, detail="Payment processing is not available")
success, checkout_url, message = stripe_service.create_made_for_you_checkout_session(
customer_email=request.email,
artist=request.artist,
title=request.title,
source_type=request.source_type,
youtube_url=request.youtube_url,
notes=request.notes,
)
if not success or not checkout_url:
raise HTTPException(status_code=400, detail=message)
return CreateCheckoutResponse(
status="success",
checkout_url=checkout_url,
message=message,
)
# =============================================================================
# Stripe Webhooks
# =============================================================================
# Admin email for made-for-you order notifications
ADMIN_EMAIL = "madeforyou@nomadkaraoke.com"
async def _handle_made_for_you_order(
session_id: str,
metadata: dict,
user_service: UserService,
email_service: EmailService,
) -> None:
"""
Handle a completed made-for-you order by creating a job and notifying Andrew.
The made-for-you flow:
1. Job is created with made_for_you=True, owned by admin during processing
2. For search orders: audio search runs, results stored, job pauses at AWAITING_AUDIO_SELECTION
3. Admin receives notification email with link to select audio source
4. Customer receives order confirmation email
5. Admin selects audio in UI, job proceeds through normal pipeline
6. On completion, ownership transfers to customer
For orders with a YouTube URL, workers are triggered immediately (no search needed).
Args:
session_id: Stripe checkout session ID
metadata: Order metadata from Stripe session
user_service: User service for marking session processed
email_service: Email service for notifications
"""
from backend.models.job import JobCreate, JobStatus
from backend.services.job_manager import JobManager
from backend.services.worker_service import get_worker_service
from backend.services.audio_search_service import (
get_audio_search_service,
NoResultsError,
AudioSearchError,
)
from backend.services.storage_service import StorageService
import asyncio
import tempfile
import os
customer_email = metadata.get("customer_email", "")
artist = metadata.get("artist", "Unknown Artist")
title = metadata.get("title", "Unknown Title")
source_type = metadata.get("source_type", "search")
youtube_url = metadata.get("youtube_url")
notes = metadata.get("notes", "")
logger.info(
f"Processing made-for-you order: {artist} - {title} for {customer_email} "
f"(session: {session_id}, source_type: {source_type})"
)
try:
job_manager = JobManager()
worker_service = get_worker_service()
storage_service = StorageService()
# Apply default theme (Nomad) - same as audio_search endpoint
theme_service = get_theme_service()
effective_theme_id = theme_service.get_default_theme_id()
if effective_theme_id:
logger.info(f"Applying default theme '{effective_theme_id}' for made-for-you order")
# Get distribution defaults using centralized service
dist = get_effective_distribution_settings()
# Resolve CDG/TXT defaults based on theme (uses centralized service)
# This ensures made-for-you jobs get the same CDG/TXT defaults as regular jobs
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(effective_theme_id)
# Create job with admin ownership during processing
# CRITICAL: made_for_you=True, user_email=ADMIN_EMAIL, customer_email for delivery
# auto_download=False to pause at audio selection for admin to choose
job_create = JobCreate(
url=youtube_url if youtube_url else None,
artist=artist,
title=title,
user_email=ADMIN_EMAIL, # Admin owns during processing
theme_id=effective_theme_id, # Apply default theme
non_interactive=False, # Admin will review lyrics/instrumental
# Made-for-you specific fields
made_for_you=True, # Flag for ownership transfer on completion
customer_email=customer_email, # Customer email for final delivery
customer_notes=notes if notes else None, # Customer's special requests
# Audio search fields for search-based orders
audio_search_artist=artist if not youtube_url else None,
audio_search_title=title if not youtube_url else None,
auto_download=False, # Pause at audio selection for admin to choose
# CDG/TXT settings (resolved via centralized service)
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
# Distribution settings from centralized defaults
enable_youtube_upload=dist.enable_youtube_upload,
dropbox_path=dist.dropbox_path,
gdrive_folder_id=dist.gdrive_folder_id,
brand_prefix=dist.brand_prefix,
discord_webhook_url=dist.discord_webhook_url,
youtube_description=dist.youtube_description,
)
# Made-for-you jobs are created by admin (via Stripe webhook) - bypass rate limits
job = job_manager.create_job(job_create, is_admin=True)
job_id = job.job_id
logger.info(f"Created made-for-you job {job_id} for {_mask_email(customer_email)} (owned by {_mask_email(ADMIN_EMAIL)})")
# Prepare theme style assets for the job (same as audio_search endpoint)
if effective_theme_id:
try:
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
job_id, effective_theme_id, None # No color overrides for made-for-you
)
theme_update = {
'style_params_gcs_path': style_params_path,
'style_assets': theme_style_assets,
}
if youtube_desc:
theme_update['youtube_description_template'] = youtube_desc
job_manager.update_job(job_id, theme_update)
logger.info(f"Applied theme '{effective_theme_id}' to made-for-you job {job_id}")
except Exception as e:
logger.warning(f"Failed to prepare theme for made-for-you job {job_id}: {e}")
# Mark session as processed for idempotency
# Note: Using internal method since this isn't a credit transaction
user_service._mark_stripe_session_processed(
stripe_session_id=session_id,
email=customer_email,
amount=0 # No credits, just tracking the session
)
# Initialize search_results for later use in email notification
search_results = None
# Handle based on whether we have a YouTube URL or need to search
if youtube_url:
# URL provided - trigger workers directly (no audio selection needed)
logger.info(f"Job {job_id}: YouTube URL provided, triggering workers")
await asyncio.gather(
worker_service.trigger_audio_worker(job_id),
worker_service.trigger_lyrics_worker(job_id)
)
else:
# No URL - use audio search flow, pause for admin selection
# Made-for-you jobs require admin to select audio source
logger.info(f"Job {job_id}: No URL, using audio search for '{artist} - {title}'")
# Update job with audio search fields
job_manager.update_job(job_id, {
'audio_search_artist': artist,
'audio_search_title': title,
'auto_download': False, # Admin must select
})
# Transition to searching state
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.SEARCHING_AUDIO,
progress=5,
message=f"Searching for audio: {artist} - {title}"
)
# Perform audio search
audio_search_service = get_audio_search_service()
try:
search_results = audio_search_service.search(artist, title)
except NoResultsError as e:
# No results found - still transition to AWAITING_AUDIO_SELECTION
# Admin can manually provide audio
logger.warning(f"Job {job_id}: No audio sources found for '{artist} - {title}'")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
progress=10,
message=f"No automatic audio sources found. Manual intervention required."
)
search_results = None
except AudioSearchError as e:
logger.error(f"Job {job_id}: Audio search failed: {e}")
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
progress=10,
message=f"Audio search error. Manual intervention required."
)
search_results = None
if search_results:
# Store search results in state_data for admin to review
results_dicts = [r.to_dict() for r in search_results]
state_data_update = {
'audio_search_results': results_dicts,
'audio_search_count': len(results_dicts),
}
if audio_search_service.last_remote_search_id:
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
job_manager.update_job(job_id, {'state_data': state_data_update})
# Transition to AWAITING_AUDIO_SELECTION for admin to choose
# Do NOT auto-select or download - admin must review and select
logger.info(f"Job {job_id}: Found {len(results_dicts)} audio sources, awaiting admin selection")
# Transition to AWAITING_AUDIO_SELECTION for admin to choose
# Admin will select from results in the UI, then job proceeds
job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
progress=10,
message=f"Found {len(results_dicts)} audio sources. Awaiting admin selection."
)
# Get audio source count for admin notification
# (search_results may be set from the search flow above, or None for YouTube URL orders)
audio_source_count = len(search_results) if search_results else 0
# Generate admin login token for one-click email access (24hr expiry)
admin_login = user_service.create_admin_login_token(
email=ADMIN_EMAIL,
expiry_hours=24,
)
# Send confirmation email to customer using professional template
email_service.send_made_for_you_order_confirmation(
to_email=customer_email,
artist=artist,
title=title,
job_id=job_id,
notes=notes,
)
# Send notification email to admin using professional template
email_service.send_made_for_you_admin_notification(
to_email=ADMIN_EMAIL,
customer_email=customer_email,
artist=artist,
title=title,
job_id=job_id,
admin_login_token=admin_login.token,
notes=notes,
audio_source_count=audio_source_count,
)
logger.info(f"Sent made-for-you order notifications for job {job_id}")
except Exception as e:
logger.error(f"Error processing made-for-you order: {e}", exc_info=True)
# Still try to notify Andrew of the failure
try:
email_service.send_email(
to_email=ADMIN_EMAIL,
subject=f"[FAILED] Made For You Order: {artist} - {title}",
html_content=f"""
<h2>Made-For-You Order Failed</h2>
<p>An error occurred processing this order:</p>
<ul>
<li><strong>Customer:</strong> {customer_email}</li>
<li><strong>Artist:</strong> {artist}</li>
<li><strong>Title:</strong> {title}</li>
<li><strong>Error:</strong> {str(e)}</li>
</ul>
<p>Please manually create this job and notify the customer.</p>
""",
text_content=f"""
Made-For-You Order Failed
Customer: {customer_email}
Artist: {artist}
Title: {title}
Error: {str(e)}
Please manually create this job and notify the customer.
""".strip(),
)
except Exception as email_error:
logger.error(f"Failed to send error notification: {email_error}")
@router.post("/webhooks/stripe")
async def stripe_webhook(
request: Request,
@@ -754,41 +365,30 @@ async def stripe_webhook(
if event_type == "checkout.session.completed":
session = event["data"]["object"]
session_id = session.get("id")
metadata = session.get("metadata", {})
# Idempotency check: Skip if this session was already processed
if session_id and user_service.is_stripe_session_processed(session_id):
logger.info(f"Skipping already processed session: {session_id}")
return {"status": "received", "type": event_type, "note": "already_processed"}
# Check if this is a made-for-you order
if metadata.get("order_type") == "made_for_you":
# Handle made-for-you order - create a job
await _handle_made_for_you_order(
session_id=session_id,
metadata=metadata,
user_service=user_service,
email_service=email_service,
# Process the completed checkout
success, user_email, credits, _ = stripe_service.handle_checkout_completed(session)
if success and user_email and credits > 0:
# Add credits to user account
ok, new_balance, credit_msg = user_service.add_credits(
email=user_email,
amount=credits,
reason="stripe_purchase",
stripe_session_id=session_id,
)
else:
# Handle regular credit purchase
success, user_email, credits, _ = stripe_service.handle_checkout_completed(session)
if success and user_email and credits > 0:
# Add credits to user account
ok, new_balance, credit_msg = user_service.add_credits(
email=user_email,
amount=credits,
reason="stripe_purchase",
stripe_session_id=session_id,
)
if ok:
# Send confirmation email
email_service.send_credits_added(user_email, credits, new_balance)
logger.info(f"Added {credits} credits to {user_email}, new balance: {new_balance}")
else:
logger.error(f"Failed to add credits: {credit_msg}")
if ok:
# Send confirmation email
email_service.send_credits_added(user_email, credits, new_balance)
logger.info(f"Added {credits} credits to {user_email}, new balance: {new_balance}")
else:
logger.error(f"Failed to add credits: {credit_msg}")
elif event_type == "checkout.session.expired":
logger.info(f"Checkout session expired: {event['data']['object'].get('id')}")
@@ -823,9 +423,6 @@ async def enroll_beta_tester(
Returns free credits and optionally a session token for new users.
"""
from backend.services.email_validation_service import get_email_validation_service
from backend.services.rate_limit_service import get_rate_limit_service
# Check if email service is configured
if not email_service.is_configured():
logger.error("Email service not configured - cannot send beta welcome emails")
@@ -835,63 +432,6 @@ async def enroll_beta_tester(
)
email = request.email.lower()
email_validation = get_email_validation_service()
rate_limit_service = get_rate_limit_service()
# ----- ABUSE PREVENTION CHECKS -----
# 1. Validate email (disposable domain, blocked email checks)
is_valid, error_message = email_validation.validate_email_for_beta(email)
if not is_valid:
logger.warning(f"Beta enrollment rejected - email validation failed: {email} - {error_message}")
raise HTTPException(
status_code=400,
detail=error_message
)
# 2. Check IP blocking
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Beta enrollment rejected - IP blocked: {ip_address}")
raise HTTPException(
status_code=403,
detail="Access denied from this location"
)
# 3. Check for E2E test bypass (allows automated testing to skip IP rate limit)
from backend.config import settings
e2e_bypass_key = http_request.headers.get("X-E2E-Bypass-Key")
skip_ip_rate_limit = False
if e2e_bypass_key and settings.e2e_bypass_key:
if e2e_bypass_key == settings.e2e_bypass_key:
logger.info(f"Beta enrollment: E2E bypass key validated for {_mask_email(email)}")
skip_ip_rate_limit = True
else:
logger.warning(f"Beta enrollment: Invalid E2E bypass key attempted for {_mask_email(email)}")
# 4. Check IP-based enrollment rate limit (1 per 24h per IP)
if ip_address and not skip_ip_rate_limit:
allowed, remaining, message = rate_limit_service.check_beta_ip_limit(ip_address)
if not allowed:
logger.warning(f"Beta enrollment rejected - IP rate limit: {ip_address} - {message}")
raise HTTPException(
status_code=429,
detail="Too many beta enrollments from your location. Please try again tomorrow."
)
# 5. Check for duplicate enrollment via normalized email
normalized_email = email_validation.normalize_email(email)
if normalized_email != email:
# Check if normalized version is already enrolled
normalized_user = user_service.get_user(normalized_email)
if normalized_user and normalized_user.is_beta_tester:
logger.warning(f"Beta enrollment rejected - normalized email already enrolled: {email} -> {normalized_email}")
raise HTTPException(
status_code=400,
detail="An account with this email address is already enrolled in the beta program"
)
# ----- END ABUSE PREVENTION CHECKS -----
# Validate acceptance
if not request.accept_corrections_work:
@@ -916,7 +456,8 @@ async def enroll_beta_tester(
detail="You are already enrolled in the beta program"
)
# Get client info (ip_address already set above in abuse prevention)
# Get client info
ip_address = http_request.client.host if http_request.client else None
user_agent = http_request.headers.get("user-agent")
# Enroll as beta tester
@@ -936,13 +477,6 @@ async def enroll_beta_tester(
reason="beta_tester_enrollment",
)
# Record IP enrollment for rate limiting
if ip_address:
try:
rate_limit_service.record_beta_enrollment(ip_address, email)
except Exception as e:
logger.warning(f"Failed to record beta IP enrollment: {e}")
# Create session for the user (so they can start using the service immediately)
session = user_service.create_session(email, ip_address=ip_address, user_agent=user_agent)
@@ -1129,7 +663,6 @@ async def list_users(
sort_by: str = "created_at",
sort_order: str = "desc",
include_inactive: bool = False,
exclude_test: bool = True,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
@@ -1143,7 +676,6 @@ async def list_users(
sort_by: Field to sort by (created_at, last_login_at, credits, email)
sort_order: Sort direction (asc, desc)
include_inactive: Include disabled users
exclude_test: If True (default), exclude test users (e.g., @inbox.testmail.app)
"""
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
@@ -1172,14 +704,9 @@ async def list_users(
else:
query = query.order_by("created_at", direction=direction)
# Get all docs and filter in Python
# Get total count (without pagination) for has_more calculation
# Note: This is expensive for large datasets, consider caching
all_docs = list(query.stream())
# Filter out test users if exclude_test is True
if exclude_test:
all_docs = [d for d in all_docs if not is_test_email(d.to_dict().get('email', ''))]
total_count = len(all_docs)
# Apply pagination manually (Firestore doesn't support offset well)
@@ -1415,64 +942,44 @@ async def list_beta_feedback(
@router.get("/admin/beta/stats")
async def get_beta_stats(
exclude_test: bool = True,
auth_data: Tuple[str, UserType, int] = Depends(require_admin),
user_service: UserService = Depends(get_user_service),
):
"""
Get beta tester program statistics (admin only).
Args:
exclude_test: If True (default), exclude test users from beta stats
"""
from google.cloud.firestore_v1 import FieldFilter
from google.cloud.firestore_v1 import aggregation
# Count beta testers by status using efficient aggregation queries
users_collection = user_service.db.collection(USERS_COLLECTION)
if exclude_test:
# Stream and filter in Python since Firestore doesn't support "not ends with"
all_beta_users = []
for doc in users_collection.where(filter=FieldFilter("is_beta_tester", "==", True)).stream():
data = doc.to_dict()
if not is_test_email(data.get("email", "")):
all_beta_users.append(data)
# Helper function to get count using aggregation
def get_count(query) -> int:
agg_query = aggregation.AggregationQuery(query)
agg_query.count(alias="count")
results = agg_query.get()
return results[0][0].value if results else 0
total_beta_testers = len(all_beta_users)
active_testers = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "active")
pending_feedback = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "pending_feedback")
completed_feedback = sum(1 for u in all_beta_users if u.get("beta_tester_status") == "completed")
total_beta_testers = get_count(
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
)
# Filter feedback by non-test users
all_feedback = []
for doc in user_service.db.collection("beta_feedback").stream():
data = doc.to_dict()
if not is_test_email(data.get("user_email", "")):
all_feedback.append(data)
feedback_docs = all_feedback
else:
# Use efficient aggregation queries when including test data
def get_count(query) -> int:
agg_query = aggregation.AggregationQuery(query)
agg_query.count(alias="count")
results = agg_query.get()
return results[0][0].value if results else 0
active_testers = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "active"))
)
total_beta_testers = get_count(
users_collection.where(filter=FieldFilter("is_beta_tester", "==", True))
)
active_testers = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "active"))
)
pending_feedback = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "pending_feedback"))
)
completed_feedback = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "completed"))
)
feedback_docs = [doc.to_dict() for doc in user_service.db.collection("beta_feedback").stream()]
pending_feedback = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "pending_feedback"))
)
completed_feedback = get_count(
users_collection.where(filter=FieldFilter("beta_tester_status", "==", "completed"))
)
# Get average ratings from feedback
feedback_docs = list(user_service.db.collection("beta_feedback").stream())
# Calculate average ratings from feedback
avg_overall = 0
avg_ease = 0
avg_accuracy = 0
@@ -1480,7 +987,8 @@ async def get_beta_stats(
if feedback_docs:
total = len(feedback_docs)
for data in feedback_docs:
for doc in feedback_docs:
data = doc.to_dict()
avg_overall += data.get("overall_rating", 0)
avg_ease += data.get("ease_of_use_rating", 0)
avg_accuracy += data.get("lyrics_accuracy_rating", 0)
+1 -34
View File
@@ -68,12 +68,6 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -101,40 +95,13 @@ class Settings(BaseSettings):
# Default values for web service jobs (YouTube/Dropbox distribution)
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Rate Limiting Configuration
# Enable/disable rate limiting system-wide (useful for development)
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
# Maximum jobs a user can create per day (0 = unlimited)
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
# Maximum YouTube uploads system-wide per day (0 = unlimited)
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
# Maximum beta enrollments from same IP per day (0 = unlimited)
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Default CDG/TXT generation settings
# When True, CDG and TXT packages are generated by default (when a theme is set)
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
# Push Notifications Configuration
# When enabled, users can subscribe to push notifications for job status updates
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
# Maximum number of push subscriptions per user (oldest removed when exceeded)
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
# VAPID subject (email or URL for push service to contact)
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
# Secret Manager cache
_secret_cache: Dict[str, str] = {}
-66
View File
@@ -1,66 +0,0 @@
"""
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for rate limiting and validation errors.
"""
class RateLimitExceededError(Exception):
"""
Raised when a rate limit is exceeded.
Includes information about when the limit will reset for client retry logic.
Attributes:
message: Human-readable error message
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
remaining_seconds: Seconds until the limit resets (for Retry-After header)
current_count: Current usage count
limit_value: The limit that was exceeded
"""
def __init__(
self,
message: str,
limit_type: str = "unknown",
remaining_seconds: int = 0,
current_count: int = 0,
limit_value: int = 0
):
self.message = message
self.limit_type = limit_type
self.remaining_seconds = remaining_seconds
self.current_count = current_count
self.limit_value = limit_value
super().__init__(message)
class EmailValidationError(Exception):
"""
Raised when email validation fails.
Attributes:
message: Human-readable error message
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
"""
def __init__(self, message: str, reason: str = "invalid"):
self.message = message
self.reason = reason
super().__init__(message)
class IPBlockedError(Exception):
"""
Raised when a request comes from a blocked IP address.
Attributes:
message: Human-readable error message
ip_address: The blocked IP address (may be partially masked)
"""
def __init__(self, message: str, ip_address: str = ""):
self.message = message
self.ip_address = ip_address
super().__init__(message)
+4 -74
View File
@@ -7,15 +7,10 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
from backend.services.nltk_preloader import preload_all_nltk_resources
from backend.services.langfuse_preloader import preload_langfuse_handler
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware
from backend.workers.registry import worker_registry
from backend.version import VERSION
@@ -72,28 +67,7 @@ async def lifespan(app: FastAPI):
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Preload NLP models and resources to avoid cold start delays
# See docs/archive/2026-01-08-performance-investigation.md for background
# 1. SpaCy model (60+ second delay without preload)
try:
preload_spacy_model("en_core_web_sm")
except Exception as e:
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
# 2. NLTK cmudict (50-100+ second delay without preload)
try:
preload_all_nltk_resources()
except Exception as e:
logger.warning(f"NLTK preload failed (will load lazily): {e}")
# 3. Langfuse callback handler (200+ second delay without preload)
try:
preload_langfuse_handler()
except Exception as e:
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
# Validate OAuth credentials (non-blocking)
try:
await validate_credentials_on_startup()
@@ -101,23 +75,8 @@ async def lifespan(app: FastAPI):
logger.error(f"Credential validation failed: {e}")
yield
# Shutdown - wait for any active workers to complete before terminating
# This prevents Cloud Run from killing workers mid-processing
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 600s)...")
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
f"Remaining workers: {worker_registry.get_active_workers()}"
)
else:
logger.info("No active workers, proceeding with shutdown")
# Shutdown
logger.info("Shutting down karaoke generation backend")
@@ -145,9 +104,6 @@ app.add_middleware(
# Add audit logging middleware (captures all requests with request_id for correlation)
app.add_middleware(AuditLoggingMiddleware)
# Add tenant detection middleware (extracts tenant from subdomain/headers)
app.add_middleware(TenantMiddleware)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
@@ -159,32 +115,6 @@ app.include_router(audio_search.router, prefix="/api") # Audio search (artist+t
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
# Exception handler for rate limiting
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import RateLimitExceededError
@app.exception_handler(RateLimitExceededError)
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
"""Handle rate limit exceeded errors with 429 status."""
return JSONResponse(
status_code=429,
content={
"detail": exc.message,
"limit_type": exc.limit_type,
"current_count": exc.current_count,
"limit_value": exc.limit_value,
},
headers={
"Retry-After": str(exc.remaining_seconds),
} if exc.remaining_seconds > 0 else None,
)
@app.get("/")
+1 -7
View File
@@ -1,11 +1,5 @@
"""Middleware package for FastAPI application."""
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
__all__ = [
"AuditLoggingMiddleware",
"TenantMiddleware",
"get_tenant_from_request",
"get_tenant_config_from_request",
]
__all__ = ["AuditLoggingMiddleware"]
-192
View File
@@ -1,192 +0,0 @@
"""
Tenant detection middleware for white-label B2B portals.
This middleware extracts the tenant ID from incoming requests and attaches
the tenant configuration to the request state for use by downstream handlers.
Tenant detection priority:
1. X-Tenant-ID header (explicitly set by frontend)
2. `tenant` query parameter (for development/testing)
3. Host header subdomain detection (production flow)
The middleware is non-blocking - if no tenant is detected, the request
proceeds as a default Nomad Karaoke request (tenant_id = None).
"""
import logging
import os
from typing import Optional
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from backend.services.tenant_service import get_tenant_service
from backend.models.tenant import TenantConfig
logger = logging.getLogger(__name__)
# Only allow query param tenant override in non-production environments
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
os.environ.get("ENVIRONMENT", "").lower() == "production"
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
class TenantMiddleware(BaseHTTPMiddleware):
"""
Middleware that detects tenant from request and attaches config to state.
After this middleware runs, routes can access:
- request.state.tenant_id: str or None
- request.state.tenant_config: TenantConfig or None
Usage in route handlers:
```python
@router.get("/something")
async def handler(request: Request):
tenant_id = getattr(request.state, "tenant_id", None)
tenant_config = getattr(request.state, "tenant_config", None)
```
"""
async def dispatch(self, request: Request, call_next) -> Response:
# Extract tenant ID from request
tenant_id = self._extract_tenant_id(request)
# Load tenant config if tenant detected
tenant_config: Optional[TenantConfig] = None
if tenant_id:
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(tenant_id)
if tenant_config and not tenant_config.is_active:
# Tenant exists but is inactive - treat as default
logger.warning(f"Inactive tenant requested: {tenant_id}")
tenant_id = None
tenant_config = None
# Attach to request state
request.state.tenant_id = tenant_id
request.state.tenant_config = tenant_config
# Log tenant detection for debugging
if tenant_id:
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
# Process request
response = await call_next(request)
# Add tenant ID to response headers for debugging
if tenant_id:
response.headers["X-Tenant-ID"] = tenant_id
return response
def _extract_tenant_id(self, request: Request) -> Optional[str]:
"""
Extract tenant ID from request using priority-based detection.
Returns:
Tenant ID if detected, None for default Nomad Karaoke
"""
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
header_tenant = request.headers.get("X-Tenant-ID")
if header_tenant:
return header_tenant.lower().strip()
# Priority 2: Query parameter (for development/testing only)
# Disabled in production to prevent tenant spoofing
if not IS_PRODUCTION:
query_tenant = request.query_params.get("tenant")
if query_tenant:
return query_tenant.lower().strip()
# Priority 3: Host header subdomain detection
host = request.headers.get("Host", "")
return self._extract_tenant_from_host(host)
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
"""
Extract tenant ID from Host header subdomain.
Patterns handled:
- vocalstar.nomadkaraoke.com -> vocalstar
- vocalstar.gen.nomadkaraoke.com -> vocalstar
- localhost:3000 -> None (local dev)
- gen.nomadkaraoke.com -> None (main app)
- api.nomadkaraoke.com -> None (API)
Returns:
Tenant ID if subdomain matches tenant, None otherwise
"""
if not host:
return None
# Normalize
host_lower = host.lower()
# Remove port if present
if ":" in host_lower:
host_lower = host_lower.split(":")[0]
# Check if this is a nomadkaraoke.com domain
if "nomadkaraoke.com" not in host_lower:
return None
# Split into parts
parts = host_lower.split(".")
# Need at least 3 parts: subdomain.nomadkaraoke.com
if len(parts) < 3:
return None
# First part is the potential tenant ID
potential_tenant = parts[0]
# Skip known non-tenant subdomains
if potential_tenant in NON_TENANT_SUBDOMAINS:
return None
# Verify tenant exists in GCS
tenant_service = get_tenant_service()
if tenant_service.tenant_exists(potential_tenant):
return potential_tenant
return None
def get_tenant_from_request(request: Request) -> Optional[str]:
"""
Helper function to get tenant ID from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_from_request
@router.get("/something")
async def handler(request: Request):
tenant_id = get_tenant_from_request(request)
```
"""
return getattr(request.state, "tenant_id", None)
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
"""
Helper function to get tenant config from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_config_from_request
@router.get("/something")
async def handler(request: Request):
config = get_tenant_config_from_request(request)
if config and not config.features.audio_search:
raise HTTPException(403, "Audio search not available")
```
"""
return getattr(request.state, "tenant_config", None)
+6 -44
View File
@@ -9,8 +9,6 @@ from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, validator
from karaoke_gen.utils import normalize_text
class JobStatus(str, Enum):
"""
@@ -207,9 +205,6 @@ class Job(BaseModel):
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -279,16 +274,7 @@ class Job(BaseModel):
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -486,12 +472,7 @@ class JobCreate(BaseModel):
prep_only: bool = False # Stop after review, don't run finalisation
finalise_only: bool = False # Skip prep, run only finalisation
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -503,33 +484,14 @@ class JobCreate(BaseModel):
- server_version: Current server version
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals
tenant_id: Optional[str] = None # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
"""Validate URL is not empty."""
@validator('url', 'artist', 'title')
def validate_inputs(cls, v):
"""Validate string inputs are not empty."""
if v is not None and isinstance(v, str) and not v.strip():
raise ValueError("Field cannot be empty string")
return v.strip() if isinstance(v, str) else v
@validator('artist', 'title')
def normalize_artist_title(cls, v):
"""Normalize artist/title text to standardize Unicode characters.
This ensures consistent data storage by converting:
- Curly quotes -> straight quotes
- Various dashes -> hyphen
- Unusual whitespace -> regular space
"""
if v is not None and isinstance(v, str):
if not v.strip():
raise ValueError("Field cannot be empty string")
# normalize_text handles stripping and Unicode normalization
return normalize_text(v)
return v
class JobResponse(BaseModel):
"""Job response model."""
+2 -2
View File
@@ -14,7 +14,7 @@ class URLSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -28,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
-208
View File
@@ -1,208 +0,0 @@
"""
Tenant data models for white-label B2B portals.
Tenants are B2B customers who get their own branded portal with custom
configuration. Each tenant has:
- Custom branding (logo, colors, site title)
- Feature flags (enable/disable audio search, distribution, etc.)
- Default settings (theme, distribution mode)
- Auth configuration (allowed email domains, fixed tokens)
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
"""
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel, Field
class TenantBranding(BaseModel):
"""Branding configuration for a tenant's portal."""
logo_url: Optional[str] = Field(
None, description="URL to tenant's logo image (PNG with transparency preferred)"
)
logo_height: int = Field(40, description="Logo height in pixels for header display")
primary_color: str = Field(
"#ff5bb8", description="Primary brand color (hex format)"
)
secondary_color: str = Field(
"#8b5cf6", description="Secondary brand color (hex format)"
)
accent_color: Optional[str] = Field(
None, description="Accent color for highlights (hex format)"
)
background_color: Optional[str] = Field(
None, description="Custom background color (defaults to dark theme)"
)
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
site_title: str = Field(
"Karaoke Generator", description="Browser tab title"
)
tagline: Optional[str] = Field(
None, description="Optional tagline shown below logo"
)
class TenantFeatures(BaseModel):
"""Feature flags controlling what's available in the tenant's portal."""
# Audio input methods
audio_search: bool = Field(
True, description="Enable audio search (flacfetch integration)"
)
file_upload: bool = Field(True, description="Enable direct file upload")
youtube_url: bool = Field(True, description="Enable YouTube URL input")
# Distribution options
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
# Customization options
theme_selection: bool = Field(
True, description="Allow user to select theme (false = use default)"
)
color_overrides: bool = Field(
True, description="Allow user to customize colors"
)
# Output formats
enable_cdg: bool = Field(True, description="Generate CDG format output")
enable_4k: bool = Field(True, description="Generate 4K video output")
# Advanced features
admin_access: bool = Field(
False, description="Allow access to admin dashboard"
)
class TenantDefaults(BaseModel):
"""Default settings applied to all jobs for this tenant."""
theme_id: Optional[str] = Field(
None, description="Default theme ID (required if theme_selection=false)"
)
locked_theme: Optional[str] = Field(
None,
description="If set, users cannot change theme - always uses this theme ID"
)
distribution_mode: str = Field(
"all",
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
)
brand_prefix: Optional[str] = Field(
None, description="Prefix for output filenames (e.g., 'VSTAR')"
)
youtube_description_template: Optional[str] = Field(
None, description="Default YouTube description template"
)
class TenantAuth(BaseModel):
"""Authentication configuration for a tenant."""
allowed_email_domains: List[str] = Field(
default_factory=list,
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
)
require_email_domain: bool = Field(
True,
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
)
fixed_token_ids: List[str] = Field(
default_factory=list,
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
)
sender_email: Optional[str] = Field(
None,
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
)
class TenantConfig(BaseModel):
"""Complete configuration for a white-label tenant."""
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
subdomain: str = Field(
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
)
is_active: bool = Field(True, description="Whether tenant portal is active")
branding: TenantBranding = Field(default_factory=TenantBranding)
features: TenantFeatures = Field(default_factory=TenantFeatures)
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
auth: TenantAuth = Field(default_factory=TenantAuth)
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
def get_sender_email(self) -> str:
"""Get the email sender address for this tenant."""
if self.auth.sender_email:
return self.auth.sender_email
# Default pattern: {tenant_id}@nomadkaraoke.com
return f"{self.id}@nomadkaraoke.com"
def is_email_allowed(self, email: str) -> bool:
"""Check if an email address is allowed for this tenant."""
if not self.auth.allowed_email_domains:
# No domain restrictions
return True
email_lower = email.lower()
for domain in self.auth.allowed_email_domains:
if email_lower.endswith(f"@{domain.lower()}"):
return True
return not self.auth.require_email_domain
class TenantPublicConfig(BaseModel):
"""
Public tenant configuration returned to frontend.
This excludes sensitive auth details like token IDs.
"""
id: str
name: str
subdomain: str
is_active: bool
branding: TenantBranding
features: TenantFeatures
defaults: TenantDefaults
# Auth info limited to what frontend needs
allowed_email_domains: List[str] = Field(default_factory=list)
@classmethod
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
"""Create public config from full config."""
return cls(
id=config.id,
name=config.name,
subdomain=config.subdomain,
is_active=config.is_active,
branding=config.branding,
features=config.features,
defaults=TenantDefaults(
theme_id=config.defaults.theme_id,
locked_theme=config.defaults.locked_theme,
distribution_mode=config.defaults.distribution_mode,
# Don't expose brand_prefix or youtube_description_template
),
allowed_email_domains=config.auth.allowed_email_domains,
)
class TenantConfigResponse(BaseModel):
"""Response from GET /api/tenant/config endpoint."""
tenant: Optional[TenantPublicConfig] = Field(
None, description="Tenant config if found, null for default Nomad Karaoke"
)
is_default: bool = Field(
True, description="True if using default Nomad Karaoke config"
)
+2 -38
View File
@@ -8,9 +8,9 @@ Supports:
- Stripe integration for payments
- Beta tester program with feedback collection
"""
from datetime import datetime, timezone
from datetime import datetime
from enum import Enum
from typing import Optional, List, Dict
from typing import Optional, List
from pydantic import BaseModel, Field
@@ -39,37 +39,17 @@ class CreditTransaction(BaseModel):
created_by: Optional[str] = None # Admin email if granted by admin
class PushSubscription(BaseModel):
"""
Web Push subscription for a user's device.
Stores the push subscription endpoint and encryption keys needed
to send push notifications to the user's browser/device.
"""
endpoint: str # Push service endpoint URL
keys: Dict[str, str] # p256dh and auth keys for encryption
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
last_used_at: Optional[datetime] = None # Last time a notification was sent
class User(BaseModel):
"""
User model stored in Firestore.
Users are identified by email address. Authentication is via magic links
(no passwords). Credits are consumed when creating karaoke jobs.
For multi-tenant white-label portals, users are scoped to a tenant.
Users with tenant_id=None are default Nomad Karaoke users.
"""
email: str # Primary identifier
role: UserRole = UserRole.USER
credits: int = 0
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Stripe integration
stripe_customer_id: Optional[str] = None
@@ -100,10 +80,6 @@ class User(BaseModel):
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
beta_feedback_email_sent: bool = False
# Push notification subscriptions (Web Push API)
# Users can subscribe from multiple devices/browsers
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
class MagicLinkToken(BaseModel):
"""
@@ -111,9 +87,6 @@ class MagicLinkToken(BaseModel):
Tokens are short-lived (15 minutes) and single-use.
Stored in Firestore with automatic TTL.
For multi-tenant portals, the tenant_id determines which portal
the user will be redirected to after verification.
"""
token: str # Secure random token
email: str
@@ -124,9 +97,6 @@ class MagicLinkToken(BaseModel):
ip_address: Optional[str] = None
user_agent: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
class Session(BaseModel):
"""
@@ -134,8 +104,6 @@ class Session(BaseModel):
Sessions are created after successful magic link verification.
Sessions expire after 7 days of inactivity or 30 days absolute.
For multi-tenant portals, sessions are scoped to a tenant.
"""
token: str # Secure random session token
user_email: str
@@ -146,9 +114,6 @@ class Session(BaseModel):
user_agent: Optional[str] = None
is_active: bool = True
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Pydantic models for API requests/responses
@@ -184,7 +149,6 @@ class UserPublic(BaseModel):
display_name: Optional[str] = None
total_jobs_created: int = 0
total_jobs_completed: int = 0
tenant_id: Optional[str] = None
class AddCreditsRequest(BaseModel):
File diff suppressed because it is too large. Load diff
@@ -1,646 +0,0 @@
"""
Email Validation Service.
Provides email validation, normalization, and blocklist checking
to prevent abuse during beta enrollment and other flows.
"""
import hashlib
import logging
import re
from datetime import datetime, timezone
from typing import Optional, Tuple, Set
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
def _mask_email(email: str) -> str:
"""Mask an email address for privacy-safe logging."""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
if len(local) <= 2:
masked_local = "*" * len(local)
else:
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
return f"{masked_local}@{domain}"
# Firestore collection for blocklist data
BLOCKLISTS_COLLECTION = "blocklists"
BLOCKLIST_CONFIG_DOC = "config"
# Default disposable email domains (can be extended via admin UI)
DEFAULT_DISPOSABLE_DOMAINS = {
# Popular temporary email services
"10minutemail.com",
"10minutemail.net",
"guerrillamail.com",
"guerrillamail.org",
"guerrillamail.net",
"guerrillamail.biz",
"guerrillamailblock.com",
"tempmail.com",
"tempmail.net",
"temp-mail.org",
"temp-mail.io",
"throwawaymail.com",
"mailinator.com",
"mailinator.net",
"mailinator.org",
"mailinator.info",
"maildrop.cc",
"yopmail.com",
"yopmail.fr",
"yopmail.net",
"fakeinbox.com",
"fakemailgenerator.com",
"dispostable.com",
"getairmail.com",
"getnada.com",
"mohmal.com",
"trashmail.com",
"trashmail.net",
"trashmail.org",
"sharklasers.com",
"grr.la",
"guerrillamail.de",
"pokemail.net",
"spam4.me",
"spamgourmet.com",
"mytrashmail.com",
"mailnesia.com",
"mailcatch.com",
"mintemail.com",
"tempr.email",
"tempail.com",
"emailondeck.com",
"incognitomail.com",
"inboxalias.com",
"33mail.com",
"spamex.com",
"spamfree24.org",
"spamspot.com",
"mailnull.com",
"mailsac.com",
"emailfake.com",
"fakemail.fr",
"tempinbox.com",
"throwaway.email",
"burnermail.io",
"anonbox.net",
"anonymbox.com",
"discard.email",
"discardmail.com",
"discardmail.de",
"mailexpire.com",
"mailforspam.com",
"meltmail.com",
"mt2009.com",
"mt2014.com",
"nospam.ze.tc",
"nospamfor.us",
"nowmymail.com",
"receiveee.com",
"safe-mail.net",
"spamavert.com",
"spambob.com",
"spambog.com",
"spambox.us",
"spamcannon.com",
"spamcannon.net",
"spamcon.org",
"spamcorptastic.com",
"spamday.com",
"spamfree.eu",
"spamherelots.com",
"spamhereplease.com",
"spamhole.com",
"spamify.com",
"spaminator.de",
"spamkill.info",
"spaml.com",
"spaml.de",
"spamoff.de",
"spamobox.com",
"spamslicer.com",
"spamstack.net",
"spamthis.co.uk",
"spamthisplease.com",
"supergreatmail.com",
"suremail.info",
"teleworm.us",
"tempemail.co.za",
"tempemail.net",
"tempmailaddress.com",
"tempmailo.com",
"thankyou2010.com",
"thisisnotmyrealemail.com",
"tm.slsrs.ru",
"tmpeml.info",
"trash-mail.at",
"trash-mail.de",
"trash2009.com",
"trashemail.de",
"trashmail.at",
"trashmailer.com",
"wegwerfmail.de",
"wegwerfmail.net",
"wegwerfmail.org",
"wh4f.org",
"willhackforfood.biz",
"willselfdestruct.com",
"xmaily.com",
"xyzfree.net",
"yep.it",
"yogamaven.com",
"yuurok.com",
"zehnminutenmail.de",
"zippymail.info",
# Additional common ones
"mailnator.com",
"bugmenot.com",
"dodgeit.com",
"dodgit.com",
"e4ward.com",
"emailsensei.com",
"hushmail.com",
"jetable.org",
"kasmail.com",
"mailblock.net",
"mailcatch.com",
"mymailoasis.com",
"nervmich.net",
"nervtmansen.de",
"oneoffemail.com",
"pookmail.com",
"shortmail.net",
"sneakemail.com",
"sogetthis.com",
"tempemailaddress.com",
"tempomail.fr",
"temporaryemail.net",
"temporaryforwarding.com",
"temporaryinbox.com",
"thankyou2010.com",
"tyldd.com",
"uggsrock.com",
"veryrealemail.com",
"yourewronghereswhy.com",
}
# Gmail-like domains that support alias normalization
GMAIL_LIKE_DOMAINS = {
"gmail.com",
"googlemail.com",
}
class EmailValidationService:
"""
Service for email validation and abuse prevention.
Features:
- Email normalization (Gmail alias stripping)
- Disposable domain detection
- Blocklist checking (emails, IPs)
- IP-based enrollment rate limiting
"""
_instance: Optional["EmailValidationService"] = None
_blocklist_cache: Optional[dict] = None
_blocklist_cache_time: Optional[datetime] = None
_cache_ttl_seconds: int = 300 # 5 minutes
def __init__(self, db: Optional[firestore.Client] = None):
"""Initialize the email validation service."""
if db is None:
from backend.services.firestore_service import get_firestore_client
db = get_firestore_client()
self.db = db
@classmethod
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
"""Get singleton instance of the service."""
if cls._instance is None:
cls._instance = cls(db)
return cls._instance
def normalize_email(self, email: str) -> str:
"""
Normalize an email address.
For Gmail-like domains:
- Removes dots from local part
- Removes everything after + in local part
- Converts to lowercase
For other domains:
- Only converts to lowercase
Args:
email: Email address to normalize
Returns:
Normalized email address
Examples:
j.o.h.n+spam@gmail.com -> john@gmail.com
John.Doe@example.com -> john.doe@example.com
"""
if not email or "@" not in email:
return email.lower().strip() if email else ""
email = email.lower().strip()
local, domain = email.rsplit("@", 1)
if domain in GMAIL_LIKE_DOMAINS:
# Remove dots from local part
local = local.replace(".", "")
# Remove everything after +
if "+" in local:
local = local.split("+")[0]
return f"{local}@{domain}"
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
"""
Get blocklist configuration from Firestore.
Uses caching to reduce Firestore reads.
Returns:
Dict with disposable_domains, blocked_emails, blocked_ips sets
"""
now = datetime.now(timezone.utc)
# Check cache validity
if (
not force_refresh
and self._blocklist_cache is not None
and self._blocklist_cache_time is not None
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
):
return self._blocklist_cache
# Fetch from Firestore
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
config = {
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
"blocked_emails": set(data.get("blocked_emails", [])),
"blocked_ips": set(data.get("blocked_ips", [])),
}
else:
# Use defaults if no config exists
config = {
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
"blocked_emails": set(),
"blocked_ips": set(),
}
# Update cache
EmailValidationService._blocklist_cache = config
EmailValidationService._blocklist_cache_time = now
return config
def is_disposable_domain(self, email: str) -> bool:
"""
Check if an email uses a disposable domain.
Args:
email: Email address to check
Returns:
True if the domain is known to be disposable
"""
if not email or "@" not in email:
return False
domain = email.lower().split("@")[-1]
config = self.get_blocklist_config()
return domain in config["disposable_domains"]
def is_email_blocked(self, email: str) -> bool:
"""
Check if an email is explicitly blocked.
Checks both the raw email and normalized version.
Args:
email: Email address to check
Returns:
True if the email is blocked
"""
if not email:
return False
config = self.get_blocklist_config()
email_lower = email.lower().strip()
email_normalized = self.normalize_email(email)
return (
email_lower in config["blocked_emails"]
or email_normalized in config["blocked_emails"]
)
def is_ip_blocked(self, ip_address: str) -> bool:
"""
Check if an IP address is blocked.
Args:
ip_address: IP address to check
Returns:
True if the IP is blocked
"""
if not ip_address:
return False
config = self.get_blocklist_config()
return ip_address in config["blocked_ips"]
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
"""
Validate an email for beta enrollment.
Performs all validation checks:
1. Basic format validation
2. Disposable domain check
3. Blocked email check
Args:
email: Email address to validate
Returns:
Tuple of (is_valid, error_message)
"""
if not email or "@" not in email:
return False, "Invalid email format"
# Check disposable domain
if self.is_disposable_domain(email):
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
return False, "Disposable email addresses are not allowed"
# Check blocked email
if self.is_email_blocked(email):
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
return False, "This email address is not allowed"
return True, ""
def hash_ip(self, ip_address: str) -> str:
"""
Hash an IP address for privacy-preserving storage.
Args:
ip_address: IP address to hash
Returns:
SHA-256 hash of the IP
"""
return hashlib.sha256(ip_address.encode()).hexdigest()
# -------------------------------------------------------------------------
# Blocklist Management (Admin)
# -------------------------------------------------------------------------
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
"""Add a domain to the disposable domains blocklist."""
domain = domain.lower().strip()
if not domain:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("disposable_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["disposable_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added disposable domain: {domain} by {admin_email}")
return True
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
"""Remove a domain from the disposable domains blocklist."""
domain = domain.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
domains = set(data.get("disposable_domains", []))
if domain not in domains:
return
result["found"] = True
domains.discard(domain)
data["disposable_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
return True
def add_blocked_email(self, email: str, admin_email: str) -> bool:
"""Add an email to the blocked emails list."""
email = email.lower().strip()
if not email:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
emails = set(data.get("blocked_emails", []))
else:
data = {}
emails = set()
emails.add(email)
data["blocked_emails"] = list(emails)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added blocked email: {email} by {admin_email}")
return True
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
"""Remove an email from the blocked emails list."""
email = email.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
emails = set(data.get("blocked_emails", []))
if email not in emails:
return
result["found"] = True
emails.discard(email)
data["blocked_emails"] = list(emails)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed blocked email: {email} by {admin_email}")
return True
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
"""Add an IP address to the blocked IPs list."""
ip_address = ip_address.strip()
if not ip_address:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
ips = set(data.get("blocked_ips", []))
else:
data = {}
ips = set()
ips.add(ip_address)
data["blocked_ips"] = list(ips)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
return True
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
"""Remove an IP address from the blocked IPs list."""
ip_address = ip_address.strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
ips = set(data.get("blocked_ips", []))
if ip_address not in ips:
return
result["found"] = True
ips.discard(ip_address)
data["blocked_ips"] = list(ips)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
return True
def get_blocklist_stats(self) -> dict:
"""Get statistics about current blocklists."""
config = self.get_blocklist_config(force_refresh=True)
return {
"disposable_domains_count": len(config["disposable_domains"]),
"blocked_emails_count": len(config["blocked_emails"]),
"blocked_ips_count": len(config["blocked_ips"]),
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
}
def get_email_validation_service(
db: Optional[firestore.Client] = None,
) -> EmailValidationService:
"""Get the singleton EmailValidationService instance."""
return EmailValidationService.get_instance(db)
+8 -15
View File
@@ -40,9 +40,6 @@ class EncodingInput:
# Output directory
output_dir: str = ""
# Instrumental selection (clean, with_backing, or custom)
instrumental_selection: str = "clean"
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@@ -331,7 +328,6 @@ class GCEEncodingBackend(EncodingBackend):
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
"instrumental_selection": input_config.instrumental_selection,
}
# Submit and wait for completion
@@ -355,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
@@ -364,18 +360,15 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
filename_lower = filename.lower()
# Map filename patterns to output format keys
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
output_files["mkv_4k"] = path
else:
output_files["mp4_4k_lossless"] = path
elif "lossy 4k" in filename_lower:
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename_lower:
elif "720p" in filename:
output_files["mp4_720p"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
+20 -243
View File
@@ -25,11 +25,6 @@ from backend.config import get_settings
logger = logging.getLogger(__name__)
# Retry configuration for handling transient failures (e.g., worker restarts)
MAX_RETRIES = 3
INITIAL_BACKOFF_SECONDS = 2.0
MAX_BACKOFF_SECONDS = 10.0
class EncodingService:
"""Service for dispatching encoding jobs to GCE worker."""
@@ -68,86 +63,6 @@ class EncodingService:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
async def _request_with_retry(
self,
method: str,
url: str,
headers: Dict[str, str],
json_payload: Optional[Dict[str, Any]] = None,
timeout: float = 30.0,
job_id: str = "unknown",
) -> Dict[str, Any]:
"""
Make an HTTP request with retry logic for transient failures.
This handles connection errors that occur when the GCE worker is
restarting (e.g., during deployments) by retrying with exponential backoff.
Args:
method: HTTP method (GET, POST)
url: Request URL
headers: Request headers
json_payload: JSON body for POST requests
timeout: Request timeout in seconds
job_id: Job ID for logging
Returns:
Dict with keys:
- status (int): HTTP status code
- json (Any): Parsed JSON response body (if status 200, else None)
- text (str): Raw response text (if status != 200, else None)
Raises:
aiohttp.ClientConnectorError: If all retries fail due to connection errors
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
asyncio.TimeoutError: If all retries fail due to timeout
"""
last_exception = None
backoff = INITIAL_BACKOFF_SECONDS
for attempt in range(MAX_RETRIES + 1):
try:
async with aiohttp.ClientSession() as session:
if method.upper() == "POST":
async with session.post(
url, json=json_payload, headers=headers, timeout=timeout
) as resp:
# Return a copy of the response data since we exit the context
return {
"status": resp.status,
"json": await resp.json() if resp.status == 200 else None,
"text": await resp.text() if resp.status != 200 else None,
}
else: # GET
async with session.get(
url, headers=headers, timeout=timeout
) as resp:
return {
"status": resp.status,
"json": await resp.json() if resp.status == 200 else None,
"text": await resp.text() if resp.status != 200 else None,
}
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
last_exception = e
if attempt < MAX_RETRIES:
logger.warning(
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
f"Retrying in {backoff:.1f}s..."
)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
else:
logger.error(
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
)
raise last_exception
async def submit_encoding_job(
self,
job_id: str,
@@ -186,23 +101,17 @@ class EncodingService:
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
raise RuntimeError(f"Encoding job {job_id} already exists")
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
return resp["json"]
return await resp.json()
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
"""
@@ -222,22 +131,17 @@ class EncodingService:
url = f"{self._url}/status/{job_id}"
headers = {"X-API-Key": self._api_key}
resp = await self._request_with_retry(
method="GET",
url=url,
headers=headers,
timeout=30.0,
job_id=job_id,
)
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 404:
raise RuntimeError(f"Encoding job {job_id} not found")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 404:
raise RuntimeError(f"Encoding job {job_id} not found")
if resp["status"] != 200:
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
return resp["json"]
return await resp.json()
async def wait_for_completion(
self,
@@ -338,133 +242,6 @@ class EncodingService:
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._url}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
if font_gcs_path:
payload["font_gcs_path"] = font_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
return resp["json"]
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
submit_result = await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
# If cached, return immediately - video already exists in GCS
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
return {"status": "complete", "output_path": submit_result.get("output_path")}
# If in_progress, another request is encoding it - just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def health_check(self) -> Dict[str, Any]:
"""
Check the health of the encoding worker.
-27
View File
@@ -116,7 +116,6 @@ class FirestoreService:
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100
) -> List[Job]:
"""
@@ -129,7 +128,6 @@ class FirestoreService:
created_after: Filter jobs created after this datetime
created_before: Filter jobs created before this datetime
user_email: Filter by user_email (owner of the job)
tenant_id: Filter by tenant_id (white-label portal scoping)
limit: Maximum number of jobs to return
Returns:
@@ -152,10 +150,6 @@ class FirestoreService:
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Filter by tenant_id (white-label portal scoping)
if tenant_id:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
# Date range filters
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
@@ -516,24 +510,3 @@ class FirestoreService:
logger.error(f"Error listing tokens: {e}")
return []
# Singleton client instance
_firestore_client: Optional[firestore.Client] = None
def get_firestore_client() -> firestore.Client:
"""
Get a shared Firestore client instance.
This returns a raw Firestore client (not the FirestoreService) for use
in services that need direct Firestore access without the job-specific
abstractions provided by FirestoreService.
Returns:
Firestore client instance
"""
global _firestore_client
if _firestore_client is None:
_firestore_client = firestore.Client(project=settings.google_cloud_project)
return _firestore_client
-72
View File
@@ -1,72 +0,0 @@
# GCE Encoding Worker
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
## Overview
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
## Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/encode` | POST | Submit a full encoding job |
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
| `/status/{job_id}` | GET | Get job status and progress |
| `/health` | GET | Health check |
## Authentication
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
## Encoding Process
### Full Encoding (`/encode`)
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
- Lossless 4K MP4
- Lossy 4K MP4
- Lossless MKV
- 720p MP4
3. Uploads results to GCS
4. Reports progress via status endpoint
### Preview Encoding (`/encode-preview`)
1. Downloads ASS subtitle and audio files from GCS
2. Runs FFmpeg to create a quick 480x270 preview video
3. Uploads result to GCS
4. Used for real-time preview in the web UI
## Dependencies
- **FFmpeg**: Static build from John Van Sickle
- **Python 3.13**: Built from source (required for karaoke-gen)
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
## Deployment
This service is deployed via the VM's startup script. The script:
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
2. Builds Python 3.13 from source
3. Creates venv and installs dependencies
4. Writes `main.py` to `/opt/encoding-worker/`
5. Creates systemd service
6. Starts the service on port 8080
## Local Development
This code is extracted from the infrastructure for maintainability. To test locally:
```bash
cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../ # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py
```
## Architecture Notes
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
- **GCS integration**: Direct download/upload via google-cloud-storage client
-22
View File
@@ -1,22 +0,0 @@
"""
GCE Encoding Worker Service.
This package contains the FastAPI application that runs on the GCE encoding worker VM.
It provides HTTP endpoints for video encoding jobs.
The service:
- Downloads input files from GCS
- Runs FFmpeg encoding via LocalEncodingService
- Uploads results back to GCS
- Provides job status tracking
Endpoints:
- POST /encode - Submit a full encoding job
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
- GET /status/{job_id} - Get job status
- GET /health - Health check
"""
from .main import app
__all__ = ["app"]
-603
View File
@@ -1,603 +0,0 @@
import asyncio
import glob as glob_module
import json
import logging
import os
import subprocess
import sys
import tempfile
import uuid
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from typing import Optional
from fastapi import FastAPI, BackgroundTasks, HTTPException, Header, Depends
from google.cloud import storage
from pydantic import BaseModel
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = FastAPI(title="Encoding Worker", version="1.0.0")
# API key authentication
API_KEY = os.environ.get("ENCODING_API_KEY", "")
async def verify_api_key(x_api_key: str = Header(None)):
# Verify API key for authentication
if not API_KEY:
logger.warning("No API key configured - authentication disabled")
return True
if x_api_key != API_KEY:
raise HTTPException(status_code=401, detail="Invalid API key")
return True
# Job tracking
jobs: dict[str, dict] = {}
executor = ThreadPoolExecutor(max_workers=4) # 4 parallel encoding jobs
# GCS client
storage_client = storage.Client()
class EncodeRequest(BaseModel):
job_id: str
input_gcs_path: str # gs://bucket/path/to/inputs/
output_gcs_path: str # gs://bucket/path/to/outputs/
encoding_config: dict # Video formats to generate
class EncodePreviewRequest(BaseModel):
job_id: str
ass_gcs_path: str # gs://bucket/path/to/subtitles.ass
audio_gcs_path: str # gs://bucket/path/to/audio.flac
output_gcs_path: str # gs://bucket/path/to/output.mp4
background_color: str = "black"
background_image_gcs_path: Optional[str] = None
font_gcs_path: Optional[str] = None # gs://bucket/path/to/custom-font.ttf
class JobStatus(BaseModel):
job_id: str
status: str # pending, running, complete, failed
progress: int # 0-100
error: Optional[str] = None
output_files: Optional[list[str]] = None
def download_from_gcs(gcs_uri: str, local_path: Path):
# Download a file or folder from GCS
# Parse gs://bucket/path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
prefix = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blobs = list(bucket.list_blobs(prefix=prefix))
for blob in blobs:
# Get relative path from prefix
rel_path = blob.name[len(prefix):].lstrip("/")
if not rel_path:
continue
dest = local_path / rel_path
dest.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(dest))
logger.info(f"Downloaded: {blob.name} -> {dest}")
def upload_to_gcs(local_path: Path, gcs_uri: str):
# Upload a file or folder to GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
if local_path.is_file():
blob_name = f"{prefix}/{local_path.name}" if prefix else local_path.name
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> gs://{bucket_name}/{blob_name}")
else:
for file in local_path.rglob("*"):
if file.is_file():
rel_path = file.relative_to(local_path)
blob_name = f"{prefix}/{rel_path}" if prefix else str(rel_path)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(file))
logger.info(f"Uploaded: {file} -> gs://{bucket_name}/{blob_name}")
def download_single_file_from_gcs(gcs_uri: str, local_path: Path):
# Download a single file from GCS
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
local_path.parent.mkdir(parents=True, exist_ok=True)
blob.download_to_filename(str(local_path))
logger.info(f"Downloaded: {gcs_uri} -> {local_path}")
def upload_single_file_to_gcs(local_path: Path, gcs_uri: str):
# Upload a single file to a specific GCS path
parts = gcs_uri.replace("gs://", "").split("/", 1)
bucket_name = parts[0]
blob_name = parts[1] if len(parts) > 1 else ""
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
blob.upload_from_filename(str(local_path))
logger.info(f"Uploaded: {local_path} -> {gcs_uri}")
def run_preview_encoding(job_id: str, work_dir: Path, request: "EncodePreviewRequest"):
# Run FFmpeg encoding for preview video (480x270, fast settings)
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Download input files
ass_path = work_dir / "subtitles.ass"
audio_path = work_dir / "audio.flac"
download_single_file_from_gcs(request.ass_gcs_path, ass_path)
jobs[job_id]["progress"] = 20
download_single_file_from_gcs(request.audio_gcs_path, audio_path)
jobs[job_id]["progress"] = 30
# Download background image if provided
bg_image_path = None
if request.background_image_gcs_path:
bg_image_path = work_dir / "background.png"
download_single_file_from_gcs(request.background_image_gcs_path, bg_image_path)
# Download custom font if provided and register with fontconfig
if request.font_gcs_path:
# Use standard fontconfig location that's already in the search path
fonts_dir = Path("/usr/local/share/fonts/custom")
fonts_dir.mkdir(parents=True, exist_ok=True)
font_filename = request.font_gcs_path.split("/")[-1]
font_path = fonts_dir / font_filename
download_single_file_from_gcs(request.font_gcs_path, font_path)
logger.info(f"Downloaded custom font: {font_path}")
# Update fontconfig cache so libass can find the font
subprocess.run(["fc-cache", "-fv"], capture_output=True)
logger.info(f"Updated fontconfig cache with custom font: {font_filename}")
# Build FFmpeg command
output_path = work_dir / "preview.mp4"
# Escape special characters in path for FFmpeg filter syntax
# FFmpeg filter parsing requires escaping: \ : , [ ] ;
def escape_ffmpeg_filter_path(path: str) -> str:
# Note: Extra escaping needed since this is inside a triple-quoted string in Pulumi
return path.replace("\\", "\\\\").replace(":", "\\:").replace(",", "\\,").replace("[", "\\[").replace("]", "\\]").replace(";", "\\;")
escaped_ass_path = escape_ffmpeg_filter_path(str(ass_path))
# Base command with frame rate
cmd = ["ffmpeg", "-y", "-r", "24"]
# Video input: background image or solid color
if bg_image_path and bg_image_path.exists():
cmd.extend(["-loop", "1", "-i", str(bg_image_path)])
# Scale and pad background to 480x270
vf = f"scale=480:270:force_original_aspect_ratio=decrease,pad=480:270:(ow-iw)/2:(oh-ih)/2,ass={escaped_ass_path}"
else:
# Solid color background
color = request.background_color or "black"
cmd.extend(["-f", "lavfi", "-i", f"color=c={color}:s=480x270:r=24"])
vf = f"ass={escaped_ass_path}"
# Audio input
cmd.extend(["-i", str(audio_path)])
# Video filter and encoding settings
cmd.extend([
"-vf", vf,
"-c:a", "aac", "-b:a", "96k",
"-c:v", "libx264",
"-preset", "superfast",
"-crf", "28",
"-pix_fmt", "yuv420p",
"-movflags", "+faststart",
"-threads", "8",
"-shortest",
str(output_path)
])
jobs[job_id]["progress"] = 40
logger.info(f"Running preview encoding: {' '.join(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
logger.error(f"FFmpeg failed: {result.stderr}")
raise RuntimeError(f"FFmpeg preview encoding failed: {result.stderr[-500:]}")
jobs[job_id]["progress"] = 80
logger.info(f"Preview encoded: {output_path}")
# Upload output to GCS
upload_single_file_to_gcs(output_path, request.output_gcs_path)
jobs[job_id]["progress"] = 95
jobs[job_id]["output_files"] = [request.output_gcs_path]
return output_path
except Exception as e:
logger.error(f"Preview encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
def ensure_latest_wheel():
'''Download and install latest karaoke-gen wheel from GCS.
Called at the start of each job to enable hot code updates without restart.
In-progress jobs continue with their version, new jobs get latest code.
'''
try:
logger.info("Checking for latest karaoke-gen wheel in GCS...")
# Download latest wheel
result = subprocess.run(
["gsutil", "cp", "gs://karaoke-gen-storage-nomadkaraoke/wheels/karaoke_gen-*.whl", "/tmp/"],
capture_output=True, text=True, timeout=60
)
# Find the downloaded wheel (get the latest by version sorting)
wheels = glob_module.glob("/tmp/karaoke_gen-*.whl")
if not wheels:
logger.warning("No wheel found in GCS, using fallback encoding logic")
return False
# Sort to get latest version
wheel_path = sorted(wheels)[-1]
logger.info(f"Installing wheel: {wheel_path}")
# Install (or upgrade) the wheel
# Use 5-minute timeout - first install at job start may need to resolve dependencies
# Subsequent installs are faster since dependencies are cached
install_result = subprocess.run(
[sys.executable, "-m", "pip", "install", "--upgrade", "--quiet", wheel_path],
capture_output=True, text=True, timeout=300
)
if install_result.returncode != 0:
logger.warning(f"Wheel installation failed: {install_result.stderr}")
return False
logger.info(f"Successfully installed wheel: {wheel_path}")
return True
except subprocess.TimeoutExpired:
logger.warning("Wheel download/install timed out, using fallback")
return False
except Exception as e:
logger.warning(f"Failed to ensure latest wheel: {e}")
return False
def find_file(work_dir: Path, *patterns):
'''Find a file matching any of the given glob patterns.'''
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
def run_encoding(job_id: str, work_dir: Path, config: dict):
'''Run encoding using LocalEncodingService (single source of truth).
Uses LocalEncodingService from the installed karaoke-gen wheel to ensure
output files match local CLI exactly:
- Proper names like "Artist - Title (Final Karaoke Lossless 4k).mp4"
- Concatenated title + karaoke + end screens
- All formats: lossless 4K MP4, lossy 4K MP4, lossless MKV, 720p MP4
Requires the karaoke-gen wheel to be installed (done by ensure_latest_wheel).
'''
jobs[job_id]["status"] = "running"
jobs[job_id]["progress"] = 10
try:
# Import LocalEncodingService from installed wheel (required, no fallback)
from backend.services.local_encoding_service import LocalEncodingService, EncodingConfig
logger.info("Using LocalEncodingService from installed wheel")
# Get artist/title from config for proper naming
artist = config.get("artist", "Unknown Artist")
title = config.get("title", "Unknown Title")
base_name = f"{artist} - {title}"
logger.info(f"Encoding for: {base_name}")
# Find input files in work_dir
# Title/end screens are in screens/ subdirectory
title_video = find_file(work_dir, "screens/title.mov", "*Title*.mov", "*title*.mov")
end_video = find_file(work_dir, "screens/end.mov", "*End*.mov", "*end*.mov")
# Karaoke video - search for With Vocals or main karaoke video
karaoke_video = find_file(
work_dir,
"*With Vocals*.mov", "*With Vocals*.mkv",
"*Vocals*.mov", "*Vocals*.mkv",
"*.mkv", "*.mov"
)
# Exclude title/end/output videos
if karaoke_video:
name_lower = karaoke_video.name.lower()
if "title" in name_lower or "end" in name_lower or "outputs" in str(karaoke_video):
# Search more specifically for karaoke video
karaoke_video = find_file(work_dir, "*Karaoke*.mkv", "*Karaoke*.mov", "*vocals*.mkv")
# Instrumental audio - respect user's selection from encoding config
instrumental_selection = config.get("instrumental_selection", "clean")
logger.info(f"Instrumental selection from config: {instrumental_selection}")
if instrumental_selection == "with_backing":
# User selected instrumental with backing vocals
instrumental = find_file(
work_dir,
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
"*with_backing*.flac", "*Backing*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
else:
# Default to clean instrumental
instrumental = find_file(
work_dir,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
logger.info(f"Found files:")
logger.info(f" Title video: {title_video}")
logger.info(f" Karaoke video: {karaoke_video}")
logger.info(f" End video: {end_video}")
logger.info(f" Instrumental ({instrumental_selection}): {instrumental}")
# Validate required files
if not title_video:
raise ValueError(f"No title video found in {work_dir}. Check screens/ subdirectory.")
if not karaoke_video:
raise ValueError(f"No karaoke video found in {work_dir}")
if not instrumental:
raise ValueError(f"No instrumental audio found in {work_dir}")
output_dir = work_dir / "outputs"
output_dir.mkdir(exist_ok=True)
jobs[job_id]["progress"] = 20
# Build encoding config with proper file names
encoding_config = EncodingConfig(
title_video=str(title_video),
karaoke_video=str(karaoke_video),
instrumental_audio=str(instrumental),
end_video=str(end_video) if end_video else None,
output_karaoke_mp4=str(output_dir / f"{base_name} (Karaoke).mp4"),
output_with_vocals_mp4=str(output_dir / f"{base_name} (With Vocals).mp4"),
output_lossless_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mp4"),
output_lossy_4k_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=str(output_dir / f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=str(output_dir / f"{base_name} (Final Karaoke Lossy 720p).mp4"),
)
# Create service and run encoding
service = LocalEncodingService(logger=logger)
jobs[job_id]["progress"] = 30
logger.info("Starting LocalEncodingService.encode_all_formats()")
result = service.encode_all_formats(encoding_config)
if not result.success:
raise RuntimeError(f"Encoding failed: {result.error}")
jobs[job_id]["progress"] = 90
# Collect output files
output_files = [str(f) for f in output_dir.glob("*") if f.is_file()]
jobs[job_id]["output_files"] = output_files
logger.info(f"Encoding complete. Output files: {output_files}")
return output_dir
except ImportError as e:
# No fallback - wheel must be installed
error_msg = (
f"LocalEncodingService not available: {e}. "
"The karaoke-gen wheel must be installed. "
"Check that ensure_latest_wheel() succeeded and wheel exists in GCS."
)
logger.error(error_msg)
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = error_msg
raise RuntimeError(error_msg) from e
except Exception as e:
logger.error(f"Encoding failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
raise
async def process_job(job_id: str, request: EncodeRequest):
# Process an encoding job asynchronously
try:
# Download and install latest wheel at job start (allows hot updates without restart)
# This means in-progress jobs continue with their version, new jobs get latest code
ensure_latest_wheel()
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Download input files
jobs[job_id]["progress"] = 5
logger.info(f"Downloading from {request.input_gcs_path}")
download_from_gcs(request.input_gcs_path, work_dir)
# Run encoding in thread pool (CPU-bound)
loop = asyncio.get_event_loop()
output_dir = await loop.run_in_executor(
executor,
run_encoding,
job_id,
work_dir,
request.encoding_config
)
# Upload outputs
jobs[job_id]["progress"] = 95
logger.info(f"Uploading to {request.output_gcs_path}")
upload_to_gcs(output_dir, request.output_gcs_path)
# Convert local paths to blob paths (backend expects blob paths, not full gs:// URIs)
# output_gcs_path is like "gs://bucket/jobs/id/encoded/"
# We need paths like "jobs/id/encoded/Artist - Title (Final Karaoke Lossless 4k).mp4"
gcs_path = request.output_gcs_path.replace("gs://", "")
parts = gcs_path.split("/", 1)
prefix = parts[1].rstrip("/") if len(parts) > 1 else ""
local_output_files = jobs[job_id].get("output_files", [])
blob_paths = []
for local_path in local_output_files:
filename = Path(local_path).name
blob_path = f"{prefix}/{filename}" if prefix else filename
blob_paths.append(blob_path)
jobs[job_id]["output_files"] = blob_paths
logger.info(f"Output files (blob paths): {blob_paths}")
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Job {job_id} complete")
except Exception as e:
logger.error(f"Job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
async def process_preview_job(job_id: str, request: EncodePreviewRequest):
# Process a preview encoding job asynchronously
try:
with tempfile.TemporaryDirectory() as temp_dir:
work_dir = Path(temp_dir) / "work"
work_dir.mkdir()
# Run preview encoding in thread pool (CPU-bound)
# Note: run_preview_encoding handles download/upload internally
loop = asyncio.get_event_loop()
await loop.run_in_executor(
executor,
run_preview_encoding,
job_id,
work_dir,
request
)
jobs[job_id]["status"] = "complete"
jobs[job_id]["progress"] = 100
logger.info(f"Preview job {job_id} complete")
except Exception as e:
logger.error(f"Preview job {job_id} failed: {e}")
jobs[job_id]["status"] = "failed"
jobs[job_id]["error"] = str(e)
@app.post("/encode-preview")
async def submit_preview_encode_job(request: EncodePreviewRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit a preview encoding job
job_id = request.job_id
# If job already exists, return cached result or current status
if job_id in jobs:
existing_job = jobs[job_id]
if existing_job["status"] == "complete":
# Return cached result - preview already encoded
return {"status": "cached", "job_id": job_id, "output_path": existing_job.get("output_path")}
elif existing_job["status"] == "failed":
# Previous attempt failed, allow retry by replacing the job
pass
else:
# Job is still in progress
return {"status": "in_progress", "job_id": job_id}
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_preview_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.post("/encode")
async def submit_encode_job(request: EncodeRequest, background_tasks: BackgroundTasks, _auth: bool = Depends(verify_api_key)):
# Submit an encoding job
job_id = request.job_id
if job_id in jobs:
raise HTTPException(status_code=409, detail=f"Job {job_id} already exists")
jobs[job_id] = {
"job_id": job_id,
"status": "pending",
"progress": 0,
"error": None,
"output_files": None,
}
background_tasks.add_task(process_job, job_id, request)
return {"status": "accepted", "job_id": job_id}
@app.get("/status/{job_id}")
async def get_job_status(job_id: str, _auth: bool = Depends(verify_api_key)) -> JobStatus:
# Get the status of an encoding job
if job_id not in jobs:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
return JobStatus(**jobs[job_id])
@app.get("/health")
async def health_check():
# Health check endpoint
active_jobs = sum(1 for j in jobs.values() if j["status"] == "running")
# Get karaoke-gen wheel version if installed
wheel_version = None
try:
from importlib.metadata import version as get_version
wheel_version = get_version("karaoke-gen")
except Exception:
pass
return {
"status": "ok",
"active_jobs": active_jobs,
"queue_length": sum(1 for j in jobs.values() if j["status"] == "pending"),
"ffmpeg_version": subprocess.run(["ffmpeg", "-version"], capture_output=True, text=True).stdout.split("\n")[0],
"wheel_version": wheel_version,
}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8080)
@@ -1,16 +0,0 @@
# GCE Encoding Worker dependencies
# These are installed in the VM's Python virtual environment
# Web framework
fastapi>=0.109.0
uvicorn>=0.27.0
# GCS client
google-cloud-storage>=2.14.0
# Async utilities (optional, for future improvements)
aiofiles>=23.2.0
aiohttp>=3.9.0
# Note: The karaoke-gen wheel is installed separately from GCS.
# It provides LocalEncodingService which does the actual encoding.
+1 -5
View File
@@ -15,7 +15,6 @@ import os
from typing import Any, Dict, Optional
from backend.config import get_settings
from karaoke_gen.utils import sanitize_filename
logger = logging.getLogger(__name__)
@@ -254,10 +253,7 @@ class GoogleDriveService:
Returns:
Dictionary mapping category to uploaded file ID
"""
# Sanitize base_name to handle Unicode characters (curly quotes, em dashes, etc.)
# that could cause issues with Google Drive API queries and file naming
safe_base_name = sanitize_filename(base_name) if base_name else base_name
filename_base = f"{brand_code} - {safe_base_name}"
filename_base = f"{brand_code} - {base_name}"
uploaded_files = {}
logger.info(
-113
View File
@@ -1,113 +0,0 @@
"""
Centralized job defaults service.
This module provides consistent handling of job creation defaults across all
endpoints (file_upload, audio_search, made-for-you webhook, etc.).
Centralizing these defaults prevents divergence between code paths and ensures
all jobs receive the same default configuration.
"""
from dataclasses import dataclass
from typing import Optional, Tuple
from backend.config import get_settings
@dataclass
class EffectiveDistributionSettings:
"""Distribution settings with defaults applied from environment variables."""
dropbox_path: Optional[str]
gdrive_folder_id: Optional[str]
discord_webhook_url: Optional[str]
brand_prefix: Optional[str]
enable_youtube_upload: bool
youtube_description: Optional[str]
def get_effective_distribution_settings(
dropbox_path: Optional[str] = None,
gdrive_folder_id: Optional[str] = None,
discord_webhook_url: Optional[str] = None,
brand_prefix: Optional[str] = None,
enable_youtube_upload: Optional[bool] = None,
youtube_description: Optional[str] = None,
) -> EffectiveDistributionSettings:
"""
Get distribution settings with defaults applied from environment variables.
This ensures consistent handling of defaults across all job creation endpoints.
Each parameter, if not provided (None), falls back to the corresponding
environment variable configured in settings.
Args:
dropbox_path: Explicit Dropbox path or None for default
gdrive_folder_id: Explicit Google Drive folder ID or None for default
discord_webhook_url: Explicit Discord webhook URL or None for default
brand_prefix: Explicit brand prefix or None for default
enable_youtube_upload: Explicit flag or None for default
youtube_description: Explicit description or None for default
Returns:
EffectiveDistributionSettings with defaults applied
"""
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=dropbox_path or settings.default_dropbox_path,
gdrive_folder_id=gdrive_folder_id or settings.default_gdrive_folder_id,
discord_webhook_url=discord_webhook_url or settings.default_discord_webhook_url,
brand_prefix=brand_prefix or settings.default_brand_prefix,
enable_youtube_upload=enable_youtube_upload if enable_youtube_upload is not None else settings.default_enable_youtube_upload,
youtube_description=youtube_description or settings.default_youtube_description,
)
def resolve_cdg_txt_defaults(
theme_id: Optional[str],
enable_cdg: Optional[bool] = None,
enable_txt: Optional[bool] = None,
) -> Tuple[bool, bool]:
"""
Resolve CDG/TXT settings based on theme and explicit settings.
The resolution logic is:
1. If explicit True/False is provided, use that value
2. Otherwise, if a theme is set, use the server defaults (settings.default_enable_cdg/txt)
3. If no theme is set, default to False (CDG/TXT require style configuration)
This ensures CDG/TXT are only enabled when:
- A theme is configured (provides necessary style params), AND
- The server defaults allow it (DEFAULT_ENABLE_CDG=true by default)
Args:
theme_id: Theme identifier (if any)
enable_cdg: Explicit CDG setting (None means use default)
enable_txt: Explicit TXT setting (None means use default)
Returns:
Tuple of (resolved_enable_cdg, resolved_enable_txt)
"""
settings = get_settings()
# Default based on whether theme is set AND server defaults
# Theme is required because CDG/TXT need style configuration
theme_is_set = theme_id is not None
default_cdg = theme_is_set and settings.default_enable_cdg
default_txt = theme_is_set and settings.default_enable_txt
# Explicit values override defaults, None uses computed default
resolved_cdg = enable_cdg if enable_cdg is not None else default_cdg
resolved_txt = enable_txt if enable_txt is not None else default_txt
return resolved_cdg, resolved_txt
# Singleton instance (optional, for convenience)
_service_instance = None
def get_job_defaults_service():
"""Get the job defaults service (module-level functions work fine, this is for consistency)."""
return {
'get_effective_distribution_settings': get_effective_distribution_settings,
'resolve_cdg_txt_defaults': resolve_cdg_txt_defaults,
}
+14 -290
View File
@@ -14,7 +14,6 @@ from datetime import datetime
from typing import Optional, Dict, Any, List
from backend.config import settings
from backend.exceptions import RateLimitExceededError
from backend.models.job import Job, JobStatus, JobCreate, STATE_TRANSITIONS
from backend.models.worker_log import WorkerLogEntry
from backend.services.firestore_service import FirestoreService
@@ -24,16 +23,6 @@ from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
def _mask_email(email: str) -> str:
"""Mask email for logging to protect PII. Shows first char + domain."""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
if len(local) <= 1:
return f"*@{domain}"
return f"{local[0]}***@{domain}"
class JobManager:
"""Manager for job lifecycle and state."""
@@ -42,52 +31,13 @@ class JobManager:
self.firestore = FirestoreService()
self.storage = StorageService()
def create_job(self, job_create: JobCreate, is_admin: bool = False) -> Job:
def create_job(self, job_create: JobCreate) -> Job:
"""
Create a new job with initial state PENDING.
Jobs start in PENDING state and transition to DOWNLOADING
when a worker picks them up.
Args:
job_create: Job creation parameters
is_admin: Whether the requesting user is an admin (bypasses rate limits)
Raises:
ValueError: If theme_id is not provided (all jobs require a theme)
RateLimitExceededError: If user has exceeded their daily job limit
"""
# Check rate limit FIRST (before any other validation)
# This prevents wasted work if user is rate limited
if job_create.user_email:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
allowed, remaining, message = rate_limit_service.check_user_job_limit(
user_email=job_create.user_email,
is_admin=is_admin
)
if not allowed:
from backend.services.rate_limit_service import _seconds_until_midnight_utc
# Get actual current count - remaining is clamped to 0 which loses info
current_count = rate_limit_service.get_user_job_count_today(job_create.user_email)
raise RateLimitExceededError(
message=message,
limit_type="jobs_per_day",
remaining_seconds=_seconds_until_midnight_utc(),
current_count=current_count,
limit_value=settings.rate_limit_jobs_per_day
)
# Enforce theme requirement - all jobs must have a theme
# This prevents unstyled videos from ever being generated
if not job_create.theme_id:
raise ValueError(
"theme_id is required for all jobs. "
"Use get_theme_service().get_default_theme_id() to get the default theme."
)
job_id = str(uuid.uuid4())[:8]
now = datetime.utcnow()
@@ -124,27 +74,11 @@ class JobManager:
keep_brand_code=job_create.keep_brand_code,
# Request metadata (for tracking and filtering)
request_metadata=job_create.request_metadata,
# Tenant scoping
tenant_id=job_create.tenant_id,
# Made-for-you order fields
made_for_you=job_create.made_for_you,
customer_email=job_create.customer_email,
customer_notes=job_create.customer_notes,
)
self.firestore.create_job(job)
logger.info(f"Created new job {job_id} with status PENDING")
# Record job creation for rate limiting (after successful persistence)
if job_create.user_email:
try:
from backend.services.rate_limit_service import get_rate_limit_service
rate_limit_service = get_rate_limit_service()
rate_limit_service.record_job_creation(job_create.user_email, job_id)
except Exception as e:
# Don't fail job creation if rate limit recording fails
logger.warning(f"Failed to record job creation for rate limiting: {e}")
return job
def get_job(self, job_id: str) -> Optional[Job]:
@@ -180,7 +114,6 @@ class JobManager:
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100
) -> List[Job]:
"""
@@ -193,7 +126,6 @@ class JobManager:
created_after: Filter jobs created after this datetime
created_before: Filter jobs created before this datetime
user_email: Filter by user_email (job owner)
tenant_id: Filter by tenant_id (white-label portal scoping)
limit: Maximum number of jobs to return
Returns:
@@ -206,7 +138,6 @@ class JobManager:
created_after=created_after,
created_before=created_before,
user_email=user_email,
tenant_id=tenant_id,
limit=limit
)
@@ -374,22 +305,22 @@ class JobManager:
updates['progress'] = progress
# Generate review token when entering AWAITING_REVIEW state
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status == JobStatus.AWAITING_REVIEW:
from backend.api.dependencies import generate_review_token
from backend.api.dependencies import generate_review_token, get_review_token_expiry
review_token = generate_review_token()
review_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['review_token'] = review_token
updates['review_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated review token for job {job_id} (no expiry)")
updates['review_token_expires_at'] = review_token_expires
logger.info(f"Generated review token for job {job_id}, expires in 48 hours")
# Generate instrumental token when entering AWAITING_INSTRUMENTAL_SELECTION state
# Tokens don't expire - they're job-scoped so low risk, and natural expiry happens when job completes
if new_status == JobStatus.AWAITING_INSTRUMENTAL_SELECTION:
from backend.api.dependencies import generate_review_token
from backend.api.dependencies import generate_review_token, get_review_token_expiry
instrumental_token = generate_review_token() # Reuse same token generator
instrumental_token_expires = get_review_token_expiry(hours=48) # 48 hour expiry
updates['instrumental_token'] = instrumental_token
updates['instrumental_token_expires_at'] = None # No expiry - token is job-scoped
logger.info(f"Generated instrumental token for job {job_id} (no expiry)")
updates['instrumental_token_expires_at'] = instrumental_token_expires
logger.info(f"Generated instrumental token for job {job_id}, expires in 48 hours")
# If we have state_data_updates, merge them with existing state_data
merged_state_data = None
@@ -428,217 +359,10 @@ class JobManager:
'instrumental_token': updates['instrumental_token'],
'instrumental_token_expires_at': updates['instrumental_token_expires_at']
})
logger.info(f"Job {job_id} transitioned to {new_status}")
# Trigger notifications asynchronously (fire-and-forget)
self._trigger_state_notifications(job_id, new_status)
return True
def _trigger_state_notifications(self, job_id: str, new_status: JobStatus) -> None:
"""
Trigger email and push notifications based on state transitions.
This is fire-and-forget - notification failures don't affect job processing.
Args:
job_id: Job ID
new_status: New job status
"""
import asyncio
try:
# Get the job to access user info
job = self.get_job(job_id)
if not job or not job.user_email:
logger.debug(f"No user email for job {job_id}, skipping notifications")
return
# Job completion notification
if new_status == JobStatus.COMPLETE:
self._schedule_completion_email(job)
self._send_push_notification(job, "complete")
# Idle reminder scheduling for blocking states
elif new_status in [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]:
self._schedule_idle_reminder(job, new_status)
# Send push notification for blocking states
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
self._send_push_notification(job, action_type)
except Exception as e:
# Never let notification failures affect job processing
logger.error(f"Error triggering notifications for job {job_id}: {e}")
def _schedule_completion_email(self, job: Job) -> None:
"""
Schedule sending a job completion email.
For made-for-you orders, also transfers ownership from admin to customer.
Uses asyncio to fire-and-forget the email sending.
"""
import asyncio
import threading
try:
from backend.services.job_notification_service import get_job_notification_service
notification_service = get_job_notification_service()
# Get youtube, dropbox URLs, and brand_code from state_data (may be None)
state_data = job.state_data or {}
youtube_url = state_data.get('youtube_url')
dropbox_url = state_data.get('dropbox_link')
brand_code = state_data.get('brand_code')
# For made-for-you orders, send to customer and transfer ownership
recipient_email = job.user_email
if job.made_for_you and job.customer_email:
recipient_email = job.customer_email
# Transfer ownership from admin to customer (non-blocking - email still goes out if this fails)
try:
self.update_job(job.job_id, {'user_email': job.customer_email})
logger.info(f"Transferred ownership of made-for-you job {job.job_id} to {_mask_email(job.customer_email)}")
except Exception as e:
logger.error(f"Failed to transfer ownership for job {job.job_id}: {e}")
# Create async task (fire-and-forget)
async def send_email():
await notification_service.send_job_completion_email(
job_id=job.job_id,
user_email=recipient_email,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
brand_code=brand_code,
)
# Try to get existing event loop, create new one if none exists
try:
loop = asyncio.get_running_loop()
# If we're in an async context, create a task
loop.create_task(send_email())
except RuntimeError:
# No event loop - we're likely in a sync context
# Use daemon thread to avoid blocking job completion
def run_in_thread():
asyncio.run(send_email())
thread = threading.Thread(target=run_in_thread, daemon=True)
thread.start()
logger.info(f"Scheduled completion email for job {job.job_id}")
except Exception as e:
logger.error(f"Failed to schedule completion email for job {job.job_id}: {e}")
def _schedule_idle_reminder(self, job: Job, new_status: JobStatus) -> None:
"""
Schedule an idle reminder for a blocking state.
Records the timestamp when the blocking state was entered and
schedules a Cloud Tasks task for 5 minutes later.
"""
import asyncio
import threading
try:
# Record when we entered blocking state (for idle detection)
blocking_entered_at = datetime.utcnow().isoformat()
action_type = "lyrics" if new_status == JobStatus.AWAITING_REVIEW else "instrumental"
# Update state_data with blocking state info (handle None state_data)
existing_state_data = job.state_data or {}
state_data_update = {
'blocking_state_entered_at': blocking_entered_at,
'blocking_action_type': action_type,
'reminder_sent': False, # Will be set to True after reminder is sent
}
self.firestore.update_job(job.job_id, {
'state_data': {**existing_state_data, **state_data_update}
})
# Schedule the idle reminder check via worker service (5 min delay)
from backend.services.worker_service import get_worker_service
async def schedule_reminder():
worker_service = get_worker_service()
await worker_service.schedule_idle_reminder(job.job_id)
# Try to get existing event loop, create new one if none exists
try:
loop = asyncio.get_running_loop()
loop.create_task(schedule_reminder())
except RuntimeError:
# No event loop - we're in a sync context
# Use daemon thread to avoid blocking job processing
def run_in_thread():
asyncio.run(schedule_reminder())
thread = threading.Thread(target=run_in_thread, daemon=True)
thread.start()
logger.info(f"Scheduled idle reminder for job {job.job_id} ({action_type})")
except Exception as e:
logger.error(f"Failed to schedule idle reminder for job {job.job_id}: {e}")
def _send_push_notification(self, job: Job, action_type: str) -> None:
"""
Send a push notification for job state changes.
Fire-and-forget - failures don't affect job processing.
Args:
job: Job object
action_type: Type of notification ("lyrics", "instrumental", or "complete")
"""
import asyncio
import threading
try:
from backend.services.push_notification_service import get_push_notification_service
push_service = get_push_notification_service()
# Skip if push notifications not enabled
if not push_service.is_enabled():
logger.debug("Push notifications not enabled, skipping")
return
# Build job dict for notification service
job_dict = {
"job_id": job.job_id,
"user_email": job.user_email,
"artist": job.artist,
"title": job.title,
}
async def send_notification():
if action_type == "complete":
await push_service.send_completion_notification(job_dict)
else:
await push_service.send_blocking_notification(job_dict, action_type)
# Try to get existing event loop, create new one if none exists
try:
loop = asyncio.get_running_loop()
loop.create_task(send_notification())
except RuntimeError:
# No event loop - we're in a sync context
def run_in_thread():
asyncio.run(send_notification())
thread = threading.Thread(target=run_in_thread, daemon=True)
thread.start()
logger.debug(f"Scheduled push notification for job {job.job_id} ({action_type})")
except Exception as e:
logger.error(f"Failed to send push notification for job {job.job_id}: {e}")
def update_state_data(self, job_id: str, key: str, value: Any) -> None:
"""
Update a specific key in the job's state_data field.
@@ -1,254 +0,0 @@
"""
Job notification service for sending emails on job state changes.
Handles:
- Job completion emails (when job enters COMPLETE status)
- Action reminder emails (when user is idle at blocking states)
This service orchestrates between the template service and email service,
building the complete notification flow for jobs.
"""
import logging
import os
from typing import Optional, Dict, Any
from backend.services.email_service import get_email_service
from backend.services.template_service import get_template_service
logger = logging.getLogger(__name__)
# Environment variable to enable/disable auto emails
ENABLE_AUTO_EMAILS = os.getenv("ENABLE_AUTO_EMAILS", "true").lower() == "true"
# Feedback form URL (configured per environment, empty by default to avoid placeholder in emails)
FEEDBACK_FORM_URL = os.getenv("FEEDBACK_FORM_URL", "")
def _mask_email(email: str) -> str:
"""Mask email for logging to protect PII. Shows first char + domain."""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
if len(local) <= 1:
return f"*@{domain}"
return f"{local[0]}***@{domain}"
class JobNotificationService:
"""
Service for sending job-related email notifications.
Coordinates between template service (for message rendering) and
email service (for sending).
"""
def __init__(self):
"""Initialize notification service."""
self.email_service = get_email_service()
self.template_service = get_template_service()
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
self.backend_url = os.getenv("BACKEND_URL", "https://api.nomadkaraoke.com")
def _build_review_url(self, job_id: str, audio_hash: Optional[str] = None, review_token: Optional[str] = None) -> str:
"""Build the lyrics review URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}/app/jobs#/{job_id}/review"
def _build_instrumental_url(self, job_id: str, instrumental_token: Optional[str] = None) -> str:
"""Build the instrumental selection URL for a job."""
# Use hash-based routing for static hosting compatibility
return f"{self.frontend_url}/app/jobs#/{job_id}/instrumental"
async def send_job_completion_email(
self,
job_id: str,
user_email: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
brand_code: Optional[str] = None,
) -> bool:
"""
Send job completion email to user.
Args:
job_id: Job ID
user_email: User's email address
user_name: User's display name (optional)
artist: Artist name
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
brand_code: Release ID (e.g., "NOMAD-1178")
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping completion email for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping completion email")
return False
try:
# Render the completion message using template service
message_content = self.template_service.render_job_completion(
name=user_name,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
artist=artist,
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
)
# Send the email with CC to admin
success = self.email_service.send_job_completion(
to_email=user_email,
message_content=message_content,
artist=artist,
title=title,
brand_code=brand_code,
cc_admin=True,
)
if success:
logger.info(f"Sent completion email for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send completion email for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending completion email for job {job_id}: {e}")
return False
async def send_action_reminder_email(
self,
job_id: str,
user_email: str,
action_type: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
audio_hash: Optional[str] = None,
review_token: Optional[str] = None,
instrumental_token: Optional[str] = None,
) -> bool:
"""
Send action-needed reminder email to user.
Args:
job_id: Job ID
user_email: User's email address
action_type: Type of action needed ("lyrics" or "instrumental")
user_name: User's display name
artist: Artist name
title: Song title
audio_hash: Audio hash for review URL
review_token: Review token for URL
instrumental_token: Instrumental token for URL
Returns:
True if email was sent successfully
"""
if not ENABLE_AUTO_EMAILS:
logger.info(f"Auto emails disabled, skipping reminder for job {job_id}")
return False
if not user_email:
logger.warning(f"No user email for job {job_id}, skipping reminder")
return False
try:
# Render the appropriate template
if action_type == "lyrics":
review_url = self._build_review_url(job_id, audio_hash, review_token)
message_content = self.template_service.render_action_needed_lyrics(
name=user_name,
artist=artist,
title=title,
review_url=review_url,
)
elif action_type == "instrumental":
instrumental_url = self._build_instrumental_url(job_id, instrumental_token)
message_content = self.template_service.render_action_needed_instrumental(
name=user_name,
artist=artist,
title=title,
instrumental_url=instrumental_url,
)
else:
logger.error(f"Unknown action type: {action_type}")
return False
# Send the reminder email (no CC)
success = self.email_service.send_action_reminder(
to_email=user_email,
message_content=message_content,
action_type=action_type,
artist=artist,
title=title,
)
if success:
logger.info(f"Sent {action_type} reminder for job {job_id} to {_mask_email(user_email)}")
else:
logger.error(f"Failed to send {action_type} reminder for job {job_id}")
return success
except Exception as e:
logger.exception(f"Error sending {action_type} reminder for job {job_id}: {e}")
return False
def get_completion_message(
self,
job_id: str,
user_name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
) -> str:
"""
Get the rendered completion message for a job (for admin copy functionality).
Args:
job_id: Job ID
user_name: User's display name
artist: Artist name
title: Song title
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
Returns:
Rendered message content as plain text
"""
return self.template_service.render_job_completion(
name=user_name,
youtube_url=youtube_url,
dropbox_url=dropbox_url,
artist=artist,
title=title,
job_id=job_id,
feedback_url=FEEDBACK_FORM_URL,
)
# Global instance
_job_notification_service: Optional[JobNotificationService] = None
def get_job_notification_service() -> JobNotificationService:
"""Get the global job notification service instance."""
global _job_notification_service
if _job_notification_service is None:
_job_notification_service = JobNotificationService()
return _job_notification_service
-98
View File
@@ -1,98 +0,0 @@
"""Langfuse callback handler preloader for container startup.
Initializes the Langfuse callback handler at container startup to avoid
slow initialization during request processing. The CallbackHandler()
constructor makes blocking network calls to the Langfuse API, which can
take 3+ minutes on Cloud Run cold starts.
See docs/archive/2026-01-08-performance-investigation.md for background.
"""
import logging
import os
import time
from typing import Optional, Any
logger = logging.getLogger(__name__)
# Singleton storage for preloaded handler
_preloaded_handler: Optional[Any] = None
_initialization_attempted: bool = False
def preload_langfuse_handler() -> Optional[Any]:
"""Preload Langfuse callback handler at startup.
Only initializes if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY
environment variables are set.
Returns:
The preloaded CallbackHandler, or None if not configured
"""
global _preloaded_handler, _initialization_attempted
if _initialization_attempted:
logger.debug("Langfuse initialization already attempted")
return _preloaded_handler
_initialization_attempted = True
# Check if Langfuse is configured
public_key = os.getenv("LANGFUSE_PUBLIC_KEY")
secret_key = os.getenv("LANGFUSE_SECRET_KEY")
if not (public_key and secret_key):
logger.info("Langfuse not configured (missing keys), skipping preload")
return None
logger.info("Preloading Langfuse callback handler...")
start_time = time.time()
try:
from langfuse.langchain import CallbackHandler
# Initialize the handler - this is the slow part that makes network calls
_preloaded_handler = CallbackHandler()
elapsed = time.time() - start_time
host = os.getenv("LANGFUSE_HOST", "cloud.langfuse.com")
logger.info(f"Langfuse handler preloaded in {elapsed:.2f}s (host: {host})")
return _preloaded_handler
except ImportError:
logger.warning("langfuse package not installed, skipping preload")
return None
except Exception as e:
elapsed = time.time() - start_time
logger.error(f"Failed to preload Langfuse handler after {elapsed:.2f}s: {e}")
# Don't raise - Langfuse is optional
return None
def get_preloaded_langfuse_handler() -> Optional[Any]:
"""Get the preloaded Langfuse callback handler if available.
Returns:
The preloaded CallbackHandler, or None if not preloaded/configured
"""
return _preloaded_handler
def is_langfuse_preloaded() -> bool:
"""Check if Langfuse handler has been preloaded."""
return _preloaded_handler is not None
def is_langfuse_configured() -> bool:
"""Check if Langfuse environment variables are configured."""
return bool(
os.getenv("LANGFUSE_PUBLIC_KEY") and os.getenv("LANGFUSE_SECRET_KEY")
)
def clear_preloaded_handler() -> None:
"""Clear preloaded handler. Useful for testing."""
global _preloaded_handler, _initialization_attempted
_preloaded_handler = None
_initialization_attempted = False
@@ -1,407 +0,0 @@
"""
Local Preview Encoding Service.
Provides local FFmpeg-based preview video encoding functionality, extracted from
VideoGenerator for use by the GCE worker. This ensures the same encoding logic
is used across local CLI, Cloud Run, and GCE worker environments.
This service handles:
- Generating preview videos with ASS subtitle overlay
- Hardware acceleration detection (NVENC) with fallback to software encoding
- Background image/color support
- Custom font support
- Optimized settings for fast preview generation
"""
import logging
import os
import subprocess
from dataclasses import dataclass
from pathlib import Path
from typing import Optional, List
logger = logging.getLogger(__name__)
@dataclass
class PreviewEncodingConfig:
"""Configuration for preview video encoding."""
ass_path: str # Path to ASS subtitles file
audio_path: str # Path to audio file
output_path: str # Path for output video file
# Optional background settings
background_image_path: Optional[str] = None # Path to background image
background_color: str = "black" # Fallback background color
# Optional font settings
font_path: Optional[str] = None # Path to custom font file
@dataclass
class PreviewEncodingResult:
"""Result of preview encoding operation."""
success: bool
output_path: Optional[str] = None
error: Optional[str] = None
class LocalPreviewEncodingService:
"""
Service for local FFmpeg-based preview video encoding.
This is the single source of truth for preview encoding logic, used by:
- Local CLI (via VideoGenerator which delegates here)
- Cloud Run (when GCE is unavailable)
- GCE worker (via installed wheel)
Supports hardware acceleration (NVENC) with automatic fallback
to software encoding (libx264) when unavailable.
"""
# Preview video settings - these are the canonical values
PREVIEW_WIDTH = 480
PREVIEW_HEIGHT = 270
PREVIEW_FPS = 24
PREVIEW_AUDIO_BITRATE = "96k"
def __init__(
self,
logger: Optional[logging.Logger] = None,
):
"""
Initialize the local preview encoding service.
Args:
logger: Optional logger instance
"""
self.logger = logger or logging.getLogger(__name__)
# Hardware acceleration settings (detected on first use)
self._nvenc_available: Optional[bool] = None
self._video_encoder: Optional[str] = None
self._hwaccel_flags: Optional[List[str]] = None
def _detect_nvenc_support(self) -> bool:
"""
Detect if NVENC hardware encoding is available.
Returns:
True if NVENC is available, False otherwise
"""
try:
self.logger.info("Detecting NVENC hardware acceleration...")
# Test h264_nvenc encoder directly
test_cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "testsrc=duration=1:size=320x240:rate=1",
"-c:v", "h264_nvenc", "-f", "null", "-"
]
result = subprocess.run(
test_cmd,
capture_output=True,
text=True,
timeout=30
)
if result.returncode == 0:
self.logger.info("NVENC hardware acceleration available")
return True
# Try alternative test with different source
alt_test_cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "color=red:size=320x240:duration=0.1",
"-c:v", "h264_nvenc", "-preset", "fast", "-f", "null", "-"
]
alt_result = subprocess.run(
alt_test_cmd,
capture_output=True,
text=True,
timeout=30
)
if alt_result.returncode == 0:
self.logger.info("NVENC hardware acceleration available")
return True
self.logger.info("NVENC not available, using software encoding")
return False
except subprocess.TimeoutExpired:
self.logger.debug("NVENC detection timed out")
return False
except Exception as e:
self.logger.debug(f"NVENC detection failed: {e}")
return False
def _configure_hardware_acceleration(self) -> None:
"""Configure hardware acceleration settings based on detected capabilities."""
if self._nvenc_available is None:
self._nvenc_available = self._detect_nvenc_support()
if self._nvenc_available:
self._video_encoder = "h264_nvenc"
self._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
self.logger.info("Using NVENC hardware acceleration for preview encoding")
else:
self._video_encoder = "libx264"
self._hwaccel_flags = []
self.logger.info("Using software encoding (libx264) for preview")
@property
def nvenc_available(self) -> bool:
"""Check if NVENC hardware acceleration is available."""
if self._nvenc_available is None:
self._configure_hardware_acceleration()
return self._nvenc_available
@property
def video_encoder(self) -> str:
"""Get the video encoder to use."""
if self._video_encoder is None:
self._configure_hardware_acceleration()
return self._video_encoder
@property
def hwaccel_flags(self) -> List[str]:
"""Get hardware acceleration flags."""
if self._hwaccel_flags is None:
self._configure_hardware_acceleration()
return self._hwaccel_flags
def _escape_ffmpeg_filter_path(self, path: str) -> str:
"""
Escape a path for FFmpeg filter expressions (for subprocess without shell).
When using subprocess with a command list (no shell), FFmpeg receives the
filter string directly. FFmpeg's filter parser requires escaping:
- Backslashes: double them (\\ -> \\\\)
- Single quotes/apostrophes: escape with three backslashes (' -> \\\\')
- Spaces: escape with backslash ( -> \\ )
- Special characters: :,[];
Example: "I'm With You" becomes "I\\\\'m\\ With\\ You"
"""
# First escape existing backslashes
escaped = path.replace("\\", "\\\\")
# Escape single quotes
escaped = escaped.replace("'", "\\\\\\'")
# Escape spaces
escaped = escaped.replace(" ", "\\ ")
# Escape FFmpeg filter special characters
escaped = escaped.replace(":", "\\:")
escaped = escaped.replace(",", "\\,")
escaped = escaped.replace("[", "\\[")
escaped = escaped.replace("]", "\\]")
escaped = escaped.replace(";", "\\;")
return escaped
def _build_ass_filter(self, ass_path: str, font_path: Optional[str] = None) -> str:
"""
Build ASS filter with optional font directory support.
Args:
ass_path: Path to ASS subtitles file
font_path: Optional path to custom font file
Returns:
FFmpeg ASS filter string
"""
escaped_ass_path = self._escape_ffmpeg_filter_path(ass_path)
ass_filter = f"ass={escaped_ass_path}"
if font_path and os.path.isfile(font_path):
font_dir = os.path.dirname(font_path)
escaped_font_dir = self._escape_ffmpeg_filter_path(font_dir)
ass_filter += f":fontsdir={escaped_font_dir}"
self.logger.debug(f"Using font directory: {font_dir}")
return ass_filter
def _build_preview_ffmpeg_command(self, config: PreviewEncodingConfig) -> List[str]:
"""
Build FFmpeg command for preview video generation.
This is the canonical preview encoding command, ensuring consistency
across all environments (local CLI, Cloud Run, GCE worker).
Args:
config: Preview encoding configuration
Returns:
FFmpeg command as a list of arguments
"""
width, height = self.PREVIEW_WIDTH, self.PREVIEW_HEIGHT
cmd = [
"ffmpeg",
"-hide_banner",
"-loglevel", "error",
"-r", str(self.PREVIEW_FPS),
]
# Add hardware acceleration flags if available
cmd.extend(self.hwaccel_flags)
# Input source (background image or solid color)
if config.background_image_path and os.path.isfile(config.background_image_path):
self.logger.debug(f"Using background image: {config.background_image_path}")
cmd.extend([
"-loop", "1",
"-i", config.background_image_path,
])
# Build video filter with scaling and ASS subtitles
video_filter = (
f"scale={width}:{height}:force_original_aspect_ratio=decrease,"
f"pad={width}:{height}:(ow-iw)/2:(oh-ih)/2,"
f"{self._build_ass_filter(config.ass_path, config.font_path)}"
)
else:
self.logger.debug(f"Using solid {config.background_color} background")
cmd.extend([
"-f", "lavfi",
"-i", f"color=c={config.background_color}:s={width}x{height}:r={self.PREVIEW_FPS}",
])
# Just ASS subtitles, no scaling needed
video_filter = self._build_ass_filter(config.ass_path, config.font_path)
cmd.extend([
"-i", config.audio_path,
"-vf", video_filter,
"-c:a", "aac",
"-b:a", self.PREVIEW_AUDIO_BITRATE,
"-c:v", self.video_encoder,
])
# Add encoder-specific settings optimized for speed
if self.nvenc_available:
cmd.extend([
"-preset", "p1", # Fastest NVENC preset
"-tune", "ll", # Low latency
"-rc", "cbr", # Constant bitrate for speed
"-b:v", "800k", # Lower bitrate for speed
"-profile:v", "baseline", # Most compatible profile
"-level", "3.1", # Lower level for speed
])
self.logger.debug("Using NVENC with maximum speed settings")
else:
cmd.extend([
"-profile:v", "baseline",
"-level", "3.0",
"-preset", "superfast",
"-tune", "fastdecode",
"-b:v", "600k",
"-maxrate", "800k",
"-bufsize", "1200k",
"-crf", "28",
])
self.logger.debug("Using software encoding with maximum speed settings")
cmd.extend([
"-pix_fmt", "yuv420p", # Required for browser compatibility
"-movflags", "+faststart+frag_keyframe+empty_moov+dash",
"-g", "48", # Keyframe every 48 frames (2 seconds at 24fps)
"-keyint_min", "48",
"-sc_threshold", "0", # Disable scene change detection for speed
"-threads", "0", # Use all available CPU threads
"-shortest",
"-y",
config.output_path,
])
return cmd
def encode_preview(self, config: PreviewEncodingConfig) -> PreviewEncodingResult:
"""
Encode a preview video.
Args:
config: Preview encoding configuration
Returns:
PreviewEncodingResult with success status and output path
"""
self.logger.info(f"Encoding preview video: {config.output_path}")
# Validate input files
if not os.path.isfile(config.ass_path):
return PreviewEncodingResult(
success=False,
error=f"ASS subtitles file not found: {config.ass_path}"
)
if not os.path.isfile(config.audio_path):
return PreviewEncodingResult(
success=False,
error=f"Audio file not found: {config.audio_path}"
)
# Ensure output directory exists
output_dir = os.path.dirname(config.output_path)
if output_dir:
os.makedirs(output_dir, exist_ok=True)
try:
# Build and execute FFmpeg command
cmd = self._build_preview_ffmpeg_command(config)
self.logger.debug(f"FFmpeg command: {' '.join(cmd)}")
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=300 # 5 minute timeout for preview encoding
)
if result.returncode != 0:
error_msg = result.stderr[-500:] if result.stderr else "Unknown error"
self.logger.error(f"FFmpeg failed: {error_msg}")
return PreviewEncodingResult(
success=False,
error=f"FFmpeg preview encoding failed: {error_msg}"
)
self.logger.info(f"Preview encoded successfully: {config.output_path}")
return PreviewEncodingResult(
success=True,
output_path=config.output_path
)
except subprocess.TimeoutExpired:
self.logger.error("Preview encoding timed out")
return PreviewEncodingResult(
success=False,
error="Preview encoding timed out after 5 minutes"
)
except Exception as e:
self.logger.error(f"Preview encoding failed: {e}")
return PreviewEncodingResult(
success=False,
error=str(e)
)
# Singleton instance and factory function
_local_preview_encoding_service: Optional[LocalPreviewEncodingService] = None
def get_local_preview_encoding_service(**kwargs) -> LocalPreviewEncodingService:
"""
Get a local preview encoding service instance.
Args:
**kwargs: Arguments passed to LocalPreviewEncodingService
Returns:
LocalPreviewEncodingService instance
"""
global _local_preview_encoding_service
if _local_preview_encoding_service is None:
_local_preview_encoding_service = LocalPreviewEncodingService(**kwargs)
return _local_preview_encoding_service
-122
View File
@@ -1,122 +0,0 @@
"""NLTK resource preloader for container startup.
Loads NLTK data at container startup to avoid slow downloads during request processing.
Cloud Run's ephemeral filesystem means NLTK data must be re-downloaded on each cold start,
which can take 30-100+ seconds for cmudict.
See docs/archive/2026-01-08-performance-investigation.md for background.
"""
import logging
import time
from typing import Optional, Dict, Any
logger = logging.getLogger(__name__)
# Singleton storage for preloaded resources
_preloaded_resources: Dict[str, Any] = {}
def preload_nltk_cmudict() -> None:
"""Preload NLTK's CMU Pronouncing Dictionary at startup.
The cmudict is used by SyllablesMatchHandler for syllable counting.
Without preloading, each SyllablesMatchHandler init downloads ~30MB,
which took 50-100+ seconds in Cloud Run.
"""
global _preloaded_resources
if "cmudict" in _preloaded_resources:
logger.info("NLTK cmudict already preloaded")
return
logger.info("Preloading NLTK cmudict...")
start_time = time.time()
try:
import nltk
# Ensure the data is downloaded
try:
from nltk.corpus import cmudict
# Try to access it - will raise LookupError if not downloaded
_ = cmudict.dict()
except LookupError:
logger.info("Downloading NLTK cmudict data...")
nltk.download("cmudict", quiet=True)
from nltk.corpus import cmudict
# Load into memory
cmu_dict = cmudict.dict()
_preloaded_resources["cmudict"] = cmu_dict
elapsed = time.time() - start_time
logger.info(f"NLTK cmudict preloaded in {elapsed:.2f}s ({len(cmu_dict)} entries)")
except Exception as e:
logger.error(f"Failed to preload NLTK cmudict: {e}")
raise
def get_preloaded_cmudict() -> Optional[Dict]:
"""Get the preloaded CMU dictionary if available.
Returns:
The preloaded cmudict dictionary, or None if not preloaded
"""
return _preloaded_resources.get("cmudict")
def is_cmudict_preloaded() -> bool:
"""Check if cmudict has been preloaded."""
return "cmudict" in _preloaded_resources
def preload_nltk_punkt() -> None:
"""Preload NLTK's punkt tokenizer (optional, used for sentence tokenization)."""
global _preloaded_resources
if "punkt" in _preloaded_resources:
logger.info("NLTK punkt already preloaded")
return
logger.info("Preloading NLTK punkt tokenizer...")
start_time = time.time()
try:
import nltk
try:
from nltk.tokenize import word_tokenize
# Test it works
_ = word_tokenize("test")
except LookupError:
logger.info("Downloading NLTK punkt data...")
nltk.download("punkt", quiet=True)
nltk.download("punkt_tab", quiet=True)
_preloaded_resources["punkt"] = True
elapsed = time.time() - start_time
logger.info(f"NLTK punkt preloaded in {elapsed:.2f}s")
except Exception as e:
logger.warning(f"Failed to preload NLTK punkt (non-critical): {e}")
def preload_all_nltk_resources() -> None:
"""Preload all NLTK resources used by the application."""
preload_nltk_cmudict()
# punkt is optional and less critical
try:
preload_nltk_punkt()
except Exception:
pass # Non-critical
def clear_preloaded_resources() -> None:
"""Clear all preloaded resources. Useful for testing."""
global _preloaded_resources
_preloaded_resources.clear()
+1 -1
View File
@@ -120,7 +120,7 @@ class PackagingService:
)
self.logger.info("Generating CDG and MP3 files")
from karaoke_gen.lyrics_transcriber.output.cdg import CDGGenerator
from lyrics_transcriber.output.cdg import CDGGenerator
output_dir = os.path.dirname(output_zip_path) or os.getcwd()
generator = CDGGenerator(output_dir=output_dir, logger=self.logger)
@@ -1,409 +0,0 @@
"""
Push Notification Service for Web Push Notifications.
Handles sending push notifications to users' browsers/devices using the Web Push protocol.
Uses pywebpush library for VAPID authentication and push message encryption.
"""
import logging
from datetime import datetime, timezone
from typing import Optional, List
from pywebpush import webpush, WebPushException
from backend.config import get_settings
from backend.models.user import User, PushSubscription
logger = logging.getLogger(__name__)
class SubscriptionGoneError(Exception):
"""Raised when a push subscription is no longer valid (410 Gone)."""
pass
class PushNotificationService:
"""
Service for sending Web Push notifications.
Requires VAPID keys to be configured in Secret Manager:
- vapid-public-key: Base64-encoded public key
- vapid-private-key: Base64-encoded private key
"""
def __init__(self, db=None):
"""
Initialize the push notification service.
Args:
db: Firestore client (optional, will use singleton if not provided)
"""
self.settings = get_settings()
self._vapid_private_key: Optional[str] = None
self._vapid_public_key: Optional[str] = None
self._db = db
@property
def db(self):
"""Get Firestore client (lazy initialization)."""
if self._db is None:
from google.cloud import firestore
self._db = firestore.Client()
return self._db
@property
def vapid_private_key(self) -> Optional[str]:
"""Get VAPID private key from Secret Manager (cached)."""
if self._vapid_private_key is None:
self._vapid_private_key = self.settings.get_secret("vapid-private-key")
return self._vapid_private_key
@property
def vapid_public_key(self) -> Optional[str]:
"""Get VAPID public key from Secret Manager (cached)."""
if self._vapid_public_key is None:
self._vapid_public_key = self.settings.get_secret("vapid-public-key")
return self._vapid_public_key
def is_enabled(self) -> bool:
"""Check if push notifications are enabled and properly configured."""
return (
self.settings.enable_push_notifications and
self.vapid_private_key is not None and
self.vapid_public_key is not None
)
def get_public_key(self) -> Optional[str]:
"""Get the VAPID public key for client-side subscription."""
if not self.settings.enable_push_notifications:
return None
return self.vapid_public_key
async def send_push(
self,
user_email: str,
title: str,
body: str,
url: str = "/app/",
tag: Optional[str] = None
) -> int:
"""
Send a push notification to all of a user's subscribed devices.
Args:
user_email: The user's email address
title: Notification title
body: Notification body text
url: URL to open when notification is clicked
tag: Optional tag for notification grouping (replaces notifications with same tag)
Returns:
Number of notifications successfully sent
"""
if not self.is_enabled():
logger.debug("Push notifications not enabled, skipping")
return 0
# Get user from Firestore
user_doc = self.db.collection("gen_users").document(user_email).get()
if not user_doc.exists:
logger.warning(f"User {user_email} not found for push notification")
return 0
user_data = user_doc.to_dict()
subscriptions = user_data.get("push_subscriptions", [])
if not subscriptions:
logger.debug(f"User {user_email} has no push subscriptions")
return 0
# Build notification payload
import json
payload = json.dumps({
"title": title,
"body": body,
"url": url,
"tag": tag or "default",
"icon": "/nomad-logo.png"
})
# Send to each subscription
success_count = 0
invalid_subscriptions = []
for sub in subscriptions:
try:
webpush(
subscription_info={
"endpoint": sub["endpoint"],
"keys": sub["keys"]
},
data=payload,
vapid_private_key=self.vapid_private_key,
vapid_claims={
"sub": self.settings.vapid_subject
}
)
success_count += 1
logger.debug(f"Push sent to {sub.get('device_name', 'unknown device')}")
except WebPushException as e:
logger.warning(f"Push failed: {e}")
# Check if subscription is gone (410) or unauthorized (401/403)
if e.response and e.response.status_code in (404, 410):
logger.info(f"Subscription gone, will remove: {sub['endpoint'][:50]}...")
invalid_subscriptions.append(sub["endpoint"])
elif e.response and e.response.status_code in (401, 403):
logger.warning(f"Push auth failed for {sub['endpoint'][:50]}...")
except Exception as e:
logger.error(f"Unexpected error sending push: {e}")
# Remove invalid subscriptions
if invalid_subscriptions:
await self._remove_invalid_subscriptions(user_email, invalid_subscriptions)
logger.info(f"Push notifications sent to {success_count}/{len(subscriptions)} devices for {user_email}")
return success_count
async def _remove_invalid_subscriptions(self, user_email: str, endpoints: List[str]) -> None:
"""Remove invalid subscriptions from user's list."""
try:
user_ref = self.db.collection("gen_users").document(user_email)
user_doc = user_ref.get()
if not user_doc.exists:
return
user_data = user_doc.to_dict()
current_subs = user_data.get("push_subscriptions", [])
# Filter out invalid subscriptions
valid_subs = [s for s in current_subs if s["endpoint"] not in endpoints]
if len(valid_subs) < len(current_subs):
user_ref.update({"push_subscriptions": valid_subs})
logger.info(f"Removed {len(current_subs) - len(valid_subs)} invalid subscriptions for {user_email}")
except Exception as e:
logger.error(f"Failed to remove invalid subscriptions: {e}")
async def send_blocking_notification(
self,
job: dict,
action_type: str # "lyrics" or "instrumental"
) -> int:
"""
Send notification when a job enters a blocking state requiring user action.
Args:
job: Job dictionary with job_id, user_email, artist, title
action_type: Type of action needed ("lyrics" or "instrumental")
Returns:
Number of notifications sent
"""
user_email = job.get("user_email")
if not user_email:
return 0
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
if action_type == "lyrics":
notif_title = "Review Lyrics"
notif_body = f'"{title}" by {artist} needs lyrics review'
url = f"/review/{job_id}"
tag = f"lyrics-{job_id}"
else: # instrumental
notif_title = "Select Instrumental"
notif_body = f'"{title}" by {artist} needs instrumental selection'
url = f"/instrumental/{job_id}"
tag = f"instrumental-{job_id}"
return await self.send_push(
user_email=user_email,
title=notif_title,
body=notif_body,
url=url,
tag=tag
)
async def send_completion_notification(self, job: dict) -> int:
"""
Send notification when a job completes successfully.
Args:
job: Job dictionary with job_id, user_email, artist, title
Returns:
Number of notifications sent
"""
user_email = job.get("user_email")
if not user_email:
return 0
job_id = job.get("job_id", "unknown")
artist = job.get("artist", "Unknown Artist")
title = job.get("title", "Unknown Title")
return await self.send_push(
user_email=user_email,
title="Video Ready!",
body=f'Your karaoke video for "{title}" by {artist} is ready to download',
url=f"/app/?job={job_id}",
tag=f"complete-{job_id}"
)
async def add_subscription(
self,
user_email: str,
endpoint: str,
keys: dict,
device_name: Optional[str] = None
) -> bool:
"""
Add a push subscription for a user.
Enforces max subscriptions per user - oldest removed if limit exceeded.
Args:
user_email: User's email address
endpoint: Push service endpoint URL
keys: Encryption keys (p256dh, auth)
device_name: Optional device identifier
Returns:
True if subscription was added successfully
"""
try:
user_ref = self.db.collection("gen_users").document(user_email)
user_doc = user_ref.get()
if not user_doc.exists:
logger.warning(f"User {user_email} not found")
return False
user_data = user_doc.to_dict()
subscriptions = user_data.get("push_subscriptions", [])
# Check if this endpoint already exists (update it)
existing_idx = None
for i, sub in enumerate(subscriptions):
if sub["endpoint"] == endpoint:
existing_idx = i
break
new_sub = {
"endpoint": endpoint,
"keys": keys,
"device_name": device_name,
"created_at": datetime.now(timezone.utc).isoformat(),
"last_used_at": None
}
if existing_idx is not None:
# Update existing subscription
subscriptions[existing_idx] = new_sub
logger.info(f"Updated existing push subscription for {user_email}")
else:
# Add new subscription
subscriptions.append(new_sub)
logger.info(f"Added new push subscription for {user_email}")
# Enforce max subscriptions (remove oldest)
max_subs = self.settings.max_push_subscriptions_per_user
if len(subscriptions) > max_subs:
# Sort by created_at and keep only the newest
subscriptions.sort(key=lambda s: s.get("created_at", ""), reverse=True)
removed = subscriptions[max_subs:]
subscriptions = subscriptions[:max_subs]
logger.info(f"Removed {len(removed)} old subscriptions for {user_email} (max {max_subs})")
user_ref.update({"push_subscriptions": subscriptions})
return True
except Exception as e:
logger.error(f"Failed to add subscription for {user_email}: {e}")
return False
async def remove_subscription(self, user_email: str, endpoint: str) -> bool:
"""
Remove a push subscription for a user.
Args:
user_email: User's email address
endpoint: Push service endpoint URL to remove
Returns:
True if subscription was removed
"""
try:
user_ref = self.db.collection("gen_users").document(user_email)
user_doc = user_ref.get()
if not user_doc.exists:
return False
user_data = user_doc.to_dict()
subscriptions = user_data.get("push_subscriptions", [])
# Filter out the subscription
new_subs = [s for s in subscriptions if s["endpoint"] != endpoint]
if len(new_subs) < len(subscriptions):
user_ref.update({"push_subscriptions": new_subs})
logger.info(f"Removed push subscription for {user_email}")
return True
return False
except Exception as e:
logger.error(f"Failed to remove subscription for {user_email}: {e}")
return False
async def list_subscriptions(self, user_email: str) -> List[dict]:
"""
List all push subscriptions for a user.
Args:
user_email: User's email address
Returns:
List of subscription info (endpoint, device_name, created_at, last_used_at)
"""
try:
user_doc = self.db.collection("gen_users").document(user_email).get()
if not user_doc.exists:
return []
user_data = user_doc.to_dict()
subscriptions = user_data.get("push_subscriptions", [])
# Return safe subset of subscription data
return [
{
"endpoint": s["endpoint"],
"device_name": s.get("device_name"),
"created_at": s.get("created_at"),
"last_used_at": s.get("last_used_at")
}
for s in subscriptions
]
except Exception as e:
logger.error(f"Failed to list subscriptions for {user_email}: {e}")
return []
# Singleton instance
_push_service: Optional[PushNotificationService] = None
def get_push_notification_service(db=None) -> PushNotificationService:
"""Get the push notification service singleton."""
global _push_service
if _push_service is None:
_push_service = PushNotificationService(db)
return _push_service
-641
View File
@@ -1,641 +0,0 @@
"""
Rate limiting service for job creation, YouTube uploads, and beta enrollment.
Uses Firestore for distributed rate limit tracking with date-based document IDs
that automatically reset at UTC midnight.
"""
import hashlib
import logging
from datetime import datetime, timezone
from typing import Tuple, Optional, List, Dict, Any
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Firestore collection names
RATE_LIMITS_COLLECTION = "rate_limits"
BLOCKLISTS_COLLECTION = "blocklists"
OVERRIDES_COLLECTION = "overrides"
def _get_today_date_str() -> str:
"""Get today's date in UTC as YYYY-MM-DD string."""
return datetime.now(timezone.utc).strftime("%Y-%m-%d")
def _seconds_until_midnight_utc() -> int:
"""Calculate seconds until UTC midnight (when rate limits reset)."""
now = datetime.now(timezone.utc)
midnight = now.replace(hour=0, minute=0, second=0, microsecond=0)
# Add 1 day to get next midnight
from datetime import timedelta
next_midnight = midnight + timedelta(days=1)
return int((next_midnight - now).total_seconds())
def _hash_ip(ip_address: str) -> str:
"""Hash an IP address for storage (privacy-preserving)."""
return hashlib.sha256(ip_address.encode()).hexdigest()[:16]
class RateLimitService:
"""
Service for managing rate limits across the platform.
Rate limits use Firestore for distributed tracking with automatic
reset at UTC midnight via date-based document IDs.
"""
def __init__(self, db: Optional[firestore.Client] = None):
"""Initialize rate limit service with Firestore client."""
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
self._blocklist_cache: Optional[Dict[str, Any]] = None
self._blocklist_cache_time: Optional[datetime] = None
self._cache_ttl_seconds = 300 # 5 minute cache
# -------------------------------------------------------------------------
# Job Rate Limiting
# -------------------------------------------------------------------------
def check_user_job_limit(self, user_email: str, is_admin: bool = False) -> Tuple[bool, int, str]:
"""
Check if a user can create a new job.
Args:
user_email: User's email address
is_admin: Whether the user is an admin (admins bypass limits)
Returns:
Tuple of (allowed, remaining, message)
- allowed: True if user can create job
- remaining: Number of jobs remaining today
- message: Human-readable status message
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
if is_admin:
return True, -1, "Admin users bypass rate limits"
limit = settings.rate_limit_jobs_per_day
if limit == 0:
return True, -1, "No job limit configured"
# Check for user override
override = self.get_user_override(user_email)
if override and override.get("bypass_job_limit"):
return True, -1, "User has rate limit bypass"
if override and override.get("custom_daily_job_limit"):
limit = override["custom_daily_job_limit"]
# Get current count
current_count = self.get_user_job_count_today(user_email)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Rate limit exceeded for {user_email}: {current_count}/{limit} jobs today"
)
return (
False,
0,
f"Daily job limit reached ({limit} jobs per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} jobs remaining today"
def get_user_job_count_today(self, user_email: str) -> int:
"""Get the number of jobs created by a user today."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_job_creation(self, user_email: str, job_id: str) -> None:
"""
Record a job creation for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"user_jobs_{user_email}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
job_ids = data.get("job_ids", [])
job_ids.append(job_id)
else:
count = 1
job_ids = [job_id]
transaction.set(doc_ref, {
"user_email": user_email,
"date": date_str,
"count": count,
"job_ids": job_ids,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded job {job_id} for user {user_email} rate limiting")
# -------------------------------------------------------------------------
# YouTube Upload Rate Limiting
# -------------------------------------------------------------------------
def check_youtube_upload_limit(self) -> Tuple[bool, int, str]:
"""
Check if the system can perform a YouTube upload.
YouTube uploads are limited system-wide due to API quota constraints.
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_youtube_uploads_per_day
if limit == 0:
return True, -1, "No YouTube upload limit configured"
current_count = self.get_youtube_uploads_today()
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"YouTube upload limit exceeded: {current_count}/{limit} uploads today"
)
return (
False,
0,
f"Daily YouTube upload limit reached ({limit} uploads per day). Resets in {seconds // 3600}h {(seconds % 3600) // 60}m."
)
return True, remaining, f"{remaining} YouTube uploads remaining today"
def get_youtube_uploads_today(self) -> int:
"""Get the number of YouTube uploads performed today (system-wide)."""
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
doc = doc_ref.get()
if not doc.exists:
return 0
return doc.to_dict().get("count", 0)
def record_youtube_upload(self, job_id: str, user_email: str) -> None:
"""
Record a YouTube upload for rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"youtube_uploads_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
uploads = data.get("uploads", [])
else:
count = 1
uploads = []
uploads.append({
"job_id": job_id,
"user_email": user_email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"date": date_str,
"count": count,
"uploads": uploads,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded YouTube upload for job {job_id}")
# -------------------------------------------------------------------------
# Beta Enrollment IP Rate Limiting
# -------------------------------------------------------------------------
def check_beta_ip_limit(self, ip_address: str) -> Tuple[bool, int, str]:
"""
Check if an IP address can enroll in the beta program.
Args:
ip_address: Client IP address
Returns:
Tuple of (allowed, remaining, message)
"""
if not settings.enable_rate_limiting:
return True, -1, "Rate limiting disabled"
limit = settings.rate_limit_beta_ip_per_day
if limit == 0:
return True, -1, "No beta IP limit configured"
# Check today's enrollment count for this IP
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
doc = doc_ref.get()
current_count = 0
if doc.exists:
current_count = doc.to_dict().get("count", 0)
remaining = max(0, limit - current_count)
if current_count >= limit:
seconds = _seconds_until_midnight_utc()
logger.warning(
f"Beta enrollment IP limit exceeded for {ip_hash}: {current_count}/{limit} enrollments today"
)
return (
False,
0,
f"Too many beta enrollments from this network today. Please try again tomorrow."
)
return True, remaining, f"{remaining} beta enrollments remaining from this IP today"
def record_beta_enrollment(self, ip_address: str, email: str) -> None:
"""
Record a beta enrollment for IP rate limiting.
Uses Firestore transactions for atomic increment.
"""
if not settings.enable_rate_limiting:
return
ip_hash = _hash_ip(ip_address)
date_str = _get_today_date_str()
doc_ref = self.db.collection(RATE_LIMITS_COLLECTION).document(
f"beta_ip_{ip_hash}_{date_str}"
)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
count = data.get("count", 0) + 1
enrollments = data.get("enrollments", [])
else:
count = 1
enrollments = []
enrollments.append({
"email": email,
"timestamp": datetime.now(timezone.utc),
})
transaction.set(doc_ref, {
"ip_hash": ip_hash,
"date": date_str,
"count": count,
"enrollments": enrollments,
"updated_at": datetime.now(timezone.utc),
})
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
logger.info(f"Recorded beta enrollment from IP {ip_hash} for {email}")
# -------------------------------------------------------------------------
# User Overrides (Whitelist)
# -------------------------------------------------------------------------
def get_user_override(self, user_email: str) -> Optional[Dict[str, Any]]:
"""
Get rate limit override settings for a user.
Returns:
Override settings dict or None if no override exists
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def set_user_override(
self,
user_email: str,
bypass_job_limit: bool = False,
custom_daily_job_limit: Optional[int] = None,
reason: str = "",
admin_email: str = ""
) -> None:
"""
Set rate limit override for a user.
Args:
user_email: User to override
bypass_job_limit: If True, user bypasses all job limits
custom_daily_job_limit: Custom limit (None = use default)
reason: Reason for override
admin_email: Admin who set the override
"""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc_ref.set({
"email": user_email.lower(),
"bypass_job_limit": bypass_job_limit,
"custom_daily_job_limit": custom_daily_job_limit,
"reason": reason,
"created_by": admin_email,
"created_at": datetime.now(timezone.utc),
})
logger.info(f"Set rate limit override for {user_email} by {admin_email}")
def remove_user_override(self, user_email: str, admin_email: str = "") -> bool:
"""Remove rate limit override for a user."""
doc_ref = self.db.collection(OVERRIDES_COLLECTION).document(user_email.lower())
doc = doc_ref.get()
if not doc.exists:
return False
doc_ref.delete()
logger.info(f"Removed rate limit override for {user_email} by {admin_email}")
return True
def list_user_overrides(self) -> List[Dict[str, Any]]:
"""List all user rate limit overrides."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return [doc.to_dict() for doc in docs]
def get_all_overrides(self) -> Dict[str, Dict[str, Any]]:
"""Get all user rate limit overrides as a dict keyed by email."""
docs = self.db.collection(OVERRIDES_COLLECTION).stream()
return {doc.id: doc.to_dict() for doc in docs}
# -------------------------------------------------------------------------
# Blocklist Management
# -------------------------------------------------------------------------
def _load_blocklist(self, force_refresh: bool = False) -> Dict[str, Any]:
"""
Load blocklist from Firestore with caching.
Returns:
Dict with keys: disposable_domains, blocked_emails, blocked_ips
"""
now = datetime.now(timezone.utc)
# Check cache
if not force_refresh and self._blocklist_cache and self._blocklist_cache_time:
cache_age = (now - self._blocklist_cache_time).total_seconds()
if cache_age < self._cache_ttl_seconds:
return self._blocklist_cache
# Load from Firestore
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc = doc_ref.get()
if not doc.exists:
# Initialize with empty blocklist
self._blocklist_cache = {
"disposable_domains": [],
"blocked_emails": [],
"blocked_ips": [],
}
else:
self._blocklist_cache = doc.to_dict()
self._blocklist_cache_time = now
return self._blocklist_cache
def is_disposable_domain(self, domain: str) -> bool:
"""Check if a domain is in the disposable domains blocklist."""
blocklist = self._load_blocklist()
return domain.lower() in [d.lower() for d in blocklist.get("disposable_domains", [])]
def is_blocked_email(self, email: str) -> bool:
"""Check if an email is explicitly blocked."""
blocklist = self._load_blocklist()
return email.lower() in [e.lower() for e in blocklist.get("blocked_emails", [])]
def is_blocked_ip(self, ip_address: str) -> bool:
"""Check if an IP address is blocked."""
blocklist = self._load_blocklist()
return ip_address in blocklist.get("blocked_ips", [])
def get_blocklist(self) -> Dict[str, Any]:
"""Get all blocklist data."""
return self._load_blocklist(force_refresh=True)
def add_disposable_domain(self, domain: str, admin_email: str) -> None:
"""Add a domain to the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
domains.add(domain.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None # Invalidate cache
logger.info(f"Added disposable domain {domain} by {admin_email}")
def remove_disposable_domain(self, domain: str) -> bool:
"""Remove a domain from the disposable domains blocklist."""
blocklist = self._load_blocklist(force_refresh=True)
domains = set(blocklist.get("disposable_domains", []))
if domain.lower() not in [d.lower() for d in domains]:
return False
domains = {d for d in domains if d.lower() != domain.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"disposable_domains": list(domains),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed disposable domain {domain}")
return True
def add_blocked_email(self, email: str, admin_email: str) -> None:
"""Add an email to the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
emails.add(email.lower())
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked email {email} by {admin_email}")
def remove_blocked_email(self, email: str) -> bool:
"""Remove an email from the blocked emails list."""
blocklist = self._load_blocklist(force_refresh=True)
emails = set(blocklist.get("blocked_emails", []))
if email.lower() not in [e.lower() for e in emails]:
return False
emails = {e for e in emails if e.lower() != email.lower()}
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_emails": list(emails),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked email {email}")
return True
def add_blocked_ip(self, ip_address: str, admin_email: str) -> None:
"""Add an IP address to the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
ips.add(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
"updated_by": admin_email,
}, merge=True)
self._blocklist_cache = None
logger.info(f"Added blocked IP {ip_address} by {admin_email}")
def remove_blocked_ip(self, ip_address: str) -> bool:
"""Remove an IP address from the blocked IPs list."""
blocklist = self._load_blocklist(force_refresh=True)
ips = set(blocklist.get("blocked_ips", []))
if ip_address not in ips:
return False
ips.discard(ip_address)
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document("config")
doc_ref.set({
**blocklist,
"blocked_ips": list(ips),
"updated_at": datetime.now(timezone.utc),
}, merge=True)
self._blocklist_cache = None
logger.info(f"Removed blocked IP {ip_address}")
return True
# -------------------------------------------------------------------------
# Stats
# -------------------------------------------------------------------------
def get_rate_limit_stats(self) -> Dict[str, Any]:
"""
Get current rate limit statistics.
Returns:
Dict with current usage stats
"""
date_str = _get_today_date_str()
# Get YouTube upload count
youtube_count = self.get_youtube_uploads_today()
# Count unique users with jobs today
users_with_jobs = set()
total_jobs_today = 0
# Query all user job documents for today
docs = self.db.collection(RATE_LIMITS_COLLECTION).where(
"date", "==", date_str
).stream()
for doc in docs:
data = doc.to_dict()
if data.get("user_email"):
users_with_jobs.add(data["user_email"])
total_jobs_today += data.get("count", 0)
return {
"date": date_str,
"youtube_uploads_today": youtube_count,
"youtube_uploads_limit": settings.rate_limit_youtube_uploads_per_day,
"total_jobs_today": total_jobs_today,
"users_with_jobs_today": len(users_with_jobs),
"job_limit_per_user": settings.rate_limit_jobs_per_day,
"beta_ip_limit_per_day": settings.rate_limit_beta_ip_per_day,
"rate_limiting_enabled": settings.enable_rate_limiting,
"seconds_until_reset": _seconds_until_midnight_utc(),
}
# Singleton instance
_rate_limit_service: Optional[RateLimitService] = None
def get_rate_limit_service() -> RateLimitService:
"""Get the singleton RateLimitService instance."""
global _rate_limit_service
if _rate_limit_service is None:
_rate_limit_service = RateLimitService()
return _rate_limit_service
-65
View File
@@ -1,65 +0,0 @@
"""SpaCy model preloader for container startup.
Loads SpaCy models at container startup to avoid slow loading during request processing.
Cloud Run filesystem I/O can cause 60+ second delays when loading SpaCy models lazily.
"""
import logging
import time
from typing import Optional
logger = logging.getLogger(__name__)
# Singleton storage for preloaded models
_preloaded_models: dict = {}
def preload_spacy_model(model_name: str = "en_core_web_sm") -> None:
"""Preload a SpaCy model at startup.
Args:
model_name: The SpaCy model to load (default: en_core_web_sm)
"""
global _preloaded_models
if model_name in _preloaded_models:
logger.info(f"SpaCy model '{model_name}' already preloaded")
return
logger.info(f"Preloading SpaCy model '{model_name}'...")
start_time = time.time()
try:
import spacy
nlp = spacy.load(model_name)
_preloaded_models[model_name] = nlp
elapsed = time.time() - start_time
logger.info(f"SpaCy model '{model_name}' preloaded in {elapsed:.2f}s")
except Exception as e:
logger.error(f"Failed to preload SpaCy model '{model_name}': {e}")
raise
def get_preloaded_model(model_name: str = "en_core_web_sm") -> Optional[object]:
"""Get a preloaded SpaCy model if available.
Args:
model_name: The SpaCy model name
Returns:
The preloaded SpaCy Language object, or None if not preloaded
"""
return _preloaded_models.get(model_name)
def is_model_preloaded(model_name: str = "en_core_web_sm") -> bool:
"""Check if a SpaCy model has been preloaded."""
return model_name in _preloaded_models
def clear_preloaded_models() -> None:
"""Clear all preloaded models. Useful for testing."""
global _preloaded_models
_preloaded_models.clear()
+11 -133
View File
@@ -46,14 +46,6 @@ CREDIT_PACKAGES = {
},
}
# Made-for-you service package (not a credit package - creates a job directly)
MADE_FOR_YOU_PACKAGE = {
"price_cents": 1500, # $15.00
"name": "Made For You Karaoke Video",
"description": "Professional 4K karaoke video with perfectly synced lyrics, delivered to your email within 24 hours",
"images": ["https://gen.nomadkaraoke.com/nomad-logo.png"],
}
class StripeService:
"""Service for Stripe payment processing."""
@@ -66,10 +58,6 @@ class StripeService:
self.frontend_url = os.getenv("FRONTEND_URL", "https://gen.nomadkaraoke.com")
# After consolidation, buy URL is the same as frontend URL
self.buy_url = os.getenv("BUY_URL", self.frontend_url)
# Payment method configuration ID from Stripe Dashboard
# Enables Google Pay, Apple Pay, Link, and other wallet methods
# Get this from: Dashboard > Settings > Payment methods > [Your Config] > Copy ID
self.payment_method_config = os.getenv("STRIPE_PAYMENT_METHOD_CONFIG")
if self.secret_key:
stripe.api_key = self.secret_key
@@ -77,11 +65,6 @@ class StripeService:
else:
logger.warning("STRIPE_SECRET_KEY not set - payments disabled")
if self.payment_method_config:
logger.info(f"Using payment method config: {self.payment_method_config}")
else:
logger.info("No STRIPE_PAYMENT_METHOD_CONFIG set - using Stripe defaults")
def is_configured(self) -> bool:
"""Check if Stripe is properly configured."""
return bool(self.secret_key)
@@ -123,40 +106,32 @@ class StripeService:
if not cancel_url:
cancel_url = f"{self.buy_url}?cancelled=true"
# Build checkout session params
session_params = {
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
'line_items': [{
# Create checkout session
session = stripe.checkout.Session.create(
payment_method_types=['card'],
line_items=[{
'price_data': {
'currency': 'usd',
'product_data': {
'name': package['name'],
'description': package['description'],
'images': ['https://gen.nomadkaraoke.com/nomad-logo.png'],
},
'unit_amount': package['price_cents'],
},
'quantity': 1,
}],
'mode': 'payment',
'success_url': success_url,
'cancel_url': cancel_url,
'customer_email': user_email,
'metadata': {
mode='payment',
success_url=success_url,
cancel_url=cancel_url,
customer_email=user_email,
metadata={
'package_id': package_id,
'credits': str(package['credits']),
'user_email': user_email,
},
# Allow promotion codes
'allow_promotion_codes': True,
}
# Add payment method configuration if set (enables Google Pay, Link, etc.)
if self.payment_method_config:
session_params['payment_method_configuration'] = self.payment_method_config
# Create checkout session
session = stripe.checkout.Session.create(**session_params)
allow_promotion_codes=True,
)
logger.info(f"Created checkout session {session.id} for {user_email}, package {package_id}")
return True, session.url, "Checkout session created"
@@ -168,103 +143,6 @@ class StripeService:
logger.error(f"Error creating checkout session: {e}")
return False, None, "Failed to create checkout session"
def create_made_for_you_checkout_session(
self,
customer_email: str,
artist: str,
title: str,
source_type: str = "search",
youtube_url: Optional[str] = None,
notes: Optional[str] = None,
success_url: Optional[str] = None,
cancel_url: Optional[str] = None,
) -> Tuple[bool, Optional[str], str]:
"""
Create a Stripe Checkout session for a made-for-you order.
This is for the full-service karaoke video creation where Nomad Karaoke
handles all the work (lyrics review, instrumental selection, etc.).
Args:
customer_email: Customer's email for delivery
artist: Song artist
title: Song title
source_type: Audio source type (search, youtube, upload)
youtube_url: YouTube URL if source_type is youtube
notes: Any special requests from customer
success_url: URL to redirect to on success (optional)
cancel_url: URL to redirect to on cancel (optional)
Returns:
(success, checkout_url, message)
"""
if not self.is_configured():
return False, None, "Payment processing is not configured"
try:
# Default URLs - redirect to gen app success page
if not success_url:
success_url = f"{self.frontend_url}/order/success?session_id={{CHECKOUT_SESSION_ID}}"
if not cancel_url:
cancel_url = "https://nomadkaraoke.com/#do-it-for-me"
# Build metadata for job creation after payment
metadata = {
'order_type': 'made_for_you',
'customer_email': customer_email,
'artist': artist,
'title': title,
'source_type': source_type,
}
if youtube_url:
metadata['youtube_url'] = youtube_url
if notes:
# Truncate notes to fit Stripe's 500 char limit per metadata value
metadata['notes'] = notes[:500] if len(notes) > 500 else notes
# Build checkout session params
session_params = {
# Omit payment_method_types to auto-enable Apple Pay, Google Pay, Link, etc.
'line_items': [{
'price_data': {
'currency': 'usd',
'product_data': {
'name': f"Karaoke Video: {artist} - {title}",
'description': MADE_FOR_YOU_PACKAGE['description'],
'images': MADE_FOR_YOU_PACKAGE['images'],
},
'unit_amount': MADE_FOR_YOU_PACKAGE['price_cents'],
},
'quantity': 1,
}],
'mode': 'payment',
'success_url': success_url,
'cancel_url': cancel_url,
'customer_email': customer_email,
'metadata': metadata,
'allow_promotion_codes': True,
}
# Add payment method configuration if set (enables Google Pay, Link, etc.)
if self.payment_method_config:
session_params['payment_method_configuration'] = self.payment_method_config
# Create checkout session
session = stripe.checkout.Session.create(**session_params)
logger.info(
f"Created made-for-you checkout session {session.id} for {customer_email}, "
f"song: {artist} - {title}"
)
return True, session.url, "Checkout session created"
except stripe.error.StripeError as e:
logger.error(f"Stripe error creating made-for-you checkout session: {e}")
return False, None, f"Payment error: {str(e)}"
except Exception as e:
logger.error(f"Error creating made-for-you checkout session: {e}")
return False, None, "Failed to create checkout session"
def verify_webhook_signature(self, payload: bytes, signature: str) -> Tuple[bool, Optional[Dict], str]:
"""
Verify a Stripe webhook signature.
-330
View File
@@ -1,330 +0,0 @@
"""
Template service for managing email templates stored in GCS.
Templates are stored in GCS bucket and rendered with job-specific variables.
This allows updating email content without code deployment.
Template locations:
- gs://{bucket}/templates/job-completion.txt - Job completion email (plain text)
- gs://{bucket}/templates/action-needed-lyrics.txt - Lyrics review reminder
- gs://{bucket}/templates/action-needed-instrumental.txt - Instrumental selection reminder
"""
import logging
import re
from typing import Optional, Dict, Any
from backend.config import get_settings
logger = logging.getLogger(__name__)
# Default templates (fallback if GCS fetch fails)
DEFAULT_JOB_COMPLETION_TEMPLATE = """Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
"""
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE = """Hi {name},
Your karaoke video for "{artist} - {title}" is ready for lyrics review!
Our system has transcribed and synchronized the lyrics, but they may need some corrections. Please review and make any corrections necessary so we can finish generating your video.
Review your lyrics here:
{review_url}
This usually takes just a few minutes.
Thanks!
-Andrew
"""
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE = """Hi {name},
Your karaoke video for "{artist} - {title}" is almost done!
We've separated the audio into different versions. Please select which instrumental track you'd like to use for the final video.
Select your instrumental here:
{instrumental_url}
Thanks!
-Andrew
"""
class TemplateService:
"""
Service for fetching and rendering email templates from GCS.
Templates support the following variables:
- {name} - User's display name or "there" if unknown
- {youtube_url} - YouTube video URL
- {dropbox_url} - Dropbox folder URL
- {artist} - Artist name
- {title} - Song title
- {job_id} - Job ID
- {review_url} - Lyrics review URL
- {instrumental_url} - Instrumental selection URL
- {feedback_url} - Feedback form URL
"""
TEMPLATE_PREFIX = "templates/"
def __init__(self):
"""Initialize template service."""
self.settings = get_settings()
self._storage_client = None
self._bucket = None
@property
def storage_client(self):
"""Lazy-initialize storage client."""
if self._storage_client is None:
from google.cloud import storage
self._storage_client = storage.Client(project=self.settings.google_cloud_project)
return self._storage_client
@property
def bucket(self):
"""Get the GCS bucket for templates."""
if self._bucket is None:
self._bucket = self.storage_client.bucket(self.settings.gcs_bucket_name)
return self._bucket
def _fetch_template_from_gcs(self, template_name: str) -> Optional[str]:
"""
Fetch a template from GCS.
Args:
template_name: Name of template file (e.g., "job-completion.txt")
Returns:
Template content as string, or None if not found
"""
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
try:
blob = self.bucket.blob(blob_path)
if blob.exists():
content = blob.download_as_text()
logger.debug(f"Fetched template from GCS: {blob_path}")
return content
else:
logger.warning(f"Template not found in GCS: {blob_path}")
return None
except Exception as e:
logger.error(f"Failed to fetch template {blob_path}: {e}")
return None
def get_job_completion_template(self) -> str:
"""Get the job completion email template."""
template = self._fetch_template_from_gcs("job-completion.txt")
if template is None:
logger.info("Using default job completion template")
return DEFAULT_JOB_COMPLETION_TEMPLATE
return template
def get_action_needed_lyrics_template(self) -> str:
"""Get the lyrics review reminder template."""
template = self._fetch_template_from_gcs("action-needed-lyrics.txt")
if template is None:
logger.info("Using default lyrics reminder template")
return DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
return template
def get_action_needed_instrumental_template(self) -> str:
"""Get the instrumental selection reminder template."""
template = self._fetch_template_from_gcs("action-needed-instrumental.txt")
if template is None:
logger.info("Using default instrumental reminder template")
return DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
return template
def render_template(self, template: str, variables: Dict[str, Any]) -> str:
"""
Render a template with the given variables.
Missing variables are replaced with empty strings.
Handles conditional sections like feedback URL.
Args:
template: Template string with {variable} placeholders
variables: Dictionary of variable values
Returns:
Rendered template string
"""
result = template
# Handle feedback URL section - remove if not provided
if not variables.get("feedback_url"):
# Remove feedback section (lines containing feedback_url placeholder and surrounding text)
result = re.sub(
r'\n*If you have a moment.*?\{feedback_url\}\n*',
'\n',
result,
flags=re.DOTALL
)
variables["feedback_url"] = ""
# Replace all variables
for key, value in variables.items():
placeholder = "{" + key + "}"
result = result.replace(placeholder, str(value) if value else "")
# Clean up any remaining unreplaced placeholders
result = re.sub(r'\{[a-z_]+\}', '', result)
return result.strip()
def render_job_completion(
self,
name: Optional[str] = None,
youtube_url: Optional[str] = None,
dropbox_url: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
job_id: Optional[str] = None,
feedback_url: Optional[str] = None,
) -> str:
"""
Render the job completion email template.
Args:
name: User's display name (defaults to "there")
youtube_url: YouTube video URL
dropbox_url: Dropbox folder URL
artist: Artist name
title: Song title
job_id: Job ID
feedback_url: Feedback form URL (optional)
Returns:
Rendered email content
"""
template = self.get_job_completion_template()
variables = {
"name": name or "there",
"youtube_url": youtube_url or "[YouTube URL not available]",
"dropbox_url": dropbox_url or "[Dropbox URL not available]",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"job_id": job_id or "",
"feedback_url": feedback_url,
}
return self.render_template(template, variables)
def render_action_needed_lyrics(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
review_url: str = "",
) -> str:
"""
Render the lyrics review reminder template.
Args:
name: User's display name
artist: Artist name
title: Song title
review_url: Lyrics review URL
Returns:
Rendered email content
"""
template = self.get_action_needed_lyrics_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"review_url": review_url,
}
return self.render_template(template, variables)
def render_action_needed_instrumental(
self,
name: Optional[str] = None,
artist: Optional[str] = None,
title: Optional[str] = None,
instrumental_url: str = "",
) -> str:
"""
Render the instrumental selection reminder template.
Args:
name: User's display name
artist: Artist name
title: Song title
instrumental_url: Instrumental selection URL
Returns:
Rendered email content
"""
template = self.get_action_needed_instrumental_template()
variables = {
"name": name or "there",
"artist": artist or "Unknown Artist",
"title": title or "Unknown Title",
"instrumental_url": instrumental_url,
}
return self.render_template(template, variables)
def upload_template(self, template_name: str, content: str) -> bool:
"""
Upload a template to GCS.
Args:
template_name: Name of template file (e.g., "job-completion.txt")
content: Template content
Returns:
True if upload successful
"""
blob_path = f"{self.TEMPLATE_PREFIX}{template_name}"
try:
blob = self.bucket.blob(blob_path)
blob.upload_from_string(content, content_type="text/plain")
logger.info(f"Uploaded template to GCS: {blob_path}")
return True
except Exception as e:
logger.error(f"Failed to upload template {blob_path}: {e}")
return False
# Global instance
_template_service: Optional[TemplateService] = None
def get_template_service() -> TemplateService:
"""Get the global template service instance."""
global _template_service
if _template_service is None:
_template_service = TemplateService()
return _template_service
-285
View File
@@ -1,285 +0,0 @@
"""
Tenant service for managing white-label B2B portal configurations.
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json.
The service provides:
- Tenant config loading with caching
- Subdomain to tenant ID resolution
- Tenant validation
Similar to ThemeService, configs are cached for 5 minutes to reduce GCS reads.
"""
import logging
import threading
from datetime import datetime
from typing import Dict, Optional
from backend.models.tenant import TenantConfig, TenantPublicConfig
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
# GCS paths for tenant configs
TENANTS_PREFIX = "tenants"
# Default sender email (shared with EmailService for consistency)
DEFAULT_SENDER_EMAIL = "gen@nomadkaraoke.com"
class TenantService:
"""Service for managing tenant configurations from GCS."""
def __init__(self, storage: Optional[StorageService] = None):
"""
Initialize the tenant service.
Args:
storage: StorageService instance (creates new one if not provided)
"""
self.storage = storage or StorageService()
self._config_cache: Dict[str, TenantConfig] = {}
self._cache_times: Dict[str, datetime] = {}
self._subdomain_map: Dict[str, str] = {} # subdomain -> tenant_id
self._subdomain_map_time: Optional[datetime] = None
self.CACHE_TTL_SECONDS = 300 # 5 minute cache
def _get_config_path(self, tenant_id: str) -> str:
"""Get the GCS path for a tenant's config file."""
return f"{TENANTS_PREFIX}/{tenant_id}/config.json"
def _is_cache_valid(self, tenant_id: str) -> bool:
"""Check if the config cache for a tenant is still valid."""
if tenant_id not in self._config_cache or tenant_id not in self._cache_times:
return False
age = datetime.now() - self._cache_times[tenant_id]
return age.total_seconds() < self.CACHE_TTL_SECONDS
def _is_subdomain_map_valid(self) -> bool:
"""Check if the subdomain map cache is still valid."""
if not self._subdomain_map or self._subdomain_map_time is None:
return False
age = datetime.now() - self._subdomain_map_time
return age.total_seconds() < self.CACHE_TTL_SECONDS
def get_tenant_config(
self, tenant_id: str, force_refresh: bool = False
) -> Optional[TenantConfig]:
"""
Load tenant configuration from GCS with caching.
Args:
tenant_id: The tenant identifier
force_refresh: Force reload from GCS even if cache is valid
Returns:
TenantConfig if found, None otherwise
"""
if not force_refresh and self._is_cache_valid(tenant_id):
return self._config_cache[tenant_id]
try:
config_path = self._get_config_path(tenant_id)
if not self.storage.file_exists(config_path):
logger.debug(f"Tenant config not found: {tenant_id}")
return None
data = self.storage.download_json(config_path)
config = TenantConfig(**data)
# Update cache
self._config_cache[tenant_id] = config
self._cache_times[tenant_id] = datetime.now()
# Update subdomain map and its timestamp
self._subdomain_map[config.subdomain.lower()] = tenant_id
self._subdomain_map_time = datetime.now()
logger.info(f"Loaded tenant config: {tenant_id}")
return config
except Exception as e:
logger.error(f"Failed to load tenant config {tenant_id}: {e}")
return None
def get_tenant_by_subdomain(self, subdomain: str) -> Optional[TenantConfig]:
"""
Get tenant config by subdomain.
Args:
subdomain: Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')
Returns:
TenantConfig if found, None otherwise
"""
subdomain_lower = subdomain.lower()
# Check cache first
if subdomain_lower in self._subdomain_map:
tenant_id = self._subdomain_map[subdomain_lower]
return self.get_tenant_config(tenant_id)
# Try to find tenant by listing all configs
# This is a fallback for when we don't have the subdomain cached
tenant_id = self._resolve_subdomain_to_tenant_id(subdomain_lower)
if tenant_id:
return self.get_tenant_config(tenant_id)
return None
def _resolve_subdomain_to_tenant_id(self, subdomain: str) -> Optional[str]:
"""
Resolve a subdomain to tenant ID by checking GCS.
This extracts the tenant ID from the subdomain pattern:
- 'vocalstar.nomadkaraoke.com' -> 'vocalstar'
- 'vocalstar.gen.nomadkaraoke.com' -> 'vocalstar'
"""
# Extract potential tenant ID from subdomain
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
parts = subdomain.split(".")
if len(parts) >= 3:
# First part is likely the tenant ID
potential_tenant_id = parts[0]
# Check if config exists
config_path = self._get_config_path(potential_tenant_id)
if self.storage.file_exists(config_path):
return potential_tenant_id
return None
def get_public_config(self, tenant_id: str) -> Optional[TenantPublicConfig]:
"""
Get the public (frontend-safe) tenant configuration.
Args:
tenant_id: The tenant identifier
Returns:
TenantPublicConfig if found, None otherwise
"""
config = self.get_tenant_config(tenant_id)
if config:
return TenantPublicConfig.from_config(config)
return None
def get_public_config_by_subdomain(
self, subdomain: str
) -> Optional[TenantPublicConfig]:
"""
Get the public tenant configuration by subdomain.
Args:
subdomain: Full subdomain
Returns:
TenantPublicConfig if found, None otherwise
"""
config = self.get_tenant_by_subdomain(subdomain)
if config:
return TenantPublicConfig.from_config(config)
return None
def tenant_exists(self, tenant_id: str) -> bool:
"""
Check if a tenant exists.
Args:
tenant_id: The tenant identifier
Returns:
True if tenant config exists, False otherwise
"""
config_path = self._get_config_path(tenant_id)
return self.storage.file_exists(config_path)
def is_email_allowed_for_tenant(self, tenant_id: str, email: str) -> bool:
"""
Check if an email is allowed for a specific tenant.
Args:
tenant_id: The tenant identifier
email: Email address to check
Returns:
True if email is allowed, False otherwise
"""
config = self.get_tenant_config(tenant_id)
if not config:
return False
return config.is_email_allowed(email)
def get_tenant_sender_email(self, tenant_id: str) -> str:
"""
Get the email sender address for a tenant.
Args:
tenant_id: The tenant identifier
Returns:
Sender email address
"""
config = self.get_tenant_config(tenant_id)
if config:
return config.get_sender_email()
# Default fallback (consistent with EmailService)
return DEFAULT_SENDER_EMAIL
def invalidate_cache(self, tenant_id: Optional[str] = None) -> None:
"""
Invalidate tenant config cache.
Args:
tenant_id: Specific tenant to invalidate, or None for all
"""
if tenant_id:
self._config_cache.pop(tenant_id, None)
self._cache_times.pop(tenant_id, None)
# Also remove any subdomain map entries pointing to this tenant
subdomains_to_remove = [
subdomain
for subdomain, tid in self._subdomain_map.items()
if tid == tenant_id
]
for subdomain in subdomains_to_remove:
self._subdomain_map.pop(subdomain, None)
logger.info(f"Tenant config cache invalidated: {tenant_id}")
else:
self._config_cache.clear()
self._cache_times.clear()
self._subdomain_map.clear()
self._subdomain_map_time = None
logger.info("All tenant config caches invalidated")
def get_asset_url(self, tenant_id: str, asset_name: str) -> Optional[str]:
"""
Get a signed URL for a tenant asset (logo, favicon, etc.).
Args:
tenant_id: The tenant identifier
asset_name: Asset filename (e.g., 'logo.png')
Returns:
Signed URL if asset exists, None otherwise
"""
asset_path = f"{TENANTS_PREFIX}/{tenant_id}/{asset_name}"
if self.storage.file_exists(asset_path):
return self.storage.generate_signed_url(asset_path, expiration_minutes=60)
return None
# Singleton instance with thread-safe initialization
_tenant_service: Optional[TenantService] = None
_tenant_service_lock = threading.Lock()
def get_tenant_service() -> TenantService:
"""Get or create the singleton TenantService instance (thread-safe)."""
global _tenant_service
if _tenant_service is None:
with _tenant_service_lock:
# Double-check after acquiring lock
if _tenant_service is None:
_tenant_service = TenantService()
return _tenant_service
+7 -85
View File
@@ -73,26 +73,16 @@ class UserService:
logger.exception(f"Error getting user {email}")
return None
def get_or_create_user(self, email: str, tenant_id: Optional[str] = None) -> User:
def get_or_create_user(self, email: str) -> User:
"""
Get existing user or create a new one.
New users receive a welcome credit to try the service.
Args:
email: User's email address
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
Note: If user exists but has a different tenant_id, the existing user is returned.
Users are uniquely identified by email, not email+tenant.
"""
email = email.lower()
user = self.get_user(email)
if user:
# If user exists but tenant_id differs, we still return the user
# This allows users to access multiple tenants with one account
# (though features may be restricted per-tenant)
return user
# Create new user with welcome credit
@@ -107,10 +97,9 @@ class UserService:
email=email,
credits=welcome_credit,
credit_transactions=[welcome_transaction],
tenant_id=tenant_id, # Associate with tenant on creation
)
self._save_user(user)
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s) (tenant: {tenant_id or 'default'})")
logger.info(f"Created new user: {email} with {welcome_credit} welcome credit(s)")
return user
def _save_user(self, user: User) -> None:
@@ -164,7 +153,6 @@ class UserService:
display_name=user.display_name,
total_jobs_created=user.total_jobs_created,
total_jobs_completed=user.total_jobs_completed,
tenant_id=user.tenant_id,
)
# =========================================================================
@@ -175,25 +163,18 @@ class UserService:
self,
email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None
user_agent: Optional[str] = None
) -> MagicLinkToken:
"""
Create a magic link token for email authentication.
Returns the token object. The actual sending of the email
should be handled by the caller (or an email service).
Args:
email: User's email address
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
"""
email = email.lower()
# Ensure user exists (with tenant association)
self.get_or_create_user(email, tenant_id=tenant_id)
# Ensure user exists
self.get_or_create_user(email)
# Generate secure token
token = secrets.token_urlsafe(32)
@@ -204,7 +185,6 @@ class UserService:
expires_at=datetime.utcnow() + timedelta(minutes=MAGIC_LINK_EXPIRY_MINUTES),
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
)
# Save to Firestore
@@ -214,54 +194,6 @@ class UserService:
logger.info(f"Created magic link for {email}")
return magic_link
def create_admin_login_token(
self,
email: str,
expiry_hours: int = 24,
) -> MagicLinkToken:
"""
Create an admin login token for email-embedded authentication links.
Similar to magic links but with configurable expiry (default 24 hours).
Used for made-for-you order notification emails to allow admin one-click login.
Args:
email: Admin's email address to authenticate as
expiry_hours: Hours until token expires (default: 24, max: 168)
Returns:
MagicLinkToken object containing the token
Raises:
ValueError: If expiry_hours is out of valid range (1-168)
"""
# Validate expiry_hours (1 hour to 7 days)
if not 1 <= expiry_hours <= 168:
raise ValueError(f"expiry_hours must be between 1 and 168, got {expiry_hours}")
email = email.lower()
# Ensure user exists
self.get_or_create_user(email)
# Generate secure token
token = secrets.token_urlsafe(32)
admin_login = MagicLinkToken(
token=token,
email=email,
expires_at=datetime.utcnow() + timedelta(hours=expiry_hours),
)
# Save to Firestore (same collection as magic links for unified verification)
doc_ref = self.db.collection(MAGIC_LINKS_COLLECTION).document(token)
doc_ref.set(admin_login.model_dump(mode='json'))
# Log with redacted email (show only domain) for PII protection
domain = email.split('@')[-1] if '@' in email else 'unknown'
logger.info(f"Created admin login token for ***@{domain} (expires in {expiry_hours}h)")
return admin_login
def verify_magic_link(self, token: str) -> Tuple[bool, Optional[User], str]:
"""
Verify a magic link token using a Firestore transaction to prevent race conditions.
@@ -330,18 +262,9 @@ class UserService:
self,
user_email: str,
ip_address: Optional[str] = None,
user_agent: Optional[str] = None,
tenant_id: Optional[str] = None
user_agent: Optional[str] = None
) -> Session:
"""
Create a new session for an authenticated user.
Args:
user_email: User's email address
ip_address: Client IP for auditing
user_agent: Client user agent for auditing
tenant_id: Tenant ID for white-label portals (None = default Nomad Karaoke)
"""
"""Create a new session for an authenticated user."""
token = secrets.token_urlsafe(32)
token_prefix = token[:12]
@@ -351,7 +274,6 @@ class UserService:
expires_at=datetime.utcnow() + timedelta(days=SESSION_ABSOLUTE_EXPIRY_DAYS),
ip_address=ip_address,
user_agent=user_agent,
tenant_id=tenant_id,
)
# Serialize and write to Firestore
-98
View File
@@ -37,7 +37,6 @@ WORKER_QUEUES = {
"screens": "screens-worker-queue",
"render-video": "render-worker-queue",
"video": "video-worker-queue",
"idle-reminder": "idle-reminder-queue", # For delayed idle reminder checks
}
# Dispatch deadlines for each worker type (in seconds)
@@ -50,12 +49,8 @@ WORKER_DISPATCH_DEADLINES = {
"screens": 600, # 10 min - Screen generation is fast
"render-video": 1800, # 30 min - Video encoding can be slow
"video": 1800, # 30 min - Video encoding can be slow
"idle-reminder": 60, # 1 min - Quick check and potential email send
}
# Default delay for idle reminders (seconds)
IDLE_REMINDER_DELAY_SECONDS = 5 * 60 # 5 minutes
class WorkerService:
"""
@@ -435,99 +430,6 @@ class WorkerService:
"""Trigger render video worker (post-review)."""
return await self.trigger_worker("render-video", job_id)
async def schedule_idle_reminder(
self,
job_id: str,
delay_seconds: int = IDLE_REMINDER_DELAY_SECONDS
) -> bool:
"""
Schedule an idle reminder check for a job.
The reminder task will be delivered after the specified delay.
When the task executes, it checks if the job is still in a blocking
state and sends a reminder email if the user hasn't taken action.
Args:
job_id: Job ID to check
delay_seconds: Delay before executing the check (default: 5 minutes)
Returns:
True if task was scheduled successfully, False otherwise
"""
if not self._use_cloud_tasks:
# In development mode, log and skip (no delayed execution support)
logger.info(
f"[job:{job_id}] Idle reminder not scheduled (Cloud Tasks disabled). "
f"Would have fired in {delay_seconds}s."
)
return True
try:
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import time
queue_name = WORKER_QUEUES.get("idle-reminder")
if not queue_name:
logger.error("Idle reminder queue not configured")
return False
project = self.settings.google_cloud_project
if not project:
logger.error("GOOGLE_CLOUD_PROJECT not set")
return False
location = self.settings.gcp_region
# Build queue path
parent = self.tasks_client.queue_path(project, location, queue_name)
# Build headers
headers = {
"Content-Type": "application/json",
}
if self._admin_token:
headers["X-Admin-Token"] = self._admin_token
# Inject trace context
headers = inject_trace_context(headers)
# Calculate schedule time
schedule_time = timestamp_pb2.Timestamp()
schedule_time.FromSeconds(int(time.time()) + delay_seconds)
dispatch_deadline_seconds = WORKER_DISPATCH_DEADLINES.get("idle-reminder", 60)
# Build task payload
task = {
"http_request": {
"http_method": tasks_v2.HttpMethod.POST,
"url": f"{self._base_url}/api/internal/jobs/{job_id}/check-idle-reminder",
"headers": headers,
"body": json.dumps({"job_id": job_id}).encode(),
"oidc_token": {
"service_account_email": f"karaoke-backend@{project}.iam.gserviceaccount.com",
},
},
"schedule_time": schedule_time,
"dispatch_deadline": duration_pb2.Duration(seconds=dispatch_deadline_seconds),
}
# Create task
response = self.tasks_client.create_task(parent=parent, task=task)
logger.info(
f"[job:{job_id}] Scheduled idle reminder check in {delay_seconds}s: {response.name}"
)
return True
except Exception as e:
logger.error(
f"[job:{job_id}] Failed to schedule idle reminder: {e}",
exc_info=True
)
return False
# Global worker service instance
_worker_service: Optional[WorkerService] = None
+1 -8
View File
@@ -91,14 +91,7 @@ def mock_auth_dependency(request):
if 'emulator' in test_path or 'integration' in test_path:
yield
return
# Skip for service-only unit tests that don't need the FastAPI app
# Also skip tests that manage their own auth mocks (e.g., push tests use AuthResult objects)
service_only_tests = ['test_rate_limit_service', 'test_email_validation_service', 'test_rate_limits_api', 'test_gce_encoding_worker', 'test_push_routes', 'test_push_notification_service']
if any(test_name in test_path for test_name in service_only_tests):
yield
return
# Skip if FIRESTORE_EMULATOR_HOST is set (running in emulator environment)
import os
if os.environ.get('FIRESTORE_EMULATOR_HOST'):
+1 -22
View File
@@ -39,24 +39,6 @@ os.environ["ADMIN_TOKENS"] = "test-admin-token"
# Only import app if emulators are running
if emulators_running():
from unittest.mock import Mock
# Mock theme service BEFORE importing app to ensure all jobs get theme_id="nomad"
_mock_theme_service = Mock()
_mock_theme_service.get_default_theme_id.return_value = "nomad"
_mock_theme_service.get_theme.return_value = None
# Apply patch at module load time (before app imports the service)
_theme_patches = [
patch("backend.services.theme_service.get_theme_service", return_value=_mock_theme_service),
patch("backend.api.routes.audio_search.get_theme_service", return_value=_mock_theme_service),
patch("backend.api.routes.file_upload.get_theme_service", return_value=_mock_theme_service),
patch("backend.api.routes.users.get_theme_service", return_value=_mock_theme_service),
patch("backend.api.routes.jobs.get_theme_service", return_value=_mock_theme_service),
]
for p in _theme_patches:
p.start()
from fastapi.testclient import TestClient
from backend.main import app
else:
@@ -94,10 +76,7 @@ def mock_worker_service():
@pytest.fixture(scope="session")
def client(mock_worker_service):
"""Create FastAPI test client with mocked workers.
Theme service is mocked at module load time (see above).
"""
"""Create FastAPI test client with mocked workers."""
with patch("backend.api.routes.file_upload.worker_service", mock_worker_service):
return TestClient(app)
@@ -1,167 +0,0 @@
"""
Emulator integration tests for made-for-you order features.
Tests with real Firestore emulator via API endpoints to verify:
1. Admin login token creation and verification
2. Made-for-you job fields are properly stored
Run with: scripts/run-emulator-tests.sh
"""
import pytest
import time
from .conftest import emulators_running
# Skip all tests if emulators not running
pytestmark = pytest.mark.skipif(
not emulators_running(),
reason="GCP emulators not running. Start with: scripts/start-emulators.sh"
)
class TestAdminLoginTokenIntegration:
"""
Integration tests for admin login token feature via API.
These tests verify that admin login tokens can be verified
through the actual API endpoint.
"""
def test_verify_invalid_token_returns_401(self, client):
"""Invalid token returns 401 unauthorized."""
response = client.get("/api/users/auth/verify?token=invalid-token-xyz")
assert response.status_code == 401
def test_verify_missing_token_returns_422(self, client):
"""Missing token parameter returns 422 validation error."""
response = client.get("/api/users/auth/verify")
assert response.status_code == 422
def test_verify_empty_token_returns_401(self, client):
"""Empty token returns 401."""
response = client.get("/api/users/auth/verify?token=")
assert response.status_code == 401
class TestMadeForYouJobModel:
"""
Tests for made-for-you job model fields.
Verifies that the Job model properly supports made_for_you fields.
"""
def test_job_model_has_made_for_you_fields(self):
"""Job model includes made_for_you and customer_email fields."""
from backend.models.job import Job, JobStatus
from datetime import datetime, timezone
# Create a made-for-you job directly
job = Job(
job_id="test-mfy-model",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="admin@nomadkaraoke.com",
made_for_you=True,
customer_email="customer@example.com",
customer_notes="Please make it perfect!",
)
assert job.made_for_you is True
assert job.customer_email == "customer@example.com"
assert job.customer_notes == "Please make it perfect!"
def test_job_model_defaults_made_for_you_false(self):
"""Regular jobs default made_for_you to False."""
from backend.models.job import Job, JobStatus
from datetime import datetime, timezone
job = Job(
job_id="test-regular-model",
status=JobStatus.PENDING,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc),
user_email="user@example.com",
)
# Default should be False or None
assert job.made_for_you in [False, None]
assert job.customer_email is None
class TestOwnershipTransferLogic:
"""
Tests for ownership transfer logic (unit-style, runs in emulator context).
These tests verify the conditional logic for made-for-you ownership transfer
by importing the job_manager and testing its behavior directly.
"""
def test_ownership_transfer_condition_made_for_you_true(self):
"""
Verify the condition: if made_for_you=True and customer_email exists,
ownership should transfer.
"""
# Replicate the exact logic from _schedule_completion_email
class MockJob:
made_for_you = True
customer_email = "customer@example.com"
user_email = "admin@nomadkaraoke.com"
job_id = "test-123"
job = MockJob()
# The logic being tested
recipient_email = job.user_email
should_transfer = False
if job.made_for_you and job.customer_email:
recipient_email = job.customer_email
should_transfer = True
assert should_transfer is True
assert recipient_email == "customer@example.com"
def test_ownership_transfer_condition_made_for_you_false(self):
"""
Verify: if made_for_you=False, no ownership transfer.
"""
class MockJob:
made_for_you = False
customer_email = "customer@example.com"
user_email = "user@example.com"
job_id = "test-456"
job = MockJob()
recipient_email = job.user_email
should_transfer = False
if job.made_for_you and job.customer_email:
recipient_email = job.customer_email
should_transfer = True
assert should_transfer is False
assert recipient_email == "user@example.com"
def test_ownership_transfer_condition_missing_customer_email(self):
"""
Verify: if made_for_you=True but no customer_email, no transfer.
"""
class MockJob:
made_for_you = True
customer_email = None
user_email = "admin@nomadkaraoke.com"
job_id = "test-789"
job = MockJob()
recipient_email = job.user_email
should_transfer = False
if job.made_for_you and job.customer_email:
recipient_email = job.customer_email
should_transfer = True
assert should_transfer is False
assert recipient_email == "admin@nomadkaraoke.com"
-352
View File
@@ -1,352 +0,0 @@
"""
Unit tests for admin delete job outputs endpoint.
Tests the POST /api/admin/jobs/{job_id}/delete-outputs endpoint that allows admins
to delete distributed outputs (YouTube, Dropbox, Google Drive) while preserving
the job record.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from fastapi.testclient import TestClient
from fastapi import FastAPI
from datetime import datetime
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
from backend.api.dependencies import AuthResult, UserType
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Admin authenticated",
user_email="admin@example.com",
is_admin=True,
)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_complete_job():
"""Create a mock job in COMPLETE status with distribution data."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Title"
job.status = "complete"
job.dropbox_path = "/Karaoke/Organized"
job.outputs_deleted_at = None
job.outputs_deleted_by = None
job.state_data = {
"youtube_url": "https://youtu.be/abc123",
"brand_code": "NOMAD-1234",
"dropbox_link": "https://dropbox.com/...",
"gdrive_files": {"mp4": "file_id_1", "mp4_720p": "file_id_2"},
}
job.timeline = []
return job
@pytest.fixture
def mock_job_no_outputs():
"""Create a mock job in COMPLETE status without distribution data."""
job = Mock(spec=Job)
job.job_id = "test-job-456"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Title"
job.status = "complete"
job.dropbox_path = None
job.outputs_deleted_at = None
job.outputs_deleted_by = None
job.state_data = {}
job.timeline = []
return job
class TestDeleteOutputsSuccess:
"""Tests for successful output deletion."""
def test_delete_outputs_success(self, client, mock_complete_job):
"""Test successfully deleting outputs from a complete job."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["job_id"] == "test-job-123"
assert "outputs_deleted_at" in data
def test_delete_outputs_clears_state_data_keys(self, client, mock_complete_job):
"""Test that output-related state_data keys are listed as cleared."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
# Should list cleared keys
assert "youtube_url" in data["cleared_state_data"]
assert "brand_code" in data["cleared_state_data"]
assert "dropbox_link" in data["cleared_state_data"]
assert "gdrive_files" in data["cleared_state_data"]
def test_delete_outputs_job_without_distribution(self, client, mock_job_no_outputs):
"""Test deleting outputs from job that has no distribution data."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_no_outputs
mock_jm_class.return_value = mock_jm
# Mock Firestore
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-456/delete-outputs")
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
# All services should be skipped
assert data["deleted_services"]["youtube"]["status"] == "skipped"
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
assert data["deleted_services"]["gdrive"]["status"] == "skipped"
# No keys to clear
assert data["cleared_state_data"] == []
class TestDeleteOutputsValidation:
"""Tests for validation on delete outputs endpoint."""
def test_rejects_non_terminal_status(self, client, mock_complete_job):
"""Test that jobs not in terminal states are rejected."""
mock_complete_job.status = "encoding"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 400
assert "terminal" in response.json()["detail"].lower()
def test_rejects_awaiting_review_status(self, client, mock_complete_job):
"""Test that AWAITING_REVIEW is rejected (not terminal)."""
mock_complete_job.status = "awaiting_review"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 400
def test_accepts_prep_complete_status(self, client, mock_complete_job):
"""Test that prep_complete is accepted as terminal."""
mock_complete_job.status = "prep_complete"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
def test_accepts_failed_status(self, client, mock_complete_job):
"""Test that failed is accepted as terminal."""
mock_complete_job.status = "failed"
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
def test_rejects_already_deleted(self, client, mock_complete_job):
"""Test that already-deleted outputs cannot be deleted again."""
mock_complete_job.outputs_deleted_at = datetime(2026, 1, 9, 12, 0, 0)
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 400
assert "already deleted" in response.json()["detail"].lower()
def test_returns_404_for_missing_job(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post("/api/admin/jobs/nonexistent/delete-outputs")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
class TestDeleteOutputsServices:
"""Tests for service deletion behavior."""
def test_handles_missing_youtube(self, client, mock_complete_job):
"""Test handling when job has no YouTube URL."""
mock_complete_job.state_data = {"brand_code": "NOMAD-1234"}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
assert data["deleted_services"]["youtube"]["status"] == "skipped"
def test_handles_missing_dropbox_path(self, client, mock_complete_job):
"""Test handling when job has no dropbox_path."""
mock_complete_job.dropbox_path = None
mock_complete_job.state_data = {"youtube_url": "https://youtu.be/abc123"}
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
data = response.json()
assert data["deleted_services"]["dropbox"]["status"] == "skipped"
class TestDeleteOutputsLogging:
"""Tests for logging on delete outputs endpoint."""
def test_logs_admin_action(self, client, mock_complete_job):
"""Test that admin delete action is logged."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.get_user_service') as mock_user_service, \
patch('backend.api.routes.admin.logger') as mock_logger:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_complete_job
mock_jm_class.return_value = mock_jm
mock_db = Mock()
mock_job_ref = Mock()
mock_db.collection.return_value.document.return_value = mock_job_ref
mock_user_service.return_value.db = mock_db
response = client.post("/api/admin/jobs/test-job-123/delete-outputs")
assert response.status_code == 200
mock_logger.info.assert_called()
log_message = mock_logger.info.call_args[0][0]
assert "admin" in log_message.lower() or "deleted" in log_message.lower()
class TestDeleteOutputsAuthorization:
"""Tests for authorization on the delete outputs endpoint."""
def test_requires_admin_access(self, client, mock_complete_job):
"""Test that non-admin users cannot access the endpoint."""
original_override = app.dependency_overrides.get(require_admin)
def get_non_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = get_non_admin
try:
response = client.post(
"/api/admin/jobs/test-job-123/delete-outputs",
headers={"Authorization": "Bearer user-token"}
)
assert response.status_code == 403
finally:
if original_override:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
-411
View File
@@ -1,411 +0,0 @@
"""
Unit tests for admin email endpoints.
Tests the completion message and send email API endpoints.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock, AsyncMock
from fastapi.testclient import TestClient
from fastapi import FastAPI
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
# The router already has prefix="/admin", so we add /api prefix
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
return ("admin@example.com", "admin", 1)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_job():
"""Create a mock job."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Song"
job.status = JobStatus.COMPLETE
job.audio_hash = "hash123"
job.review_token = "review123"
job.instrumental_token = "inst123"
job.state_data = {
"youtube_url": "https://youtube.com/watch?v=test123",
"dropbox_link": "https://dropbox.com/folder/test",
"brand_code": "NOMAD-1234",
}
return job
class TestGetCompletionMessage:
"""Tests for GET /api/admin/jobs/{job_id}/completion-message endpoint."""
def test_returns_completion_message(self, client, mock_job):
"""Test successful completion message retrieval."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
# Setup mocks
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["job_id"] == "test-job-123"
assert data["message"] == "Your video is ready!"
# Subject format: "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
assert data["subject"] == "NOMAD-1234: Test Artist - Test Song (Your karaoke video is ready!)"
assert data["youtube_url"] == "https://youtube.com/watch?v=test123"
assert data["dropbox_url"] == "https://dropbox.com/folder/test"
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.get(
"/api/admin/jobs/nonexistent-job/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
def test_handles_none_state_data(self, client, mock_job):
"""Test handling of job with None state_data."""
mock_job.state_data = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["youtube_url"] is None
assert data["dropbox_url"] is None
def test_default_subject_without_song_info(self, client, mock_job):
"""Test default subject when no artist/title."""
mock_job.artist = None
mock_job.title = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
response = client.get(
"/api/admin/jobs/test-job-123/completion-message",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["subject"] == "Your karaoke video is ready!"
class TestSendCompletionEmail:
"""Tests for POST /api/admin/jobs/{job_id}/send-completion-email endpoint."""
def test_sends_email_successfully(self, client, mock_job):
"""Test successful email sending."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Your video is ready!"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com", "cc_admin": True},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["success"] is True
assert data["job_id"] == "test-job-123"
assert data["to_email"] == "customer@example.com"
# Verify email was sent with correct params
mock_es.send_job_completion.assert_called_once_with(
to_email="customer@example.com",
message_content="Your video is ready!",
artist="Test Artist",
title="Test Song",
brand_code="NOMAD-1234",
cc_admin=True,
)
def test_sends_email_without_cc(self, client, mock_job):
"""Test email sending without CC."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com", "cc_admin": False},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
# Verify CC was not included
call_kwargs = mock_es.send_job_completion.call_args.kwargs
assert call_kwargs["cc_admin"] is False
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/nonexistent-job/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
def test_returns_500_when_email_fails(self, client, mock_job):
"""Test 500 when email sending fails."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = False # Email failed
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 500
assert "Failed to send email" in response.json()["detail"]
def test_handles_none_state_data(self, client, mock_job):
"""Test handling of job with None state_data."""
mock_job.state_data = None
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "customer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
def test_accepts_any_email_string(self, client, mock_job):
"""Test that any string is currently accepted as email (no validation).
Documents current behavior: FastAPI/Pydantic doesn't validate email
format by default. If validation is added later, this test should
be updated to expect 422 for invalid emails.
"""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns, \
patch('backend.services.email_service.get_email_service') as mock_get_es:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.get_completion_message.return_value = "Message"
mock_get_ns.return_value = mock_ns
mock_es = Mock()
mock_es.send_job_completion.return_value = True
mock_get_es.return_value = mock_es
response = client.post(
"/api/admin/jobs/test-job-123/send-completion-email",
json={"to_email": "invalid-email"}, # Not a valid email format
headers={"Authorization": "Bearer admin-token"}
)
# Currently accepts any string - endpoint succeeds
assert response.status_code == 200
# Verify the invalid email was passed through
mock_es.send_job_completion.assert_called_once()
call_kwargs = mock_es.send_job_completion.call_args.kwargs
assert call_kwargs["to_email"] == "invalid-email"
class TestIdleReminderEndpoint:
"""Tests for the internal idle reminder endpoint."""
def test_sends_reminder_when_still_idle(self, mock_job):
"""Test that reminder is sent when user is still idle."""
from backend.api.routes.internal import router as internal_router
from backend.api.dependencies import require_admin
# Create test app for internal router
# The router already has prefix="/internal", so we add /api prefix
test_app = FastAPI()
test_app.include_router(internal_router, prefix="/api")
test_app.dependency_overrides[require_admin] = get_mock_admin
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
mock_job.state_data = {
"blocking_state_entered_at": "2024-01-01T00:00:00",
"blocking_action_type": "lyrics",
"reminder_sent": False,
}
with patch('backend.api.routes.internal.JobManager') as mock_jm_class, \
patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_ns:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.firestore = Mock() # Mock firestore for update_job call
mock_jm_class.return_value = mock_jm
mock_ns = Mock()
mock_ns.send_action_reminder_email = AsyncMock(return_value=True)
mock_get_ns.return_value = mock_ns
test_client = TestClient(test_app)
response = test_client.post(
"/api/internal/jobs/test-job-123/check-idle-reminder",
headers={"Authorization": "Bearer admin-token"}
)
# Should succeed and return "sent" status
assert response.status_code == 200
data = response.json()
assert data["status"] == "sent"
assert data["job_id"] == "test-job-123"
def test_skips_reminder_when_already_sent(self, mock_job):
"""Test that reminder is skipped if already sent."""
from backend.api.routes.internal import router as internal_router
from backend.api.dependencies import require_admin
# Create test app for internal router
# The router already has prefix="/internal", so we add /api prefix
test_app = FastAPI()
test_app.include_router(internal_router, prefix="/api")
test_app.dependency_overrides[require_admin] = get_mock_admin
mock_job.status = JobStatus.AWAITING_REVIEW.value # Use .value for string comparison
mock_job.state_data = {
"blocking_state_entered_at": "2024-01-01T00:00:00",
"blocking_action_type": "lyrics",
"reminder_sent": True, # Already sent
}
with patch('backend.api.routes.internal.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
test_client = TestClient(test_app)
response = test_client.post(
"/api/internal/jobs/test-job-123/check-idle-reminder",
headers={"Authorization": "Bearer admin-token"}
)
# Should succeed with "already_sent" status
assert response.status_code == 200
data = response.json()
assert data["status"] == "already_sent"
assert data["job_id"] == "test-job-123"
-337
View File
@@ -1,337 +0,0 @@
"""
Unit tests for admin job files endpoint.
Tests the GET /api/admin/jobs/{job_id}/files endpoint that returns
all files for a job with signed download URLs.
"""
import pytest
from unittest.mock import Mock, patch
from fastapi.testclient import TestClient
from fastapi import FastAPI
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
from backend.api.dependencies import AuthResult, UserType
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Admin authenticated",
user_email="admin@example.com",
is_admin=True,
)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_job_with_files():
"""Create a mock job with comprehensive file_urls."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Song"
job.status = JobStatus.COMPLETE
job.file_urls = {
"input": "gs://bucket/jobs/test-job-123/input.flac",
"stems": {
"instrumental_clean": "gs://bucket/jobs/test-job-123/stems/instrumental_clean.flac",
"vocals": "gs://bucket/jobs/test-job-123/stems/vocals.flac",
},
"lyrics": {
"corrections": "gs://bucket/jobs/test-job-123/lyrics/corrections.json",
"lrc": "gs://bucket/jobs/test-job-123/lyrics/output.lrc",
},
"finals": {
"lossy_720p_mp4": "gs://bucket/jobs/test-job-123/finals/video_720p.mp4",
},
"youtube": {
"url": "https://youtube.com/watch?v=xyz789",
"video_id": "xyz789",
},
}
return job
@pytest.fixture
def mock_job_no_files():
"""Create a mock job with empty file_urls."""
job = Mock(spec=Job)
job.job_id = "test-job-empty"
job.user_email = "user@example.com"
job.artist = "Empty Artist"
job.title = "Empty Song"
job.status = JobStatus.PENDING
job.file_urls = {}
return job
@pytest.fixture
def mock_job_partial_files():
"""Create a mock job with only some files (partial processing)."""
job = Mock(spec=Job)
job.job_id = "test-job-partial"
job.user_email = "user@example.com"
job.artist = "Partial Artist"
job.title = "Partial Song"
job.status = JobStatus.SEPARATING_STAGE1
job.file_urls = {
"input": "gs://bucket/jobs/test-job-partial/input.flac",
"stems": {
"instrumental_clean": "gs://bucket/jobs/test-job-partial/stems/instrumental_clean.flac",
},
}
return job
class TestGetJobFiles:
"""Tests for GET /api/admin/jobs/{job_id}/files endpoint."""
def test_returns_files_with_signed_urls(self, client, mock_job_with_files):
"""Test that endpoint returns all files with signed URLs."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
# Setup JobManager mock
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
# Setup StorageService mock to return predictable signed URLs
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get(
"/api/admin/jobs/test-job-123/files",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["job_id"] == "test-job-123"
assert data["artist"] == "Test Artist"
assert data["title"] == "Test Song"
assert "files" in data
assert data["total_files"] > 0
def test_returns_correct_file_count(self, client, mock_job_with_files):
"""Test that file count matches actual GCS files (excluding non-GCS entries)."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-123/files")
data = response.json()
# Should have: input, stems.instrumental_clean, stems.vocals,
# lyrics.corrections, lyrics.lrc, finals.lossy_720p_mp4
# NOT youtube.url or youtube.video_id (not GCS paths)
assert data["total_files"] == 6
def test_handles_nested_file_structure(self, client, mock_job_with_files):
"""Test that nested file_urls structure is properly traversed."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-123/files")
data = response.json()
files = data["files"]
# Check that files from different categories are present
categories = {f["category"] for f in files}
assert "input" in categories or any(f["category"] == "" and f["file_key"] == "input" for f in files)
assert "stems" in categories
assert "lyrics" in categories
assert "finals" in categories
def test_file_info_structure(self, client, mock_job_with_files):
"""Test that each file has correct info structure."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-123/files")
data = response.json()
files = data["files"]
# Every file should have these fields
for file_info in files:
assert "name" in file_info
assert "path" in file_info
assert "download_url" in file_info
assert "category" in file_info
assert "file_key" in file_info
# download_url should be a signed URL
assert file_info["download_url"].startswith("https://signed.url/")
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.get(
"/api/admin/jobs/nonexistent-job/files",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_returns_empty_files_for_new_job(self, client, mock_job_no_files):
"""Test that new jobs with no files return empty list."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_no_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-empty/files")
assert response.status_code == 200
data = response.json()
assert data["job_id"] == "test-job-empty"
assert data["files"] == []
assert data["total_files"] == 0
def test_handles_partial_processing(self, client, mock_job_partial_files):
"""Test job with only some files processed."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_partial_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-partial/files")
assert response.status_code == 200
data = response.json()
# Should have input + stems.instrumental_clean = 2 files
assert data["total_files"] == 2
def test_skips_non_gcs_urls(self, client, mock_job_with_files):
"""Test that non-GCS URLs (like youtube URLs) are skipped."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.side_effect = lambda path, **kwargs: f"https://signed.url/{path}"
mock_storage_class.return_value = mock_storage
response = client.get("/api/admin/jobs/test-job-123/files")
data = response.json()
files = data["files"]
# Ensure no youtube URLs or video IDs are in the files
for file_info in files:
assert "youtube.com" not in file_info["path"]
assert file_info["path"].startswith("gs://")
def test_signed_url_expiration(self, client, mock_job_with_files):
"""Test that signed URLs are generated with appropriate expiration."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.StorageService') as mock_storage_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job_with_files
mock_jm_class.return_value = mock_jm
mock_storage = Mock()
mock_storage.generate_signed_url.return_value = "https://signed.url/test"
mock_storage_class.return_value = mock_storage
client.get("/api/admin/jobs/test-job-123/files")
# Verify signed URLs were requested with expiration (default 120 minutes)
assert mock_storage.generate_signed_url.called
# Check that expiration_minutes was passed (could be any reasonable value)
call_kwargs = mock_storage.generate_signed_url.call_args_list[0][1]
if "expiration_minutes" in call_kwargs:
assert call_kwargs["expiration_minutes"] >= 60
class TestGetJobFilesAuthorization:
"""Tests for authorization on the files endpoint."""
def test_requires_admin_access(self, client, mock_job_with_files):
"""Test that non-admin users cannot access the endpoint."""
# Reset the dependency override to test auth
original_override = app.dependency_overrides.get(require_admin)
def get_non_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = get_non_admin
try:
response = client.get(
"/api/admin/jobs/test-job-123/files",
headers={"Authorization": "Bearer user-token"}
)
assert response.status_code == 403
finally:
# Restore the original override
if original_override:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
-384
View File
@@ -1,384 +0,0 @@
"""
Unit tests for admin job reset endpoint.
Tests the POST /api/admin/jobs/{job_id}/reset endpoint that allows admins
to reset a job to a specific state for re-processing.
"""
import pytest
from unittest.mock import Mock, patch
from fastapi.testclient import TestClient
from fastapi import FastAPI
from datetime import datetime
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
from backend.api.dependencies import AuthResult, UserType
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Admin authenticated",
user_email="admin@example.com",
is_admin=True,
)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_job():
"""Create a mock job in COMPLETE status for testing."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Test Artist"
job.title = "Test Title"
job.status = JobStatus.COMPLETE
job.theme_id = "nomad"
job.file_urls = {
"input": "gs://bucket/input.flac",
"stems": {
"instrumental_clean": "gs://bucket/stems/clean.flac",
},
"lyrics": {
"corrections": "gs://bucket/lyrics/corrections.json",
},
}
job.state_data = {
"audio_search_results": [{"provider": "test"}],
"audio_selection": {"index": 0},
"review_complete": True,
"corrected_lyrics": {"lines": []},
"instrumental_selection": "clean",
"brand_code": "NOMAD-1234",
}
job.timeline = []
job.created_at = datetime(2026, 1, 9, 10, 0, 0)
job.updated_at = datetime(2026, 1, 9, 12, 0, 0)
return job
class TestResetJobToPending:
"""Tests for resetting job to PENDING state."""
def test_reset_to_pending_success(self, client, mock_job):
"""Test resetting a job to PENDING state."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["job_id"] == "test-job-123"
assert data["new_status"] == "pending"
assert "pending" in data["message"].lower()
def test_reset_to_pending_clears_state_data(self, client, mock_job):
"""Test that resetting to PENDING clears appropriate state_data keys."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
# Verify update_job was called with cleared state_data keys
call_args = mock_jm.update_job.call_args
update_data = call_args[0][1]
# Verify status was set
assert update_data["status"] == "pending"
class TestResetJobToAwaitingAudioSelection:
"""Tests for resetting job to AWAITING_AUDIO_SELECTION state."""
def test_reset_to_awaiting_audio_selection(self, client, mock_job):
"""Test resetting a job to AWAITING_AUDIO_SELECTION state."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_audio_selection"},
)
assert response.status_code == 200
data = response.json()
assert data["new_status"] == "awaiting_audio_selection"
class TestResetJobToAwaitingReview:
"""Tests for resetting job to AWAITING_REVIEW state."""
def test_reset_to_awaiting_review(self, client, mock_job):
"""Test resetting a job to AWAITING_REVIEW state."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_review"},
)
assert response.status_code == 200
data = response.json()
assert data["new_status"] == "awaiting_review"
class TestResetJobToAwaitingInstrumentalSelection:
"""Tests for resetting job to AWAITING_INSTRUMENTAL_SELECTION state."""
def test_reset_to_awaiting_instrumental_selection(self, client, mock_job):
"""Test resetting a job to AWAITING_INSTRUMENTAL_SELECTION state."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_instrumental_selection"},
)
assert response.status_code == 200
data = response.json()
assert data["new_status"] == "awaiting_instrumental_selection"
class TestResetJobValidation:
"""Tests for reset endpoint validation."""
def test_rejects_invalid_target_state(self, client, mock_job):
"""Test that invalid target states are rejected."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "encoding"},
)
assert response.status_code == 400
assert "invalid" in response.json()["detail"].lower() or "not allowed" in response.json()["detail"].lower()
def test_rejects_complete_as_target(self, client, mock_job):
"""Test that COMPLETE is not a valid reset target."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "complete"},
)
assert response.status_code == 400
def test_rejects_failed_as_target(self, client, mock_job):
"""Test that FAILED is not a valid reset target."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "failed"},
)
assert response.status_code == 400
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/nonexistent-job/reset",
json={"target_state": "pending"},
)
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_rejects_missing_target_state(self, client, mock_job):
"""Test that target_state is required."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={},
)
assert response.status_code == 422 # Validation error
class TestResetJobLogging:
"""Tests for logging on the reset endpoint."""
def test_logs_admin_reset_action(self, client, mock_job):
"""Test that admin reset actions are logged."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.logger') as mock_logger:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
# Verify logging was called
mock_logger.info.assert_called()
log_message = mock_logger.info.call_args[0][0]
assert "admin" in log_message.lower() or "reset" in log_message.lower()
class TestResetJobTimeline:
"""Tests for timeline event on reset."""
def test_adds_timeline_event_on_reset(self, client, mock_job):
"""Test that a timeline event is added when job is reset."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_review"},
)
assert response.status_code == 200
data = response.json()
# Timeline event should be included in response
assert "timeline_event" in data or data.get("message")
class TestResetJobAuthorization:
"""Tests for authorization on the reset endpoint."""
def test_requires_admin_access(self, client, mock_job):
"""Test that non-admin users cannot access the endpoint."""
original_override = app.dependency_overrides.get(require_admin)
def get_non_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = get_non_admin
try:
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
headers={"Authorization": "Bearer user-token"}
)
assert response.status_code == 403
finally:
if original_override:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
class TestResetJobClearsStateData:
"""Tests for state_data clearing based on target state."""
def test_reset_to_pending_clears_all_processing_data(self, client, mock_job):
"""Test that resetting to PENDING clears audio search results."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "pending"},
)
assert response.status_code == 200
# The cleared_data field should list what was cleared
data = response.json()
assert "cleared_data" in data or "state_data" in str(data)
def test_reset_to_awaiting_review_preserves_audio_data(self, client, mock_job):
"""Test that resetting to AWAITING_REVIEW preserves audio stems."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_review"},
)
assert response.status_code == 200
def test_reset_to_instrumental_preserves_review_data(self, client, mock_job):
"""Test that resetting to AWAITING_INSTRUMENTAL_SELECTION preserves review data."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.post(
"/api/admin/jobs/test-job-123/reset",
json={"target_state": "awaiting_instrumental_selection"},
)
assert response.status_code == 200
-326
View File
@@ -1,326 +0,0 @@
"""
Unit tests for admin job update endpoint.
Tests the PATCH /api/admin/jobs/{job_id} endpoint that allows admins
to update editable job fields.
"""
import pytest
from unittest.mock import Mock, patch
from fastapi.testclient import TestClient
from fastapi import FastAPI
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.models.job import Job, JobStatus
# Create a test app with the admin router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
from backend.api.dependencies import AuthResult, UserType
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=999,
message="Admin authenticated",
user_email="admin@example.com",
is_admin=True,
)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_job():
"""Create a mock job for testing."""
job = Mock(spec=Job)
job.job_id = "test-job-123"
job.user_email = "user@example.com"
job.artist = "Original Artist"
job.title = "Original Title"
job.status = JobStatus.AWAITING_REVIEW
job.theme_id = "nomad"
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = True
job.customer_email = None
job.customer_notes = None
job.brand_prefix = None
job.non_interactive = False
job.prep_only = False
job.discord_webhook_url = None
job.youtube_description = None
job.youtube_description_template = None
job.file_urls = {}
job.state_data = {}
job.created_at = "2026-01-09T10:00:00Z"
job.updated_at = "2026-01-09T10:30:00Z"
return job
class TestUpdateJob:
"""Tests for PATCH /api/admin/jobs/{job_id} endpoint."""
def test_update_artist_field(self, client, mock_job):
"""Test updating the artist field."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"artist": "New Artist"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["job_id"] == "test-job-123"
assert "artist" in data["updated_fields"]
# Verify update_job was called with correct args
mock_jm.update_job.assert_called_once()
call_args = mock_jm.update_job.call_args
assert call_args[0][0] == "test-job-123"
assert call_args[0][1]["artist"] == "New Artist"
def test_update_multiple_fields(self, client, mock_job):
"""Test updating multiple fields at once."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={
"artist": "New Artist",
"title": "New Title",
"theme_id": "default",
"enable_cdg": True,
},
)
assert response.status_code == 200
data = response.json()
assert len(data["updated_fields"]) == 4
assert "artist" in data["updated_fields"]
assert "title" in data["updated_fields"]
assert "theme_id" in data["updated_fields"]
assert "enable_cdg" in data["updated_fields"]
def test_update_user_email(self, client, mock_job):
"""Test updating user_email field."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"user_email": "newuser@example.com"},
)
assert response.status_code == 200
call_args = mock_jm.update_job.call_args
assert call_args[0][1]["user_email"] == "newuser@example.com"
def test_update_boolean_fields(self, client, mock_job):
"""Test updating boolean fields."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={
"enable_cdg": True,
"enable_txt": True,
"enable_youtube_upload": False,
"non_interactive": True,
"prep_only": True,
},
)
assert response.status_code == 200
call_args = mock_jm.update_job.call_args
assert call_args[0][1]["enable_cdg"] is True
assert call_args[0][1]["enable_txt"] is True
assert call_args[0][1]["enable_youtube_upload"] is False
def test_rejects_non_editable_job_id(self, client, mock_job):
"""Test that job_id cannot be updated."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"job_id": "new-job-id"},
)
assert response.status_code == 400
assert "not editable" in response.json()["detail"].lower()
def test_rejects_non_editable_created_at(self, client, mock_job):
"""Test that created_at cannot be updated."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"created_at": "2025-01-01T00:00:00Z"},
)
assert response.status_code == 400
assert "not editable" in response.json()["detail"].lower()
def test_rejects_status_update(self, client, mock_job):
"""Test that status cannot be updated via PATCH (use reset endpoint)."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"status": "complete"},
)
assert response.status_code == 400
assert "not editable" in response.json()["detail"].lower()
def test_rejects_state_data_update(self, client, mock_job):
"""Test that state_data cannot be updated directly."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"state_data": {"foo": "bar"}},
)
assert response.status_code == 400
assert "not editable" in response.json()["detail"].lower()
def test_returns_404_when_job_not_found(self, client):
"""Test 404 when job doesn't exist."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = None
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/nonexistent-job",
json={"artist": "New Artist"},
)
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_returns_400_for_empty_update(self, client, mock_job):
"""Test 400 when no valid fields are provided."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={},
)
assert response.status_code == 400
assert "no valid" in response.json()["detail"].lower()
def test_logs_admin_changes(self, client, mock_job):
"""Test that admin changes are logged."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class, \
patch('backend.api.routes.admin.logger') as mock_logger:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"artist": "New Artist"},
)
assert response.status_code == 200
# Verify logging was called
mock_logger.info.assert_called()
log_message = mock_logger.info.call_args[0][0]
assert "admin@example.com" in log_message.lower() or "admin" in log_message.lower()
def test_update_customer_fields(self, client, mock_job):
"""Test updating made-for-you customer fields."""
with patch('backend.api.routes.admin.JobManager') as mock_jm_class:
mock_jm = Mock()
mock_jm.get_job.return_value = mock_job
mock_jm.update_job.return_value = True
mock_jm_class.return_value = mock_jm
response = client.patch(
"/api/admin/jobs/test-job-123",
json={
"customer_email": "customer@example.com",
"customer_notes": "Special request: add extra countdown",
},
)
assert response.status_code == 200
call_args = mock_jm.update_job.call_args
assert call_args[0][1]["customer_email"] == "customer@example.com"
assert call_args[0][1]["customer_notes"] == "Special request: add extra countdown"
class TestUpdateJobAuthorization:
"""Tests for authorization on the update endpoint."""
def test_requires_admin_access(self, client, mock_job):
"""Test that non-admin users cannot access the endpoint."""
original_override = app.dependency_overrides.get(require_admin)
def get_non_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = get_non_admin
try:
response = client.patch(
"/api/admin/jobs/test-job-123",
json={"artist": "New Artist"},
headers={"Authorization": "Bearer user-token"}
)
assert response.status_code == 403
finally:
if original_override:
app.dependency_overrides[require_admin] = original_override
else:
app.dependency_overrides[require_admin] = get_mock_admin
+8 -12
View File
@@ -845,8 +845,7 @@ class TestAudioSearchThemeSupport:
This is the key behavior: selecting a theme should automatically
enable CDG and TXT output formats.
"""
from backend.api.routes.audio_search import AudioSearchRequest
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# When theme_id is set, enable_cdg/enable_txt should default to True
request = AudioSearchRequest(
@@ -856,7 +855,7 @@ class TestAudioSearchThemeSupport:
# enable_cdg and enable_txt are None (not specified)
)
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
@@ -865,8 +864,7 @@ class TestAudioSearchThemeSupport:
def test_audio_search_request_no_theme_no_cdg_txt(self):
"""Test that without theme_id, CDG/TXT defaults to disabled."""
from backend.api.routes.audio_search import AudioSearchRequest
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
request = AudioSearchRequest(
artist="Test Artist",
@@ -874,7 +872,7 @@ class TestAudioSearchThemeSupport:
# No theme_id, no enable_cdg, no enable_txt
)
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
@@ -883,8 +881,7 @@ class TestAudioSearchThemeSupport:
def test_explicit_cdg_txt_overrides_theme_default(self):
"""Test that explicit enable_cdg/enable_txt values override theme defaults."""
from backend.api.routes.audio_search import AudioSearchRequest
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# Theme set (would default to True), but explicitly disabled
request = AudioSearchRequest(
@@ -895,7 +892,7 @@ class TestAudioSearchThemeSupport:
enable_txt=False,
)
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
@@ -904,8 +901,7 @@ class TestAudioSearchThemeSupport:
def test_explicit_cdg_txt_enables_without_theme(self):
"""Test that explicit True enables CDG/TXT even without theme."""
from backend.api.routes.audio_search import AudioSearchRequest
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.api.routes.audio_search import AudioSearchRequest, _resolve_cdg_txt_defaults
# No theme (would default to False), but explicitly enabled
request = AudioSearchRequest(
@@ -915,7 +911,7 @@ class TestAudioSearchThemeSupport:
enable_txt=True,
)
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
resolved_cdg, resolved_txt = _resolve_cdg_txt_defaults(
request.theme_id, request.enable_cdg, request.enable_txt
)
-725
View File
@@ -1,725 +0,0 @@
"""
Unit tests for email service.
Tests the new job completion and action reminder email methods,
as well as CC support.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from backend.services.email_service import (
EmailService,
EmailProvider,
ConsoleEmailProvider,
SendGridEmailProvider,
get_email_service,
)
class TestConsoleEmailProvider:
"""Tests for console email provider."""
def test_send_email_basic(self):
"""Test basic email logging."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test content</p>",
)
assert result is True
def test_send_email_with_text_content(self):
"""Test email with text content."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
text_content="Test plain text",
)
assert result is True
def test_send_email_with_cc(self):
"""Test email with CC recipients."""
provider = ConsoleEmailProvider()
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
cc_emails=["cc1@example.com", "cc2@example.com"],
)
assert result is True
class TestSendGridEmailProvider:
"""Tests for SendGrid email provider."""
def test_send_email_success(self):
"""Test successful email sending via SendGrid."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
from_name="Test Sender",
)
mock_response = Mock()
mock_response.status_code = 202
# Patch at the sendgrid module level since it's imported inline
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is True
mock_client.send.assert_called_once()
def test_send_email_with_cc(self):
"""Test email sending with CC via SendGrid."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 202
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
cc_emails=["cc@example.com"],
)
assert result is True
def test_send_email_failure_status(self):
"""Test email sending failure due to bad status."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 400
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is False
def test_send_email_exception(self):
"""Test email sending exception handling."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_sg.return_value.send.side_effect = Exception("API error")
result = provider.send_email(
to_email="user@example.com",
subject="Test Subject",
html_content="<p>Test</p>",
)
assert result is False
class TestEmailServiceJobCompletion:
"""Tests for job completion email method."""
def test_send_job_completion_basic(self):
"""Test basic job completion email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
)
assert result is True
service.provider.send_email.assert_called_once()
def test_send_job_completion_with_song_info(self):
"""Test job completion email includes song in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
artist="Test Artist",
title="Test Song",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
assert "Test Artist" in subject
assert "Test Song" in subject
def test_send_job_completion_with_brand_code(self):
"""Test job completion email includes brand code in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
artist="Seether",
title="Tonight",
brand_code="NOMAD-1178",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
# Subject format: "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
assert subject == "NOMAD-1178: Seether - Tonight (Your karaoke video is ready!)"
def test_send_job_completion_default_subject(self):
"""Test job completion email default subject without song info."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
)
call_args = service.provider.send_email.call_args
subject = call_args.kwargs.get('subject') or call_args[0][1]
assert "karaoke video is ready" in subject.lower()
def test_send_job_completion_with_cc(self):
"""Test job completion email with CC to admin."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
cc_admin=True,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert "gen@nomadkaraoke.com" in call_kwargs.get('cc_emails', [])
def test_send_job_completion_without_cc(self):
"""Test job completion email without CC."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Your video is ready!",
cc_admin=False,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert call_kwargs.get('cc_emails') is None
def test_send_job_completion_escapes_html(self):
"""Test that message content is HTML-escaped."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_job_completion(
to_email="user@example.com",
message_content="Test <script>alert('xss')</script>",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "<script>" not in html_content
assert "&lt;script&gt;" in html_content
def test_send_job_completion_includes_plain_text(self):
"""Test that plain text content is included."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
message = "Your video is ready!"
service.send_job_completion(
to_email="user@example.com",
message_content=message,
)
call_kwargs = service.provider.send_email.call_args.kwargs
assert call_kwargs.get('text_content') == message
class TestEmailServiceActionReminder:
"""Tests for action reminder email method."""
def test_send_action_reminder_lyrics(self):
"""Test lyrics action reminder email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please review your lyrics",
action_type="lyrics",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "lyrics" in subject.lower()
def test_send_action_reminder_instrumental(self):
"""Test instrumental action reminder email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please select instrumental",
action_type="instrumental",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "instrumental" in subject.lower()
def test_send_action_reminder_with_song_info(self):
"""Test action reminder includes song in subject."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Please review",
action_type="lyrics",
artist="Test Artist",
title="Test Song",
)
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "Test Artist" in subject
assert "Test Song" in subject
def test_send_action_reminder_unknown_type(self):
"""Test action reminder with unknown type still sends."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_action_reminder(
to_email="user@example.com",
message_content="Please take action",
action_type="unknown",
)
assert result is True
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "Action needed" in subject
def test_send_action_reminder_escapes_html(self):
"""Test that message content is HTML-escaped."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Test <script>alert('xss')</script>",
action_type="lyrics",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "<script>" not in html_content
assert "&lt;script&gt;" in html_content
def test_send_action_reminder_no_cc(self):
"""Test that action reminders don't have CC."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_action_reminder(
to_email="user@example.com",
message_content="Please review",
action_type="lyrics",
)
call_kwargs = service.provider.send_email.call_args.kwargs
# Action reminders should not have CC
assert call_kwargs.get('cc_emails') is None
class TestEmailServiceConfiguration:
"""Tests for email service configuration."""
def test_uses_sendgrid_when_configured(self):
"""Test that SendGrid is used when API key is set."""
with patch.dict('os.environ', {'SENDGRID_API_KEY': 'test-key'}):
service = EmailService()
assert isinstance(service.provider, SendGridEmailProvider)
def test_uses_console_when_not_configured(self):
"""Test that console is used when no API key."""
with patch.dict('os.environ', {}, clear=True):
# Remove SENDGRID_API_KEY if it exists
import os
original = os.environ.pop('SENDGRID_API_KEY', None)
try:
service = EmailService()
assert isinstance(service.provider, ConsoleEmailProvider)
finally:
if original:
os.environ['SENDGRID_API_KEY'] = original
def test_is_configured_true_for_sendgrid(self):
"""Test is_configured returns True for SendGrid."""
service = EmailService()
service.provider = SendGridEmailProvider("key", "from@example.com")
assert service.is_configured() is True
def test_is_configured_false_for_console(self):
"""Test is_configured returns False for console."""
service = EmailService()
service.provider = ConsoleEmailProvider()
assert service.is_configured() is False
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_email_service_returns_same_instance(self):
"""Test that get_email_service returns singleton."""
# Reset global
import backend.services.email_service as es
es._email_service = None
service1 = get_email_service()
service2 = get_email_service()
assert service1 is service2
class TestCCFunctionality:
"""Integration tests for CC functionality across providers."""
def test_console_provider_logs_cc(self, caplog):
"""Test that console provider logs CC recipients."""
import logging
caplog.set_level(logging.INFO)
provider = ConsoleEmailProvider()
provider.send_email(
to_email="user@example.com",
subject="Test",
html_content="<p>Test</p>",
cc_emails=["cc1@example.com", "cc2@example.com"],
)
# Check that CC was logged
assert "cc1@example.com" in caplog.text or "CC:" in caplog.text
def test_sendgrid_provider_adds_cc_recipients(self):
"""Test that SendGrid provider adds CC recipients to message."""
provider = SendGridEmailProvider(
api_key="test-api-key",
from_email="from@example.com",
)
mock_response = Mock()
mock_response.status_code = 202
# Patch at sendgrid module level since imports are inline
with patch('sendgrid.SendGridAPIClient') as mock_sg:
mock_client = Mock()
mock_client.send.return_value = mock_response
mock_sg.return_value = mock_client
# We need to capture the Mail object
with patch('sendgrid.helpers.mail.Mail') as mock_mail_class:
mock_mail = MagicMock()
mock_mail_class.return_value = mock_mail
provider.send_email(
to_email="user@example.com",
subject="Test",
html_content="<p>Test</p>",
cc_emails=["cc@example.com"],
)
# Verify add_cc was called
mock_mail.add_cc.assert_called()
class TestMadeForYouOrderConfirmation:
"""Tests for send_made_for_you_order_confirmation method."""
def test_send_order_confirmation_success(self):
"""Test successful order confirmation email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_made_for_you_order_confirmation(
to_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="test-job-123",
)
assert result is True
service.provider.send_email.assert_called_once()
# Verify BCC is set
call_kwargs = service.provider.send_email.call_args.kwargs
assert call_kwargs.get('bcc_emails') == ["done@nomadkaraoke.com"]
def test_send_order_confirmation_subject_format(self):
"""Test order confirmation has correct subject format."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_order_confirmation(
to_email="customer@example.com",
artist="Seether",
title="Tonight",
job_id="test-job-456",
)
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
assert "Order Confirmed" in subject
assert "Seether" in subject
assert "Tonight" in subject
def test_send_order_confirmation_with_notes(self):
"""Test order confirmation includes notes."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_order_confirmation(
to_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="test-job-789",
notes="Wedding anniversary!",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "Wedding anniversary!" in html_content
def test_send_order_confirmation_includes_delivery_promise(self):
"""Test order confirmation mentions delivery timeframe."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_order_confirmation(
to_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="test-job-abc",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content', '')
text_content = call_kwargs.get('text_content', '')
# Should mention delivery timeline
assert "24" in html_content or "24" in text_content
def test_send_order_confirmation_failure(self):
"""Test handling of email send failure."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = False
result = service.send_made_for_you_order_confirmation(
to_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="test-job-def",
)
assert result is False
class TestMadeForYouAdminNotification:
"""Tests for send_made_for_you_admin_notification method."""
def test_send_admin_notification_success(self):
"""Test successful admin notification email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
result = service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="job-123",
admin_login_token="test-token-abc",
)
assert result is True
service.provider.send_email.assert_called_once()
def test_send_admin_notification_subject_format(self):
"""Test admin notification has correct subject format."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Seether",
title="Tonight",
job_id="job-123",
admin_login_token="test-token-abc",
)
call_kwargs = service.provider.send_email.call_args.kwargs
subject = call_kwargs.get('subject')
# Subject format: "Karaoke Order: {artist} - {title} [ID: {job_id}]"
assert "Karaoke Order" in subject
assert "Seether" in subject
assert "Tonight" in subject
def test_send_admin_notification_includes_customer_email(self):
"""Test admin notification includes customer email."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="vip@example.com",
artist="Test Artist",
title="Test Song",
job_id="job-123",
admin_login_token="test-token-abc",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "vip@example.com" in html_content
def test_send_admin_notification_includes_job_link(self):
"""Test admin notification includes job link."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="abc-def-123",
admin_login_token="test-token-abc",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "abc-def-123" in html_content
def test_send_admin_notification_includes_audio_count(self):
"""Test admin notification includes audio source count."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="job-123",
admin_login_token="test-token-abc",
audio_source_count=5,
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "5" in html_content
def test_send_admin_notification_includes_notes(self):
"""Test admin notification includes customer notes."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = True
service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="job-123",
admin_login_token="test-token-abc",
notes="Rush order please!",
)
call_kwargs = service.provider.send_email.call_args.kwargs
html_content = call_kwargs.get('html_content')
assert "Rush order please!" in html_content
def test_send_admin_notification_failure(self):
"""Test handling of email send failure."""
service = EmailService()
service.provider = Mock()
service.provider.send_email.return_value = False
result = service.send_made_for_you_admin_notification(
to_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
artist="Test Artist",
title="Test Song",
job_id="job-123",
admin_login_token="test-token-abc",
)
assert result is False
@@ -1,298 +0,0 @@
"""
Unit tests for EmailValidationService.
Tests email normalization, disposable domain detection, and blocklist management.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
# Mock Google Cloud before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.storage'] = MagicMock()
class TestEmailNormalization:
"""Test email normalization logic."""
@pytest.fixture
def mock_db(self):
"""Create a mock Firestore client."""
mock = MagicMock()
# Default: empty blocklist config
mock_doc = Mock()
mock_doc.exists = False
mock.collection.return_value.document.return_value.get.return_value = mock_doc
return mock
@pytest.fixture
def email_service(self, mock_db):
"""Create EmailValidationService instance with mocks."""
from backend.services.email_validation_service import EmailValidationService
service = EmailValidationService(db=mock_db)
return service
# =========================================================================
# Gmail Normalization Tests
# =========================================================================
def test_normalize_gmail_removes_dots(self, email_service):
"""Test that dots are removed from Gmail local part."""
result = email_service.normalize_email("j.o.h.n@gmail.com")
assert result == "john@gmail.com"
def test_normalize_gmail_removes_plus_suffix(self, email_service):
"""Test that +tag is removed from Gmail local part."""
result = email_service.normalize_email("john+spam@gmail.com")
assert result == "john@gmail.com"
def test_normalize_gmail_removes_dots_and_plus(self, email_service):
"""Test both dots and +tag are removed from Gmail."""
result = email_service.normalize_email("j.o.h.n+newsletter@gmail.com")
assert result == "john@gmail.com"
def test_normalize_googlemail_treated_like_gmail(self, email_service):
"""Test that googlemail.com is normalized like gmail.com."""
result = email_service.normalize_email("j.o.h.n+test@googlemail.com")
assert result == "john@googlemail.com"
def test_normalize_gmail_to_lowercase(self, email_service):
"""Test Gmail addresses are lowercased."""
result = email_service.normalize_email("JOHN.DOE+Test@GMAIL.COM")
assert result == "johndoe@gmail.com"
# =========================================================================
# Non-Gmail Normalization Tests
# =========================================================================
def test_normalize_non_gmail_preserves_dots(self, email_service):
"""Test dots are preserved for non-Gmail domains."""
result = email_service.normalize_email("j.o.h.n@example.com")
assert result == "j.o.h.n@example.com"
def test_normalize_non_gmail_preserves_plus(self, email_service):
"""Test +tag is preserved for non-Gmail domains."""
result = email_service.normalize_email("john+tag@company.com")
assert result == "john+tag@company.com"
def test_normalize_non_gmail_to_lowercase(self, email_service):
"""Test non-Gmail addresses are lowercased."""
result = email_service.normalize_email("John.Doe@Example.Com")
assert result == "john.doe@example.com"
def test_normalize_strips_whitespace(self, email_service):
"""Test whitespace is stripped."""
result = email_service.normalize_email(" john@example.com ")
assert result == "john@example.com"
# =========================================================================
# Edge Cases
# =========================================================================
def test_normalize_empty_string(self, email_service):
"""Test empty string returns empty."""
result = email_service.normalize_email("")
assert result == ""
def test_normalize_none_returns_empty(self, email_service):
"""Test None returns empty string."""
result = email_service.normalize_email(None)
assert result == ""
def test_normalize_invalid_no_at_sign(self, email_service):
"""Test email without @ is returned as-is (lowercase)."""
result = email_service.normalize_email("notanemail")
assert result == "notanemail"
class TestDisposableDomainDetection:
"""Test disposable domain detection."""
@pytest.fixture
def mock_db(self):
"""Create a mock Firestore client."""
mock = MagicMock()
mock_doc = Mock()
mock_doc.exists = False
mock.collection.return_value.document.return_value.get.return_value = mock_doc
return mock
@pytest.fixture
def email_service(self, mock_db):
"""Create EmailValidationService instance with mocks."""
from backend.services.email_validation_service import EmailValidationService
# Clear any cached blocklist
EmailValidationService._blocklist_cache = None
EmailValidationService._blocklist_cache_time = None
service = EmailValidationService(db=mock_db)
return service
def test_detect_known_disposable_domain(self, email_service):
"""Test known disposable domains are detected."""
# These are in DEFAULT_DISPOSABLE_DOMAINS
assert email_service.is_disposable_domain("user@tempmail.com") is True
assert email_service.is_disposable_domain("user@mailinator.com") is True
assert email_service.is_disposable_domain("user@guerrillamail.com") is True
def test_detect_legitimate_domain(self, email_service):
"""Test legitimate domains are not flagged."""
assert email_service.is_disposable_domain("user@gmail.com") is False
assert email_service.is_disposable_domain("user@yahoo.com") is False
assert email_service.is_disposable_domain("user@company.com") is False
def test_detect_case_insensitive(self, email_service):
"""Test domain detection is case-insensitive."""
assert email_service.is_disposable_domain("user@TEMPMAIL.COM") is True
assert email_service.is_disposable_domain("user@TempMail.Com") is True
def test_detect_invalid_email_returns_false(self, email_service):
"""Test invalid email returns False."""
assert email_service.is_disposable_domain("notanemail") is False
assert email_service.is_disposable_domain("") is False
class TestBlocklistManagement:
"""Test blocklist management functionality."""
@pytest.fixture
def mock_db(self):
"""Create a mock Firestore client with blocklist."""
mock = MagicMock()
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"disposable_domains": ["custom-temp.com"],
"blocked_emails": ["spammer@example.com"],
"blocked_ips": ["192.168.1.100"],
}
mock.collection.return_value.document.return_value.get.return_value = mock_doc
return mock
@pytest.fixture
def email_service(self, mock_db):
"""Create EmailValidationService instance with mocks."""
from backend.services.email_validation_service import EmailValidationService
# Clear any cached blocklist
EmailValidationService._blocklist_cache = None
EmailValidationService._blocklist_cache_time = None
service = EmailValidationService(db=mock_db)
return service
def test_is_email_blocked(self, email_service):
"""Test blocked email detection."""
assert email_service.is_email_blocked("spammer@example.com") is True
assert email_service.is_email_blocked("legitimate@example.com") is False
def test_is_email_blocked_case_insensitive(self, email_service):
"""Test blocked email detection is case-insensitive."""
assert email_service.is_email_blocked("SPAMMER@example.com") is True
assert email_service.is_email_blocked("Spammer@Example.Com") is True
def test_is_ip_blocked(self, email_service):
"""Test blocked IP detection."""
assert email_service.is_ip_blocked("192.168.1.100") is True
assert email_service.is_ip_blocked("10.0.0.1") is False
def test_custom_disposable_domain_added(self, email_service):
"""Test custom disposable domains from Firestore are included."""
assert email_service.is_disposable_domain("user@custom-temp.com") is True
def test_blocklist_caching(self, email_service, mock_db):
"""Test blocklist is cached."""
# First call
email_service.is_disposable_domain("user@test.com")
# Second call should use cache
email_service.is_disposable_domain("user@test2.com")
# Should only call Firestore once due to caching
assert mock_db.collection.return_value.document.return_value.get.call_count == 1
class TestBetaEnrollmentValidation:
"""Test beta enrollment validation."""
@pytest.fixture
def mock_db(self):
"""Create a mock Firestore client."""
mock = MagicMock()
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"disposable_domains": [],
"blocked_emails": ["blocked@example.com"],
"blocked_ips": [],
}
mock.collection.return_value.document.return_value.get.return_value = mock_doc
return mock
@pytest.fixture
def email_service(self, mock_db):
"""Create EmailValidationService instance with mocks."""
from backend.services.email_validation_service import EmailValidationService
EmailValidationService._blocklist_cache = None
EmailValidationService._blocklist_cache_time = None
service = EmailValidationService(db=mock_db)
return service
def test_validate_legitimate_email(self, email_service):
"""Test legitimate email passes validation."""
is_valid, error = email_service.validate_email_for_beta("user@gmail.com")
assert is_valid is True
assert error == ""
def test_validate_disposable_email_rejected(self, email_service):
"""Test disposable email is rejected."""
is_valid, error = email_service.validate_email_for_beta("user@tempmail.com")
assert is_valid is False
assert "Disposable email" in error
def test_validate_blocked_email_rejected(self, email_service):
"""Test blocked email is rejected."""
is_valid, error = email_service.validate_email_for_beta("blocked@example.com")
assert is_valid is False
assert "not allowed" in error
def test_validate_invalid_format_rejected(self, email_service):
"""Test invalid email format is rejected."""
is_valid, error = email_service.validate_email_for_beta("notanemail")
assert is_valid is False
assert "Invalid email format" in error
def test_validate_empty_email_rejected(self, email_service):
"""Test empty email is rejected."""
is_valid, error = email_service.validate_email_for_beta("")
assert is_valid is False
class TestIPHashing:
"""Test IP address hashing."""
@pytest.fixture
def email_service(self):
"""Create EmailValidationService instance with mocks."""
mock_db = MagicMock()
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
from backend.services.email_validation_service import EmailValidationService
return EmailValidationService(db=mock_db)
def test_hash_ip_consistent(self, email_service):
"""Test IP hashing produces consistent results."""
hash1 = email_service.hash_ip("192.168.1.1")
hash2 = email_service.hash_ip("192.168.1.1")
assert hash1 == hash2
def test_hash_ip_different_for_different_ips(self, email_service):
"""Test different IPs produce different hashes."""
hash1 = email_service.hash_ip("192.168.1.1")
hash2 = email_service.hash_ip("192.168.1.2")
assert hash1 != hash2
def test_hash_ip_is_sha256(self, email_service):
"""Test IP hash is SHA-256 (64 hex chars)."""
hash_result = email_service.hash_ip("192.168.1.1")
assert len(hash_result) == 64
assert all(c in "0123456789abcdef" for c in hash_result)
+7 -10
View File
@@ -1128,8 +1128,7 @@ class TestFinaliseOnlyModels:
assert request.enable_txt is None
assert request.brand_prefix is None
assert request.keep_brand_code is None
# YouTube upload default is None (server applies default_enable_youtube_upload)
assert request.enable_youtube_upload is None
assert request.enable_youtube_upload is False
assert request.dropbox_path is None
assert request.gdrive_folder_id is None
@@ -1681,19 +1680,17 @@ class TestUploadEndpointThemeSupport:
)
def test_upload_endpoint_uses_resolve_cdg_txt_defaults(self):
"""Verify the upload endpoint uses resolve_cdg_txt_defaults for theme-based defaults."""
import inspect
"""Verify the upload endpoint uses _resolve_cdg_txt_defaults for theme-based defaults."""
from backend.api.routes import file_upload as file_upload_module
source_code = inspect.getsource(file_upload_module)
with open(file_upload_module.__file__, 'r') as f:
source_code = f.read()
# Check for the centralized resolve_cdg_txt_defaults function (imported from job_defaults_service)
has_resolve_call = 'resolve_cdg_txt_defaults(' in source_code
has_resolve_call = '_resolve_cdg_txt_defaults(' in source_code
assert has_resolve_call, (
"file_upload.py does not call resolve_cdg_txt_defaults(). "
"Theme-driven CDG/TXT defaults (controlled by DEFAULT_ENABLE_CDG/DEFAULT_ENABLE_TXT "
"settings) must be applied via the centralized job_defaults_service."
"file_upload.py does not call _resolve_cdg_txt_defaults(). "
"When theme_id is set, enable_cdg and enable_txt should default to True."
)
def test_upload_endpoint_has_optional_cdg_txt_params(self):
-229
View File
@@ -1,229 +0,0 @@
"""
Tests for GCE Encoding Worker (gce_encoding/main.py).
These tests verify the file-finding logic in the GCE encoding worker,
particularly ensuring that instrumental selection is respected.
NOTE: This test file is self-contained and doesn't import the full app
to avoid dependency issues in CI/CD environments.
"""
import pytest
from pathlib import Path
# Copy the find_file function here to test it in isolation
# This avoids importing the full gce_encoding module which has GCS dependencies
def find_file(work_dir: Path, *patterns):
'''Find a file matching any of the given glob patterns.'''
for pattern in patterns:
matches = list(work_dir.glob(f"**/{pattern}"))
if matches:
return matches[0]
return None
class TestFindFile:
"""Test the find_file function used to locate input files."""
def test_find_file_single_match(self, tmp_path):
"""Test finding a file with a single match."""
# Create test file
test_file = tmp_path / "test_instrumental.flac"
test_file.touch()
result = find_file(tmp_path, "*instrumental*.flac")
assert result is not None
assert result.name == "test_instrumental.flac"
def test_find_file_no_match(self, tmp_path):
"""Test finding a file with no matches returns None."""
result = find_file(tmp_path, "*nonexistent*.flac")
assert result is None
def test_find_file_priority_order(self, tmp_path):
"""Test that find_file returns first matching pattern."""
# Create multiple files that match different patterns
first = tmp_path / "first_pattern.flac"
second = tmp_path / "second_pattern.flac"
first.touch()
second.touch()
# First pattern should win
result = find_file(tmp_path, "*first*.flac", "*second*.flac")
assert result.name == "first_pattern.flac"
class TestInstrumentalSelection:
"""Test that instrumental selection is properly respected."""
def test_clean_instrumental_prioritized_when_selected(self, tmp_path):
"""Test that clean instrumental is found when 'clean' is selected.
When both instrumentals exist and 'clean' is selected,
the clean version should be found.
"""
# Create both instrumental files (as they exist in GCS)
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
clean.touch()
backing.touch()
# Simulate finding instrumental with 'clean' selection
# (mimics the logic in run_encoding when instrumental_selection == 'clean')
result = find_file(
tmp_path,
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
assert result is not None
assert "Clean" in result.name
def test_backing_instrumental_prioritized_when_selected(self, tmp_path):
"""Test that backing instrumental is found when 'with_backing' is selected.
When both instrumentals exist and 'with_backing' is selected,
the backing version should be found.
This test would have caught the bug where the GCE worker always
used the clean instrumental regardless of user selection.
"""
# Create both instrumental files (as they exist in GCS)
clean = tmp_path / "Artist - Title (Instrumental Clean).flac"
backing = tmp_path / "Artist - Title (Instrumental Backing).flac"
clean.touch()
backing.touch()
# Simulate finding instrumental with 'with_backing' selection
# (mimics the logic in run_encoding when instrumental_selection == 'with_backing')
result = find_file(
tmp_path,
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
"*with_backing*.flac", "*Backing*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
assert result is not None
assert "Backing" in result.name
def test_backing_instrumental_with_stem_naming(self, tmp_path):
"""Test finding backing instrumental with GCS stem naming convention.
In GCS, stems are named like 'stems/instrumental_with_backing.flac'
"""
stems_dir = tmp_path / "stems"
stems_dir.mkdir()
clean = stems_dir / "instrumental_clean.flac"
backing = stems_dir / "instrumental_with_backing.flac"
clean.touch()
backing.touch()
# Simulate finding instrumental with 'with_backing' selection
result = find_file(
tmp_path,
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
"*with_backing*.flac", "*Backing*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
assert result is not None
assert "with_backing" in result.name
def test_fallback_to_generic_when_specific_not_found(self, tmp_path):
"""Test fallback to generic pattern when specific not found."""
# Only create a generically named instrumental
generic = tmp_path / "instrumental.flac"
generic.touch()
# Both clean and backing selections should fall back
for selection in ["clean", "with_backing"]:
if selection == "with_backing":
patterns = (
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
"*with_backing*.flac", "*Backing*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
else:
patterns = (
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
result = find_file(tmp_path, *patterns)
assert result is not None
assert result.name == "instrumental.flac"
class TestInstrumentalSelectionRegression:
"""Regression tests for instrumental selection bug.
Bug: User selected 'with_backing' instrumental in the UI but the final
video used the clean instrumental instead.
Root cause: The GCE worker's find_file() always prioritized
'*instrumental_clean*' patterns before generic patterns, ignoring
the user's selection.
"""
def test_bug_scenario_both_instrumentals_present(self, tmp_path):
"""Recreate the exact bug scenario where both instrumentals exist.
This is the key regression test. When both instrumentals are downloaded
from GCS (which happens with the GCE worker), the correct one must
be selected based on the user's choice.
"""
# Setup: Both instrumentals in stems/ (as downloaded from GCS)
stems = tmp_path / "stems"
stems.mkdir()
(stems / "instrumental_clean.flac").touch()
(stems / "instrumental_with_backing.flac").touch()
# Also have properly named versions in the root (from _setup_working_directory)
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
# When user selected 'with_backing', the backing version MUST be found
backing_patterns = (
"*instrumental_with_backing*.flac", "*Instrumental Backing*.flac",
"*with_backing*.flac", "*Backing*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
result = find_file(tmp_path, *backing_patterns)
assert result is not None
# Must find a backing-related file, not the clean one
name = result.name.lower()
assert "backing" in name or "with_backing" in name, \
f"Expected backing instrumental but found: {result.name}"
def test_clean_selection_still_works(self, tmp_path):
"""Verify clean selection still works correctly after the fix."""
stems = tmp_path / "stems"
stems.mkdir()
(stems / "instrumental_clean.flac").touch()
(stems / "instrumental_with_backing.flac").touch()
(tmp_path / "Artist - Song (Instrumental Clean).flac").touch()
(tmp_path / "Artist - Song (Instrumental Backing).flac").touch()
# When user selected 'clean', the clean version must be found
clean_patterns = (
"*instrumental_clean*.flac", "*Instrumental Clean*.flac",
"*instrumental*.flac", "*Instrumental*.flac",
"*instrumental*.wav"
)
result = find_file(tmp_path, *clean_patterns)
assert result is not None
name = result.name.lower()
assert "clean" in name, f"Expected clean instrumental but found: {result.name}"
-238
View File
@@ -1,238 +0,0 @@
"""
Unit tests for user impersonation endpoint.
Tests the admin impersonation API that allows admins to view the app as other users.
"""
import pytest
from unittest.mock import Mock, patch
from fastapi.testclient import TestClient
from fastapi import FastAPI
from backend.api.routes.admin import router
from backend.api.dependencies import require_admin
from backend.services.user_service import get_user_service
from backend.services.auth_service import AuthResult, UserType
from backend.models.user import User, Session
# Create a test app with the admin router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency - returns admin AuthResult."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="Admin session valid",
user_email="admin@nomadkaraoke.com",
is_admin=True,
)
def get_mock_regular_user():
"""Override for require_admin dependency - returns regular user (should fail)."""
# This simulates what happens when a non-admin tries to access
# In reality, require_admin raises 403, but we test the logic
return AuthResult(
is_valid=True,
user_type=UserType.LIMITED,
remaining_uses=5,
message="User session valid",
user_email="user@example.com",
is_admin=False,
)
@pytest.fixture
def client():
"""Create a test client with admin access."""
app.dependency_overrides[require_admin] = get_mock_admin
yield TestClient(app)
app.dependency_overrides.clear()
@pytest.fixture
def mock_user_service():
"""Create a mock user service."""
service = Mock()
return service
@pytest.fixture
def mock_target_user():
"""Create a mock target user to impersonate."""
user = Mock(spec=User)
user.email = "target@example.com"
user.credits = 5
user.role = "user"
user.is_active = True
return user
@pytest.fixture
def mock_session():
"""Create a mock session."""
session = Mock(spec=Session)
session.token = "test-session-token-12345678901234567890"
session.user_email = "target@example.com"
return session
class TestImpersonateUser:
"""Tests for POST /api/admin/users/{email}/impersonate endpoint."""
def test_admin_can_impersonate_user(self, client, mock_target_user, mock_session):
"""Admin gets session token for target user."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
mock_service = Mock()
mock_service.get_user.return_value = mock_target_user
mock_service.create_session.return_value = mock_session
mock_get_service.return_value = mock_service
# Override the dependency
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/target@example.com/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["session_token"] == mock_session.token
assert data["user_email"] == "target@example.com"
assert "impersonating" in data["message"].lower()
# Verify user lookup was called
mock_service.get_user.assert_called_once_with("target@example.com")
# Verify session was created for target user
mock_service.create_session.assert_called_once()
call_args = mock_service.create_session.call_args
assert call_args.kwargs["user_email"] == "target@example.com"
assert "Impersonation by admin@nomadkaraoke.com" in call_args.kwargs["user_agent"]
def test_impersonate_nonexistent_user_returns_404(self, client):
"""Target user must exist."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
mock_service = Mock()
mock_service.get_user.return_value = None # User not found
mock_get_service.return_value = mock_service
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/nonexistent@example.com/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_cannot_impersonate_self(self, client, mock_target_user):
"""Admin cannot impersonate themselves."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
mock_service = Mock()
# Set up user to be found (same as admin)
mock_target_user.email = "admin@nomadkaraoke.com"
mock_service.get_user.return_value = mock_target_user
mock_get_service.return_value = mock_service
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/admin@nomadkaraoke.com/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 400
assert "yourself" in response.json()["detail"].lower()
def test_impersonation_creates_valid_session(self, client, mock_target_user, mock_session):
"""Returned token is a valid session for target user."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
mock_service = Mock()
mock_service.get_user.return_value = mock_target_user
mock_service.create_session.return_value = mock_session
mock_get_service.return_value = mock_service
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/target@example.com/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
# Verify create_session was called with correct user
mock_service.create_session.assert_called_once()
call_kwargs = mock_service.create_session.call_args.kwargs
assert call_kwargs["user_email"] == "target@example.com"
def test_impersonation_logs_audit_trail(self, client, mock_target_user, mock_session):
"""Impersonation is logged for security audit."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service, \
patch('backend.api.routes.admin.logger') as mock_logger:
mock_service = Mock()
mock_service.get_user.return_value = mock_target_user
mock_service.create_session.return_value = mock_session
mock_get_service.return_value = mock_service
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/target@example.com/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
# Verify logging was called with impersonation info
mock_logger.info.assert_called()
log_message = mock_logger.info.call_args[0][0]
assert "IMPERSONATION" in log_message
assert "admin@nomadkaraoke.com" in log_message
assert "target@example.com" in log_message
def test_email_is_normalized_to_lowercase(self, client, mock_target_user, mock_session):
"""Email addresses are normalized to lowercase."""
with patch('backend.api.routes.admin.get_user_service') as mock_get_service:
mock_service = Mock()
mock_service.get_user.return_value = mock_target_user
mock_service.create_session.return_value = mock_session
mock_get_service.return_value = mock_service
app.dependency_overrides[get_user_service] = lambda: mock_service
response = client.post(
"/api/admin/users/TARGET@EXAMPLE.COM/impersonate",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
# Verify user lookup was with lowercase
mock_service.get_user.assert_called_once_with("target@example.com")
class TestImpersonateUserNonAdmin:
"""Tests to verify non-admins cannot impersonate."""
def test_non_admin_cannot_impersonate(self):
"""
Regular user gets 403 Forbidden.
Note: This is enforced by the require_admin dependency at the route level.
We test that the dependency is correctly applied.
"""
# Create a fresh client without admin override
test_app = FastAPI()
test_app.include_router(router, prefix="/api")
# Don't override require_admin - it should reject non-admin requests
# In a real test, we'd mock the auth service to return a non-admin user
# For now, we just verify the endpoint requires admin via dependency
# The require_admin dependency will reject requests without proper admin auth
# This is tested implicitly by the fact that all other tests use admin override
pass # Actual enforcement tested via integration tests
@@ -128,8 +128,6 @@ class TestUserEmailExtraction:
mock_request.headers = {}
mock_request.client = Mock(host="127.0.0.1")
mock_request.url = Mock(path="/api/jobs/create-from-url")
# Mock tenant state (no tenant = default Nomad Karaoke)
mock_request.state.tenant_config = None
mock_background_tasks = Mock()
body = CreateJobFromUrlRequest(
@@ -170,8 +168,6 @@ class TestUserEmailExtraction:
mock_request.headers = {}
mock_request.client = Mock(host="127.0.0.1")
mock_request.url = Mock(path="/api/audio-search/search")
# Mock tenant state (no tenant = default Nomad Karaoke)
mock_request.state.tenant_config = None
mock_background_tasks = Mock()
body = AudioSearchRequest(
+9 -171
View File
@@ -33,28 +33,13 @@ def job_manager(mock_firestore_service):
class TestJobCreation:
"""Test job creation logic."""
def test_create_job_requires_theme_id(self, job_manager, mock_firestore_service):
"""Test that jobs without theme_id are rejected."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song"
# No theme_id - should fail
)
with pytest.raises(ValueError, match="theme_id is required"):
job_manager.create_job(job_create)
# Verify Firestore was NOT called
mock_firestore_service.create_job.assert_not_called()
def test_create_job_with_url(self, job_manager, mock_firestore_service):
"""Test creating a job with YouTube URL."""
job_create = JobCreate(
url="https://youtube.com/watch?v=test",
artist="Test Artist",
title="Test Song",
theme_id="nomad" # Required for all jobs
title="Test Song"
)
# The actual create_job method creates the job and returns it
@@ -74,8 +59,7 @@ class TestJobCreation:
"""Test creating a job without URL (for file upload)."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad" # Required for all jobs
title="Test Song"
)
job = job_manager.create_job(job_create)
@@ -86,8 +70,8 @@ class TestJobCreation:
def test_create_job_generates_unique_id(self, job_manager, mock_firestore_service):
"""Test that each job gets a unique ID."""
job_create = JobCreate(theme_id="nomad") # Required for all jobs
job_create = JobCreate()
# Create multiple jobs
ids = []
for i in range(5):
@@ -105,8 +89,8 @@ class TestJobCreation:
def test_create_job_sets_initial_status(self, job_manager, mock_firestore_service):
"""Test that new jobs start with PENDING status."""
job_create = JobCreate(theme_id="nomad") # Required for all jobs
job_create = JobCreate()
mock_firestore_service.create_job.return_value = Job(
job_id="test123",
status=JobStatus.PENDING,
@@ -121,14 +105,13 @@ class TestJobCreation:
def test_create_job_with_distribution_settings(self, job_manager, mock_firestore_service):
"""Test that distribution settings are passed from JobCreate to Job.
This was a bug where brand_prefix, dropbox_path, gdrive_folder_id, and
discord_webhook_url were NOT being passed to the Job constructor.
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad", # Required for all jobs
brand_prefix="NOMAD",
discord_webhook_url="https://discord.com/webhook/test",
dropbox_path="/Karaoke/Tracks-Organized",
@@ -313,154 +296,9 @@ class TestJobFailure:
assert call_args[1]['error_message'] == error_message
class TestMadeForYouFieldMapping:
"""
CRITICAL: Test that made-for-you fields are properly copied from JobCreate to Job.
These tests verify that JobManager.create_job() properly maps all fields from
JobCreate to the Job object that gets persisted to Firestore.
Bug context (2026-01-09): Production made-for-you orders were created with
made_for_you=False, customer_email=None, customer_notes=None because
JobManager.create_job() wasn't copying these fields from JobCreate to Job.
"""
def test_create_job_copies_made_for_you_flag(self, job_manager, mock_firestore_service):
"""
CRITICAL: made_for_you flag must be copied from JobCreate to Job.
This flag is essential for:
- Ownership transfer on job completion
- Email suppression for intermediate states
- Identifying made-for-you jobs in admin UI
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
made_for_you=True, # This MUST be copied to the Job
)
job = job_manager.create_job(job_create)
# CRITICAL ASSERTION: made_for_you must be True on the created Job
assert job.made_for_you is True, \
"made_for_you=True on JobCreate must be copied to Job"
# Also verify what was saved to Firestore
mock_firestore_service.create_job.assert_called_once()
saved_job = mock_firestore_service.create_job.call_args[0][0]
assert saved_job.made_for_you is True, \
"made_for_you=True must be persisted to Firestore"
def test_create_job_copies_customer_email(self, job_manager, mock_firestore_service):
"""
CRITICAL: customer_email must be copied from JobCreate to Job.
This field is essential for:
- Ownership transfer on job completion (transferring to this email)
- Sending completion email with download links to customer
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
made_for_you=True,
customer_email="customer@example.com", # This MUST be copied to the Job
)
job = job_manager.create_job(job_create)
# CRITICAL ASSERTION: customer_email must be copied
assert job.customer_email == "customer@example.com", \
"customer_email on JobCreate must be copied to Job"
# Also verify what was saved to Firestore
saved_job = mock_firestore_service.create_job.call_args[0][0]
assert saved_job.customer_email == "customer@example.com", \
"customer_email must be persisted to Firestore"
def test_create_job_copies_customer_notes(self, job_manager, mock_firestore_service):
"""
customer_notes must be copied from JobCreate to Job.
Customer notes contain special requests that admin needs to see.
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
made_for_you=True,
customer_notes="Please make this perfect for my wedding!",
)
job = job_manager.create_job(job_create)
assert job.customer_notes == "Please make this perfect for my wedding!", \
"customer_notes on JobCreate must be copied to Job"
saved_job = mock_firestore_service.create_job.call_args[0][0]
assert saved_job.customer_notes == "Please make this perfect for my wedding!", \
"customer_notes must be persisted to Firestore"
def test_create_job_full_made_for_you_config(self, job_manager, mock_firestore_service):
"""
Test complete made-for-you job creation with all fields.
This is the realistic scenario: a made-for-you order creates a job with:
- made_for_you=True
- user_email=admin (owner during processing)
- customer_email=customer (for final delivery)
- customer_notes=notes (customer's special requests)
"""
job_create = JobCreate(
artist="Avril Lavigne",
title="Complicated",
theme_id="nomad",
made_for_you=True,
user_email="admin@nomadkaraoke.com",
customer_email="customer@example.com",
customer_notes="Anniversary gift!",
# Distribution settings that should also be applied
enable_youtube_upload=True,
dropbox_path="/Production/Ready",
brand_prefix="NOMAD",
)
job = job_manager.create_job(job_create)
# Verify all made-for-you fields
assert job.made_for_you is True
assert job.user_email == "admin@nomadkaraoke.com"
assert job.customer_email == "customer@example.com"
assert job.customer_notes == "Anniversary gift!"
# Verify distribution settings too
assert job.enable_youtube_upload is True
assert job.dropbox_path == "/Production/Ready"
assert job.brand_prefix == "NOMAD"
def test_create_job_made_for_you_false_by_default(self, job_manager, mock_firestore_service):
"""
Regular jobs should have made_for_you=False by default.
"""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
theme_id="nomad",
# No made_for_you specified - should default to False
)
job = job_manager.create_job(job_create)
assert job.made_for_you is False
assert job.customer_email is None
assert job.customer_notes is None
class TestJobDeletion:
"""Test job deletion logic."""
def test_delete_job(self, job_manager, mock_firestore_service):
"""Test deleting a job."""
existing_job = Job(
@@ -1,329 +0,0 @@
"""
Unit tests for job manager notification methods.
Tests the state transition triggers for email notifications.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock, AsyncMock
from datetime import datetime
from backend.services.job_manager import JobManager
from backend.models.job import Job, JobStatus
class TestTriggerStateNotifications:
"""Tests for _trigger_state_notifications method."""
def test_triggers_completion_email_on_complete(self):
"""Test that completion email is triggered when job completes."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {"youtube_url": "https://youtube.com/123"}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_called_once_with(mock_job)
def test_triggers_idle_reminder_on_awaiting_review(self):
"""Test that idle reminder is triggered when job enters awaiting review."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.AWAITING_REVIEW)
mock_schedule.assert_called_once_with(mock_job, JobStatus.AWAITING_REVIEW)
def test_triggers_idle_reminder_on_awaiting_instrumental(self):
"""Test that idle reminder is triggered when job enters awaiting instrumental."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_idle_reminder') as mock_schedule:
manager._trigger_state_notifications(
"job-123",
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
mock_schedule.assert_called_once_with(
mock_job,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
def test_skips_notification_when_no_job(self):
"""Test that notification is skipped when job not found."""
manager = JobManager()
manager.firestore = Mock()
manager.get_job = Mock(return_value=None)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_not_called()
def test_skips_notification_when_no_user_email(self):
"""Test that notification is skipped when job has no user email."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = None
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_schedule:
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
mock_schedule.assert_not_called()
def test_does_not_trigger_on_other_states(self):
"""Test that no notification is triggered for non-notification states."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_schedule_completion_email') as mock_complete, \
patch.object(manager, '_schedule_idle_reminder') as mock_idle:
# Test various non-notification states
for status in [JobStatus.PENDING, JobStatus.DOWNLOADING, JobStatus.TRANSCRIBING,
JobStatus.RENDERING_VIDEO, JobStatus.FAILED]:
manager._trigger_state_notifications("job-123", status)
mock_complete.assert_not_called()
mock_idle.assert_not_called()
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate from notifications."""
manager = JobManager()
manager.firestore = Mock()
manager.get_job = Mock(side_effect=Exception("Database error"))
# Should not raise
manager._trigger_state_notifications("job-123", JobStatus.COMPLETE)
class TestScheduleCompletionEmail:
"""Tests for _schedule_completion_email method."""
def test_extracts_urls_from_state_data(self):
"""Test that YouTube and Dropbox URLs are extracted from state_data."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.artist = "Test Artist"
mock_job.title = "Test Song"
mock_job.state_data = {
"youtube_url": "https://youtube.com/watch?v=123",
"dropbox_link": "https://dropbox.com/folder/abc",
}
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_service = Mock()
mock_service.send_job_completion_email = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Run in sync context - this will start a daemon thread
manager._schedule_completion_email(mock_job)
# Give the thread a moment to execute
import time
time.sleep(0.1)
def test_handles_none_state_data(self):
"""Test that None state_data is handled gracefully."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.artist = None
mock_job.title = None
mock_job.state_data = None # None state_data
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_service = Mock()
mock_service.send_job_completion_email = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Should not raise
manager._schedule_completion_email(mock_job)
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
with patch('backend.services.job_notification_service.get_job_notification_service') as mock_get_service:
mock_get_service.side_effect = Exception("Service error")
# Should not raise
manager._schedule_completion_email(mock_job)
class TestScheduleIdleReminder:
"""Tests for _schedule_idle_reminder method."""
def test_updates_state_data_for_lyrics_review(self):
"""Test that state_data is updated with blocking state info for lyrics."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {"existing_key": "value"}
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
# Verify state_data was updated
update_call = manager.firestore.update_job.call_args
assert update_call is not None
updated_data = update_call[0][1] # Second positional arg
state_data = updated_data.get('state_data', {})
assert state_data.get('blocking_action_type') == 'lyrics'
assert state_data.get('reminder_sent') is False
assert 'blocking_state_entered_at' in state_data
assert state_data.get('existing_key') == 'value' # Preserved
def test_updates_state_data_for_instrumental_selection(self):
"""Test that state_data is updated with blocking state info for instrumental."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
manager._schedule_idle_reminder(
mock_job,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION
)
# Verify state_data was updated
update_call = manager.firestore.update_job.call_args
updated_data = update_call[0][1]
state_data = updated_data.get('state_data', {})
assert state_data.get('blocking_action_type') == 'instrumental'
def test_handles_none_state_data(self):
"""Test that None state_data is handled gracefully."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = None
with patch('backend.services.worker_service.get_worker_service') as mock_get_service:
mock_service = Mock()
mock_service.schedule_idle_reminder = AsyncMock(return_value=True)
mock_get_service.return_value = mock_service
# Should not raise
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
# Verify update was still called
manager.firestore.update_job.assert_called_once()
def test_handles_exception_gracefully(self):
"""Test that exceptions don't propagate."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
# Make firestore raise an exception
manager.firestore.update_job.side_effect = Exception("Database error")
# Should not raise
manager._schedule_idle_reminder(mock_job, JobStatus.AWAITING_REVIEW)
class TestTransitionTriggersNotifications:
"""Integration tests for transition_to_state triggering notifications."""
def test_transition_to_complete_triggers_notification(self):
"""Test that transitioning to COMPLETE triggers notification."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
# Use ENCODING as the starting state since ENCODING -> COMPLETE is valid
mock_job.status = JobStatus.ENCODING
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
manager.transition_to_state("job-123", JobStatus.COMPLETE)
mock_trigger.assert_called_once_with("job-123", JobStatus.COMPLETE)
def test_transition_to_awaiting_review_triggers_notification(self):
"""Test that transitioning to AWAITING_REVIEW triggers notification."""
manager = JobManager()
manager.firestore = Mock()
mock_job = Mock(spec=Job)
mock_job.job_id = "job-123"
# Use APPLYING_PADDING as the starting state since APPLYING_PADDING -> AWAITING_REVIEW is valid
mock_job.status = JobStatus.APPLYING_PADDING
mock_job.user_email = "user@example.com"
mock_job.state_data = {}
manager.get_job = Mock(return_value=mock_job)
with patch.object(manager, '_trigger_state_notifications') as mock_trigger:
manager.transition_to_state("job-123", JobStatus.AWAITING_REVIEW)
mock_trigger.assert_called_once_with("job-123", JobStatus.AWAITING_REVIEW)
@@ -1,409 +0,0 @@
"""
Unit tests for job notification service.
"""
import pytest
from unittest.mock import Mock, patch, AsyncMock
from backend.services.job_notification_service import (
JobNotificationService,
get_job_notification_service,
ENABLE_AUTO_EMAILS,
FEEDBACK_FORM_URL,
)
class TestURLBuilding:
"""Tests for URL building methods."""
def test_build_review_url_basic(self):
"""Test basic review URL building."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
url = service._build_review_url("job-123")
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
def test_build_review_url_ignores_legacy_params(self):
"""Test review URL ignores legacy audio_hash and review_token params."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
# Legacy params are still accepted but not used in the URL
url = service._build_review_url("job-123", audio_hash="abc123", review_token="token456")
# URL should be the simple consolidated route
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
# No query params
assert "?" not in url
def test_build_review_url_preserves_job_id_characters(self):
"""Test that job ID is used directly in URL path."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
# Note: job IDs with slashes would be unusual in real usage
url = service._build_review_url("abc-def-123")
assert url == "https://gen.nomadkaraoke.com/app/jobs#/abc-def-123/review"
def test_build_instrumental_url_basic(self):
"""Test basic instrumental URL building."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
url = service._build_instrumental_url("job-123")
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
def test_build_instrumental_url_ignores_legacy_params(self):
"""Test instrumental URL ignores legacy instrumental_token param."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
# Legacy param is still accepted but not used in the URL
url = service._build_instrumental_url("job-123", instrumental_token="inst-token")
# URL should be the simple consolidated route
assert url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
# No query params
assert "?" not in url
class TestCompletionEmail:
"""Tests for job completion email sending."""
@pytest.mark.asyncio
async def test_send_completion_email_success(self):
"""Test successful completion email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
user_name="Test User",
artist="Test Artist",
title="Test Song",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
)
assert result is True
service.template_service.render_job_completion.assert_called_once()
service.email_service.send_job_completion.assert_called_once()
@pytest.mark.asyncio
async def test_send_completion_email_disabled(self):
"""Test that completion email is skipped when auto emails are disabled."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_no_email(self):
"""Test that completion email is skipped when no user email."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email=None,
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_empty_email(self):
"""Test that completion email is skipped when user email is empty."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="",
)
assert result is False
service.email_service.send_job_completion.assert_not_called()
@pytest.mark.asyncio
async def test_send_completion_email_send_failure(self):
"""Test handling of email send failure."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = False
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
@pytest.mark.asyncio
async def test_send_completion_email_exception(self):
"""Test handling of exceptions during email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.side_effect = Exception("Send error")
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
assert result is False
@pytest.mark.asyncio
async def test_send_completion_email_passes_cc_admin(self):
"""Test that completion email is sent with CC to admin."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_job_completion.return_value = True
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Test message"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_job_completion_email(
job_id="job-123",
user_email="user@example.com",
)
# Verify cc_admin=True was passed
call_kwargs = service.email_service.send_job_completion.call_args.kwargs
assert call_kwargs.get('cc_admin') is True
class TestActionReminderEmail:
"""Tests for action reminder email sending."""
@pytest.mark.asyncio
async def test_send_lyrics_reminder_success(self):
"""Test successful lyrics reminder email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_lyrics.return_value = "Review your lyrics"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
artist="Test Artist",
title="Test Song",
)
assert result is True
service.template_service.render_action_needed_lyrics.assert_called_once()
service.email_service.send_action_reminder.assert_called_once()
@pytest.mark.asyncio
async def test_send_instrumental_reminder_success(self):
"""Test successful instrumental reminder email sending."""
service = JobNotificationService()
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_instrumental.return_value = "Select instrumental"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="instrumental",
)
assert result is True
service.template_service.render_action_needed_instrumental.assert_called_once()
service.email_service.send_action_reminder.assert_called_once()
@pytest.mark.asyncio
async def test_send_reminder_disabled(self):
"""Test that reminder email is skipped when auto emails are disabled."""
service = JobNotificationService()
service.email_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', False):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
)
assert result is False
service.email_service.send_action_reminder.assert_not_called()
@pytest.mark.asyncio
async def test_send_reminder_no_email(self):
"""Test that reminder email is skipped when no user email."""
service = JobNotificationService()
service.email_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email=None,
action_type="lyrics",
)
assert result is False
@pytest.mark.asyncio
async def test_send_reminder_unknown_action_type(self):
"""Test handling of unknown action type."""
service = JobNotificationService()
service.email_service = Mock()
service.template_service = Mock()
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
result = await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="unknown",
)
assert result is False
service.email_service.send_action_reminder.assert_not_called()
@pytest.mark.asyncio
async def test_send_lyrics_reminder_includes_review_url(self):
"""Test that lyrics reminder includes correct review URL."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_lyrics.return_value = "Review"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="lyrics",
)
# Verify the review URL was passed to template
call_kwargs = service.template_service.render_action_needed_lyrics.call_args.kwargs
review_url = call_kwargs.get('review_url')
assert review_url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/review"
@pytest.mark.asyncio
async def test_send_instrumental_reminder_includes_url(self):
"""Test that instrumental reminder includes correct URL."""
service = JobNotificationService()
service.frontend_url = "https://gen.nomadkaraoke.com"
service.email_service = Mock()
service.email_service.send_action_reminder.return_value = True
service.template_service = Mock()
service.template_service.render_action_needed_instrumental.return_value = "Select"
with patch('backend.services.job_notification_service.ENABLE_AUTO_EMAILS', True):
await service.send_action_reminder_email(
job_id="job-123",
user_email="user@example.com",
action_type="instrumental",
)
# Verify the instrumental URL was passed to template
call_kwargs = service.template_service.render_action_needed_instrumental.call_args.kwargs
instrumental_url = call_kwargs.get('instrumental_url')
assert instrumental_url == "https://gen.nomadkaraoke.com/app/jobs#/job-123/instrumental"
class TestGetCompletionMessage:
"""Tests for get_completion_message method."""
def test_get_completion_message_basic(self):
"""Test basic completion message retrieval."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Your video is ready!"
result = service.get_completion_message(
job_id="job-123",
user_name="Test User",
artist="Test Artist",
title="Test Song",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
)
assert result == "Your video is ready!"
service.template_service.render_job_completion.assert_called_once()
def test_get_completion_message_includes_feedback_url(self):
"""Test that completion message includes feedback URL."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Message"
service.get_completion_message(job_id="job-123")
# Verify feedback URL was passed
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
assert 'feedback_url' in call_kwargs
def test_get_completion_message_with_defaults(self):
"""Test completion message with default values."""
service = JobNotificationService()
service.template_service = Mock()
service.template_service.render_job_completion.return_value = "Message"
service.get_completion_message(job_id="job-123")
call_kwargs = service.template_service.render_job_completion.call_args.kwargs
assert call_kwargs.get('job_id') == "job-123"
# Other params should be None/default
assert call_kwargs.get('name') is None
assert call_kwargs.get('artist') is None
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_job_notification_service_returns_same_instance(self):
"""Test that get_job_notification_service returns singleton."""
# Reset global
import backend.services.job_notification_service as jns
jns._job_notification_service = None
service1 = get_job_notification_service()
service2 = get_job_notification_service()
assert service1 is service2
def test_service_initializes_with_dependencies(self):
"""Test that service initializes with email and template services."""
service = JobNotificationService()
assert service.email_service is not None
assert service.template_service is not None
assert service.frontend_url is not None
assert service.backend_url is not None
+1 -11
View File
@@ -50,16 +50,7 @@ def mock_worker_service():
@pytest.fixture
def mock_theme_service():
"""Create a mock ThemeService that returns 'nomad' as default theme."""
service = MagicMock()
service.get_default_theme_id.return_value = "nomad"
service.get_theme.return_value = None
return service
@pytest.fixture
def client(mock_job_manager, mock_worker_service, mock_theme_service):
def client(mock_job_manager, mock_worker_service):
"""Create TestClient with mocked dependencies."""
mock_creds = MagicMock()
mock_creds.universe_domain = 'googleapis.com'
@@ -72,7 +63,6 @@ def client(mock_job_manager, mock_worker_service, mock_theme_service):
# Also patch JobManager class used in dependencies.py for auth checks
with patch('backend.api.routes.jobs.job_manager', mock_job_manager), \
patch('backend.api.routes.jobs.worker_service', mock_worker_service), \
patch('backend.api.routes.jobs.get_theme_service', return_value=mock_theme_service), \
patch('backend.services.job_manager.JobManager', mock_job_manager_factory), \
patch('backend.services.firestore_service.firestore'), \
patch('backend.services.storage_service.storage'), \
@@ -1,567 +0,0 @@
"""
Tests for LocalPreviewEncodingService.
Tests cover:
- Service initialization
- Hardware acceleration detection (NVENC)
- FFmpeg filter path escaping
- ASS filter building
- Preview FFmpeg command construction
- Preview encoding execution
- Error handling
"""
import os
import tempfile
import pytest
from unittest.mock import MagicMock, patch
from backend.services.local_preview_encoding_service import (
LocalPreviewEncodingService,
PreviewEncodingConfig,
PreviewEncodingResult,
get_local_preview_encoding_service,
)
class TestLocalPreviewEncodingServiceInit:
"""Test service initialization."""
def test_init_default_values(self):
"""Test default initialization."""
service = LocalPreviewEncodingService()
assert service._nvenc_available is None # Lazy detection
assert service._video_encoder is None # Lazy detection
assert service._hwaccel_flags is None # Lazy detection
def test_init_with_custom_logger(self):
"""Test initialization with custom logger."""
import logging
custom_logger = logging.getLogger("test")
service = LocalPreviewEncodingService(logger=custom_logger)
assert service.logger is custom_logger
class TestLocalPreviewEncodingServiceHWAccel:
"""Test hardware acceleration detection."""
@patch("subprocess.run")
def test_detect_nvenc_available(self, mock_run):
"""Test NVENC detection when available."""
mock_run.return_value = MagicMock(returncode=0)
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is True
@patch("subprocess.run")
def test_detect_nvenc_not_available(self, mock_run):
"""Test fallback when NVENC not available."""
import subprocess
mock_run.side_effect = subprocess.CalledProcessError(1, "test")
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is False
@patch("subprocess.run")
def test_detect_nvenc_timeout(self, mock_run):
"""Test NVENC detection timeout handling."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("test", 30)
service = LocalPreviewEncodingService()
result = service._detect_nvenc_support()
assert result is False
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
def test_configure_nvenc_available(self, mock_detect):
"""Test configuration when NVENC is available."""
mock_detect.return_value = True
service = LocalPreviewEncodingService()
service._configure_hardware_acceleration()
assert service._nvenc_available is True
assert service._video_encoder == "h264_nvenc"
assert "-hwaccel" in service._hwaccel_flags
@patch.object(LocalPreviewEncodingService, "_detect_nvenc_support")
def test_configure_no_nvenc(self, mock_detect):
"""Test configuration when NVENC not available."""
mock_detect.return_value = False
service = LocalPreviewEncodingService()
service._configure_hardware_acceleration()
assert service._nvenc_available is False
assert service._video_encoder == "libx264"
assert service._hwaccel_flags == []
class TestLocalPreviewEncodingServiceProperties:
"""Test property accessors that trigger lazy detection."""
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_nvenc_available_triggers_detection(self, mock_config):
"""Test that nvenc_available property triggers detection."""
service = LocalPreviewEncodingService()
service._nvenc_available = None
# Access property
_ = service.nvenc_available
mock_config.assert_called_once()
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_video_encoder_triggers_detection(self, mock_config):
"""Test that video_encoder property triggers detection."""
service = LocalPreviewEncodingService()
service._video_encoder = None
# Access property
_ = service.video_encoder
mock_config.assert_called_once()
@patch.object(LocalPreviewEncodingService, "_configure_hardware_acceleration")
def test_hwaccel_flags_triggers_detection(self, mock_config):
"""Test that hwaccel_flags property triggers detection."""
service = LocalPreviewEncodingService()
service._hwaccel_flags = None
# Access property
_ = service.hwaccel_flags
mock_config.assert_called_once()
class TestLocalPreviewEncodingServicePathEscaping:
"""Test FFmpeg filter path escaping."""
def test_escape_simple_path(self):
"""Test escaping a simple path."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/simple/path.ass")
assert result == "/simple/path.ass"
def test_escape_path_with_spaces(self):
"""Test escaping path with spaces."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path/with spaces/file.ass")
assert "\\ " in result # Spaces should be escaped
def test_escape_path_with_apostrophe(self):
"""Test escaping path with apostrophe (common in song titles)."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path/I'm With You/file.ass")
assert "\\\\\\'" in result # Apostrophe should be triple-backslash escaped
def test_escape_path_with_special_chars(self):
"""Test escaping path with FFmpeg special characters."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path:with[special];chars,here.ass")
# Should escape :,[];
assert "\\:" in result
assert "\\[" in result
assert "\\]" in result
assert "\\;" in result
assert "\\," in result
def test_escape_path_with_backslashes(self):
"""Test escaping path with existing backslashes."""
service = LocalPreviewEncodingService()
result = service._escape_ffmpeg_filter_path("/path\\with\\backslashes.ass")
assert "\\\\" in result
class TestLocalPreviewEncodingServiceASSFilter:
"""Test ASS filter building."""
def test_build_ass_filter_simple(self):
"""Test building ASS filter without font."""
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass")
assert result.startswith("ass=")
assert "fontsdir" not in result
def test_build_ass_filter_with_font(self):
"""Test building ASS filter with custom font."""
with tempfile.TemporaryDirectory() as tmpdir:
font_path = os.path.join(tmpdir, "custom.ttf")
with open(font_path, "w") as f:
f.write("fake font")
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass", font_path)
assert "fontsdir=" in result
def test_build_ass_filter_nonexistent_font(self):
"""Test building ASS filter with nonexistent font file."""
service = LocalPreviewEncodingService()
result = service._build_ass_filter("/path/to/file.ass", "/nonexistent/font.ttf")
# Should not include fontsdir for nonexistent font
assert "fontsdir" not in result
class TestLocalPreviewEncodingServiceFFmpegCommand:
"""Test FFmpeg command building."""
def test_build_command_solid_background_libx264(self):
"""Test command building with solid background and libx264."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_color="black"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "ffmpeg" in cmd
assert "-r" in cmd and "24" in cmd # Frame rate
assert "-c:v" in cmd
assert "libx264" in cmd
assert "-preset" in cmd and "superfast" in cmd
assert "-crf" in cmd and "28" in cmd
assert "color=c=black" in " ".join(cmd)
def test_build_command_with_nvenc(self):
"""Test command building with NVENC hardware acceleration."""
service = LocalPreviewEncodingService()
service._nvenc_available = True
service._video_encoder = "h264_nvenc"
service._hwaccel_flags = ["-hwaccel", "cuda", "-hwaccel_output_format", "cuda"]
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "h264_nvenc" in cmd
assert "-hwaccel" in cmd
assert "-preset" in cmd and "p1" in cmd # Fastest NVENC preset
def test_build_command_with_background_image(self):
"""Test command building with background image."""
with tempfile.TemporaryDirectory() as tmpdir:
bg_path = os.path.join(tmpdir, "background.png")
with open(bg_path, "wb") as f:
f.write(b"fake png")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_image_path=bg_path
)
cmd = service._build_preview_ffmpeg_command(config)
assert "-loop" in cmd
assert "1" in cmd
assert bg_path in cmd
# Should have scale filter for background image
vf_index = cmd.index("-vf")
vf_value = cmd[vf_index + 1]
assert "scale=" in vf_value
assert "pad=" in vf_value
def test_build_command_resolution(self):
"""Test that command uses correct preview resolution."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
cmd_str = " ".join(cmd)
# Resolution should be 480x270
assert "480" in cmd_str
assert "270" in cmd_str
def test_build_command_audio_settings(self):
"""Test that command uses correct audio settings."""
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
cmd = service._build_preview_ffmpeg_command(config)
assert "-c:a" in cmd
assert "aac" in cmd
assert "-b:a" in cmd
assert "96k" in cmd
class TestLocalPreviewEncodingServiceEncode:
"""Test preview encoding execution."""
@patch("subprocess.run")
def test_encode_preview_success(self, mock_run):
"""Test successful preview encoding."""
mock_run.return_value = MagicMock(returncode=0, stderr="")
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
# Create input files
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is True
assert result.output_path == output_path
assert result.error is None
mock_run.assert_called_once()
@patch("subprocess.run")
def test_encode_preview_ffmpeg_failure(self, mock_run):
"""Test preview encoding with FFmpeg failure."""
mock_run.return_value = MagicMock(returncode=1, stderr="FFmpeg error message")
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "FFmpeg preview encoding failed" in result.error
def test_encode_preview_missing_ass_file(self):
"""Test preview encoding with missing ASS file."""
with tempfile.TemporaryDirectory() as tmpdir:
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
config = PreviewEncodingConfig(
ass_path="/nonexistent/subs.ass",
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "not found" in result.error
def test_encode_preview_missing_audio_file(self):
"""Test preview encoding with missing audio file."""
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
service = LocalPreviewEncodingService()
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path="/nonexistent/audio.flac",
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "not found" in result.error
@patch("subprocess.run")
def test_encode_preview_timeout(self, mock_run):
"""Test preview encoding timeout."""
import subprocess
mock_run.side_effect = subprocess.TimeoutExpired("ffmpeg", 300)
with tempfile.TemporaryDirectory() as tmpdir:
ass_path = os.path.join(tmpdir, "subs.ass")
audio_path = os.path.join(tmpdir, "audio.flac")
output_path = os.path.join(tmpdir, "output.mp4")
with open(ass_path, "w") as f:
f.write("[Script Info]\n")
with open(audio_path, "wb") as f:
f.write(b"fake audio")
service = LocalPreviewEncodingService()
service._nvenc_available = False
service._video_encoder = "libx264"
service._hwaccel_flags = []
config = PreviewEncodingConfig(
ass_path=ass_path,
audio_path=audio_path,
output_path=output_path
)
result = service.encode_preview(config)
assert result.success is False
assert "timed out" in result.error
class TestPreviewEncodingConfig:
"""Test PreviewEncodingConfig dataclass."""
def test_config_required_fields(self):
"""Test that required fields must be provided."""
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4"
)
assert config.ass_path == "/path/to/subs.ass"
assert config.background_color == "black" # Default
assert config.background_image_path is None # Optional
def test_config_all_fields(self):
"""Test config with all fields."""
config = PreviewEncodingConfig(
ass_path="/path/to/subs.ass",
audio_path="/path/to/audio.flac",
output_path="/path/to/output.mp4",
background_image_path="/path/to/bg.png",
background_color="red",
font_path="/path/to/font.ttf"
)
assert config.background_image_path == "/path/to/bg.png"
assert config.background_color == "red"
assert config.font_path == "/path/to/font.ttf"
class TestPreviewEncodingResult:
"""Test PreviewEncodingResult dataclass."""
def test_result_success(self):
"""Test successful result."""
result = PreviewEncodingResult(
success=True,
output_path="/path/to/output.mp4"
)
assert result.success is True
assert result.error is None
assert result.output_path == "/path/to/output.mp4"
def test_result_failure(self):
"""Test failure result."""
result = PreviewEncodingResult(
success=False,
error="Something went wrong"
)
assert result.success is False
assert result.error == "Something went wrong"
assert result.output_path is None
class TestGetLocalPreviewEncodingService:
"""Test factory function."""
def test_get_service_creates_instance(self):
"""Test that factory function creates a new instance."""
import backend.services.local_preview_encoding_service as module
module._local_preview_encoding_service = None
service = get_local_preview_encoding_service()
assert service is not None
assert isinstance(service, LocalPreviewEncodingService)
def test_get_service_returns_singleton(self):
"""Test that factory function returns the same instance."""
import backend.services.local_preview_encoding_service as module
module._local_preview_encoding_service = None
service1 = get_local_preview_encoding_service()
service2 = get_local_preview_encoding_service()
assert service1 is service2
class TestPreviewEncodingConstants:
"""Test that encoding constants match expected values."""
def test_preview_resolution(self):
"""Test preview resolution constants."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_WIDTH == 480
assert service.PREVIEW_HEIGHT == 270
def test_preview_fps(self):
"""Test preview frame rate constant."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_FPS == 24
def test_preview_audio_bitrate(self):
"""Test preview audio bitrate constant."""
service = LocalPreviewEncodingService()
assert service.PREVIEW_AUDIO_BITRATE == "96k"
File diff suppressed because it is too large. Load diff
-139
View File
@@ -913,145 +913,6 @@ class TestJobWithFullTwoPhaseConfig:
assert "title" in job.file_urls["screens"]
class TestMadeForYouFields:
"""Tests for made-for-you order tracking fields."""
def test_job_has_made_for_you_field(self):
"""Test Job model has made_for_you field with default False."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
)
assert hasattr(job, 'made_for_you')
assert job.made_for_you is False
def test_job_made_for_you_true(self):
"""Test Job model can set made_for_you to True."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
made_for_you=True,
)
assert job.made_for_you is True
def test_job_has_customer_email_field(self):
"""Test Job model has customer_email field with default None."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
)
assert hasattr(job, 'customer_email')
assert job.customer_email is None
def test_job_customer_email_set(self):
"""Test Job model can set customer_email."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
customer_email="customer@example.com",
)
assert job.customer_email == "customer@example.com"
def test_job_has_customer_notes_field(self):
"""Test Job model has customer_notes field with default None."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
)
assert hasattr(job, 'customer_notes')
assert job.customer_notes is None
def test_job_customer_notes_set(self):
"""Test Job model can set customer_notes."""
job = Job(
job_id="test123",
status=JobStatus.PENDING,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
customer_notes="Please make it extra special!",
)
assert job.customer_notes == "Please make it extra special!"
def test_job_create_has_made_for_you_fields(self):
"""Test JobCreate model has all made-for-you fields."""
job_create = JobCreate(
artist="Test Artist",
title="Test Song",
)
assert hasattr(job_create, 'made_for_you')
assert hasattr(job_create, 'customer_email')
assert hasattr(job_create, 'customer_notes')
assert job_create.made_for_you is False
assert job_create.customer_email is None
assert job_create.customer_notes is None
def test_job_create_with_made_for_you_config(self):
"""Test JobCreate with full made-for-you configuration."""
job_create = JobCreate(
artist="Seether",
title="Tonight",
user_email="admin@nomadkaraoke.com",
made_for_you=True,
customer_email="customer@example.com",
customer_notes="Wedding anniversary!",
)
assert job_create.made_for_you is True
assert job_create.customer_email == "customer@example.com"
assert job_create.customer_notes == "Wedding anniversary!"
assert job_create.user_email == "admin@nomadkaraoke.com"
def test_made_for_you_serialization_roundtrip(self):
"""Test made-for-you fields survive dict serialization."""
job = Job(
job_id="test123",
status=JobStatus.AWAITING_AUDIO_SELECTION,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
made_for_you=True,
customer_email="customer@example.com",
customer_notes="Test notes",
user_email="admin@nomadkaraoke.com",
)
job_dict = job.model_dump()
assert job_dict['made_for_you'] is True
assert job_dict['customer_email'] == "customer@example.com"
assert job_dict['customer_notes'] == "Test notes"
def test_made_for_you_job_with_distribution_settings(self):
"""Test made-for-you job with distribution settings."""
job = Job(
job_id="test123",
status=JobStatus.AWAITING_AUDIO_SELECTION,
created_at=datetime.now(UTC),
updated_at=datetime.now(UTC),
made_for_you=True,
customer_email="customer@example.com",
user_email="admin@nomadkaraoke.com",
enable_youtube_upload=True,
dropbox_path="/Production/Ready To Upload",
gdrive_folder_id="1ABC123",
brand_prefix="NOMAD",
)
assert job.made_for_you is True
assert job.enable_youtube_upload is True
assert job.dropbox_path == "/Production/Ready To Upload"
assert job.gdrive_folder_id == "1ABC123"
assert job.brand_prefix == "NOMAD"
if __name__ == "__main__":
pytest.main([__file__, "-v"])
+1 -1
View File
@@ -201,7 +201,7 @@ class TestPackagingServiceCDGPackage:
assert result[1] == mp3_file
assert result[2] == cdg_file
@patch("karaoke_gen.lyrics_transcriber.output.cdg.CDGGenerator")
@patch("lyrics_transcriber.output.cdg.CDGGenerator")
def test_create_cdg_package_with_generator(self, mock_cdg_generator_class):
"""Test CDG package creation using CDGGenerator."""
with tempfile.TemporaryDirectory() as tmpdir:
@@ -1,460 +0,0 @@
"""
Tests for PushNotificationService.
Tests the Web Push notification sending and subscription management.
"""
import pytest
from unittest.mock import Mock, patch, AsyncMock, MagicMock
from datetime import datetime, timezone
from backend.services.push_notification_service import (
PushNotificationService,
get_push_notification_service,
SubscriptionGoneError,
)
@pytest.fixture
def mock_settings():
"""Mock settings with push notifications enabled."""
settings = Mock()
settings.enable_push_notifications = True
settings.max_push_subscriptions_per_user = 5
settings.vapid_subject = "mailto:test@example.com"
settings.get_secret = Mock(side_effect=lambda x: {
"vapid-public-key": "test-public-key",
"vapid-private-key": "test-private-key"
}.get(x))
return settings
@pytest.fixture
def mock_db():
"""Mock Firestore client."""
return Mock()
@pytest.fixture
def push_service(mock_settings, mock_db):
"""Create PushNotificationService with mocked dependencies."""
with patch('backend.services.push_notification_service.get_settings', return_value=mock_settings):
service = PushNotificationService(db=mock_db)
return service
class TestPushNotificationServiceInit:
"""Tests for service initialization and configuration."""
def test_is_enabled_when_configured(self, push_service):
"""Service reports enabled when all config present."""
assert push_service.is_enabled() is True
def test_is_disabled_when_feature_flag_off(self, mock_db):
"""Service reports disabled when feature flag off."""
settings = Mock()
settings.enable_push_notifications = False
settings.get_secret = Mock(return_value="key")
with patch('backend.services.push_notification_service.get_settings', return_value=settings):
service = PushNotificationService(db=mock_db)
assert service.is_enabled() is False
def test_is_disabled_when_vapid_keys_missing(self, mock_db):
"""Service reports disabled when VAPID keys missing."""
settings = Mock()
settings.enable_push_notifications = True
settings.get_secret = Mock(return_value=None)
with patch('backend.services.push_notification_service.get_settings', return_value=settings):
service = PushNotificationService(db=mock_db)
assert service.is_enabled() is False
def test_get_public_key(self, push_service):
"""Service returns public key when enabled."""
assert push_service.get_public_key() == "test-public-key"
def test_get_public_key_returns_none_when_disabled(self, mock_db):
"""Service returns None for public key when disabled."""
settings = Mock()
settings.enable_push_notifications = False
settings.get_secret = Mock(return_value="key")
with patch('backend.services.push_notification_service.get_settings', return_value=settings):
service = PushNotificationService(db=mock_db)
assert service.get_public_key() is None
class TestSendPush:
"""Tests for sending push notifications."""
@pytest.mark.asyncio
async def test_send_push_skips_when_disabled(self, mock_db):
"""send_push returns 0 when push notifications disabled."""
settings = Mock()
settings.enable_push_notifications = False
settings.get_secret = Mock(return_value=None)
with patch('backend.services.push_notification_service.get_settings', return_value=settings):
service = PushNotificationService(db=mock_db)
result = await service.send_push("test@example.com", "Title", "Body")
assert result == 0
@pytest.mark.asyncio
async def test_send_push_no_user(self, push_service):
"""send_push returns 0 when user not found."""
# Mock user not existing
push_service.db.collection.return_value.document.return_value.get.return_value.exists = False
result = await push_service.send_push("unknown@example.com", "Title", "Body")
assert result == 0
@pytest.mark.asyncio
async def test_send_push_no_subscriptions(self, push_service):
"""send_push returns 0 when user has no subscriptions."""
# Mock user with no subscriptions
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"push_subscriptions": []}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.send_push("test@example.com", "Title", "Body")
assert result == 0
@pytest.mark.asyncio
async def test_send_push_success(self, push_service):
"""send_push successfully sends to subscription."""
# Mock user with subscription
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1", "auth": "key2"},
"device_name": "Test Device"
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.push_notification_service.webpush') as mock_webpush:
result = await push_service.send_push("test@example.com", "Title", "Body")
assert result == 1
mock_webpush.assert_called_once()
call_args = mock_webpush.call_args
assert call_args[1]["subscription_info"]["endpoint"] == "https://push.example.com/endpoint"
@pytest.mark.asyncio
async def test_send_push_removes_gone_subscription(self, push_service):
"""send_push removes subscription when 410 Gone returned."""
from pywebpush import WebPushException
# Mock user with subscription
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1", "auth": "key2"},
"device_name": "Test Device"
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
# Mock webpush to raise 410 error
mock_response = Mock()
mock_response.status_code = 410
error = WebPushException("Gone", response=mock_response)
with patch('backend.services.push_notification_service.webpush', side_effect=error):
result = await push_service.send_push("test@example.com", "Title", "Body")
assert result == 0
# Verify invalid subscription was cleaned up
push_service.db.collection.return_value.document.return_value.update.assert_called()
class TestSubscriptionManagement:
"""Tests for adding, removing, and listing subscriptions."""
@pytest.mark.asyncio
async def test_add_subscription_new_user(self, push_service):
"""add_subscription returns False for non-existent user."""
push_service.db.collection.return_value.document.return_value.get.return_value.exists = False
result = await push_service.add_subscription(
"unknown@example.com",
"https://push.example.com/endpoint",
{"p256dh": "key1", "auth": "key2"},
"Test Device"
)
assert result is False
@pytest.mark.asyncio
async def test_add_subscription_success(self, push_service):
"""add_subscription adds new subscription."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"push_subscriptions": []}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.add_subscription(
"test@example.com",
"https://push.example.com/endpoint",
{"p256dh": "key1", "auth": "key2"},
"Test Device"
)
assert result is True
push_service.db.collection.return_value.document.return_value.update.assert_called_once()
@pytest.mark.asyncio
async def test_add_subscription_updates_existing(self, push_service):
"""add_subscription updates existing subscription with same endpoint."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "old-key", "auth": "old-auth"},
"device_name": "Old Device"
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.add_subscription(
"test@example.com",
"https://push.example.com/endpoint",
{"p256dh": "new-key", "auth": "new-auth"},
"New Device"
)
assert result is True
# Verify update was called (subscription replaced, not added)
update_call = push_service.db.collection.return_value.document.return_value.update.call_args
subs = update_call[0][0]["push_subscriptions"]
assert len(subs) == 1
assert subs[0]["device_name"] == "New Device"
@pytest.mark.asyncio
async def test_add_subscription_enforces_max_limit(self, push_service):
"""add_subscription removes oldest when max exceeded."""
# Create 5 existing subscriptions
existing_subs = [
{
"endpoint": f"https://push.example.com/endpoint{i}",
"keys": {"p256dh": "key", "auth": "auth"},
"device_name": f"Device {i}",
"created_at": f"2024-01-0{i+1}T00:00:00Z"
}
for i in range(5)
]
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"push_subscriptions": existing_subs}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.add_subscription(
"test@example.com",
"https://push.example.com/new-endpoint",
{"p256dh": "new-key", "auth": "new-auth"},
"New Device"
)
assert result is True
# Verify oldest was removed (max 5 subscriptions)
update_call = push_service.db.collection.return_value.document.return_value.update.call_args
subs = update_call[0][0]["push_subscriptions"]
assert len(subs) == 5
@pytest.mark.asyncio
async def test_remove_subscription_success(self, push_service):
"""remove_subscription removes existing subscription."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key", "auth": "auth"},
"device_name": "Test Device"
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.remove_subscription(
"test@example.com",
"https://push.example.com/endpoint"
)
assert result is True
update_call = push_service.db.collection.return_value.document.return_value.update.call_args
subs = update_call[0][0]["push_subscriptions"]
assert len(subs) == 0
@pytest.mark.asyncio
async def test_remove_subscription_not_found(self, push_service):
"""remove_subscription returns False when subscription not found."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"push_subscriptions": []}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.remove_subscription(
"test@example.com",
"https://push.example.com/unknown-endpoint"
)
assert result is False
@pytest.mark.asyncio
async def test_list_subscriptions_success(self, push_service):
"""list_subscriptions returns user's subscriptions."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [
{
"endpoint": "https://push.example.com/endpoint1",
"keys": {"p256dh": "key", "auth": "auth"},
"device_name": "Device 1",
"created_at": "2024-01-01T00:00:00Z",
"last_used_at": None
},
{
"endpoint": "https://push.example.com/endpoint2",
"keys": {"p256dh": "key", "auth": "auth"},
"device_name": "Device 2",
"created_at": "2024-01-02T00:00:00Z",
"last_used_at": "2024-01-03T00:00:00Z"
}
]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
result = await push_service.list_subscriptions("test@example.com")
assert len(result) == 2
assert result[0]["device_name"] == "Device 1"
assert result[1]["device_name"] == "Device 2"
# Verify keys are NOT included in response (security)
assert "keys" not in result[0]
class TestNotificationFormatting:
"""Tests for blocking and completion notification formatting."""
@pytest.mark.asyncio
async def test_send_blocking_notification_lyrics(self, push_service):
"""send_blocking_notification formats lyrics review notification."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key", "auth": "auth"}
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
job = {
"job_id": "test-job-123",
"user_email": "test@example.com",
"artist": "Test Artist",
"title": "Test Song"
}
with patch('backend.services.push_notification_service.webpush') as mock_webpush:
await push_service.send_blocking_notification(job, "lyrics")
call_args = mock_webpush.call_args
import json
payload = json.loads(call_args[1]["data"])
assert payload["title"] == "Review Lyrics"
assert "Test Song" in payload["body"]
assert "Test Artist" in payload["body"]
assert "/review/test-job-123" in payload["url"]
@pytest.mark.asyncio
async def test_send_blocking_notification_instrumental(self, push_service):
"""send_blocking_notification formats instrumental selection notification."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key", "auth": "auth"}
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
job = {
"job_id": "test-job-123",
"user_email": "test@example.com",
"artist": "Test Artist",
"title": "Test Song"
}
with patch('backend.services.push_notification_service.webpush') as mock_webpush:
await push_service.send_blocking_notification(job, "instrumental")
call_args = mock_webpush.call_args
import json
payload = json.loads(call_args[1]["data"])
assert payload["title"] == "Select Instrumental"
assert "/instrumental/test-job-123" in payload["url"]
@pytest.mark.asyncio
async def test_send_completion_notification(self, push_service):
"""send_completion_notification formats completion notification."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"push_subscriptions": [{
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key", "auth": "auth"}
}]
}
push_service.db.collection.return_value.document.return_value.get.return_value = mock_doc
job = {
"job_id": "test-job-123",
"user_email": "test@example.com",
"artist": "Test Artist",
"title": "Test Song"
}
with patch('backend.services.push_notification_service.webpush') as mock_webpush:
await push_service.send_completion_notification(job)
call_args = mock_webpush.call_args
import json
payload = json.loads(call_args[1]["data"])
assert payload["title"] == "Video Ready!"
assert "Test Song" in payload["body"]
assert "Test Artist" in payload["body"]
assert "download" in payload["body"].lower()
class TestSingleton:
"""Tests for singleton pattern."""
def test_get_push_notification_service_returns_singleton(self):
"""get_push_notification_service returns same instance."""
# Reset singleton for test
import backend.services.push_notification_service as module
module._push_service = None
with patch('backend.services.push_notification_service.get_settings') as mock_get_settings:
mock_settings = Mock()
mock_settings.enable_push_notifications = False
mock_settings.get_secret = Mock(return_value=None)
mock_get_settings.return_value = mock_settings
service1 = get_push_notification_service()
service2 = get_push_notification_service()
assert service1 is service2
# Clean up
module._push_service = None
-357
View File
@@ -1,357 +0,0 @@
"""
Tests for Push Notification API routes.
Tests the /api/push/* endpoints for subscription management.
"""
import pytest
from unittest.mock import Mock, patch, AsyncMock
from fastapi.testclient import TestClient
from dataclasses import dataclass
from typing import Optional
from backend.main import app
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import UserType, AuthResult
@pytest.fixture
def client():
"""Create test client."""
return TestClient(app)
@pytest.fixture
def mock_auth_result():
"""Create a mock AuthResult for regular user."""
return AuthResult(
is_valid=True,
user_type=UserType.UNLIMITED,
remaining_uses=-1,
message="Valid",
user_email="test@example.com",
is_admin=False,
)
@pytest.fixture
def mock_admin_auth_result():
"""Create a mock AuthResult for admin user."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="Valid",
user_email="admin@nomadkaraoke.com",
is_admin=True,
)
class TestGetVapidPublicKey:
"""Tests for GET /api/push/vapid-public-key."""
def test_returns_disabled_when_feature_off(self, client):
"""Returns enabled=false when push notifications disabled."""
with patch('backend.api.routes.push.get_settings') as mock_settings:
mock_settings.return_value.enable_push_notifications = False
response = client.get("/api/push/vapid-public-key")
assert response.status_code == 200
data = response.json()
assert data["enabled"] is False
assert data["vapid_public_key"] is None
def test_returns_key_when_enabled(self, client):
"""Returns public key when push notifications enabled."""
with patch('backend.api.routes.push.get_settings') as mock_settings, \
patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_settings.return_value.enable_push_notifications = True
mock_service.return_value.get_public_key.return_value = "test-public-key-123"
response = client.get("/api/push/vapid-public-key")
assert response.status_code == 200
data = response.json()
assert data["enabled"] is True
assert data["vapid_public_key"] == "test-public-key-123"
class TestSubscribe:
"""Tests for POST /api/push/subscribe."""
def test_requires_authentication(self, client):
"""Returns 401 when not authenticated."""
# Clear any auth overrides to test real auth
app.dependency_overrides.clear()
response = client.post(
"/api/push/subscribe",
json={
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1", "auth": "key2"}
}
)
assert response.status_code == 401
def test_returns_503_when_disabled(self, client, mock_auth_result):
"""Returns 503 when push notifications disabled."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_settings') as mock_settings:
mock_settings.return_value.enable_push_notifications = False
response = client.post(
"/api/push/subscribe",
json={
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1", "auth": "key2"}
}
)
assert response.status_code == 503
assert "not enabled" in response.json()["detail"].lower()
finally:
app.dependency_overrides.clear()
def test_validates_required_keys(self, client, mock_auth_result):
"""Returns 400 when keys missing."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_settings') as mock_settings:
mock_settings.return_value.enable_push_notifications = True
response = client.post(
"/api/push/subscribe",
json={
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1"} # Missing auth
}
)
assert response.status_code == 400
assert "missing" in response.json()["detail"].lower()
finally:
app.dependency_overrides.clear()
def test_successful_subscription(self, client, mock_auth_result):
"""Successfully subscribes user to push notifications."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_settings') as mock_settings, \
patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_settings.return_value.enable_push_notifications = True
mock_service.return_value.add_subscription = AsyncMock(return_value=True)
response = client.post(
"/api/push/subscribe",
json={
"endpoint": "https://push.example.com/endpoint",
"keys": {"p256dh": "key1", "auth": "key2"},
"device_name": "Test Device"
}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
mock_service.return_value.add_subscription.assert_called_once_with(
user_email="test@example.com",
endpoint="https://push.example.com/endpoint",
keys={"p256dh": "key1", "auth": "key2"},
device_name="Test Device"
)
finally:
app.dependency_overrides.clear()
class TestUnsubscribe:
"""Tests for POST /api/push/unsubscribe."""
def test_requires_authentication(self, client):
"""Returns 401 when not authenticated."""
app.dependency_overrides.clear()
response = client.post(
"/api/push/unsubscribe",
json={"endpoint": "https://push.example.com/endpoint"}
)
assert response.status_code == 401
def test_successful_unsubscribe(self, client, mock_auth_result):
"""Successfully unsubscribes from push notifications."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_service.return_value.remove_subscription = AsyncMock(return_value=True)
response = client.post(
"/api/push/unsubscribe",
json={"endpoint": "https://push.example.com/endpoint"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
mock_service.return_value.remove_subscription.assert_called_once()
finally:
app.dependency_overrides.clear()
def test_unsubscribe_not_found_succeeds(self, client, mock_auth_result):
"""Returns success even when subscription not found."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_service.return_value.remove_subscription = AsyncMock(return_value=False)
response = client.post(
"/api/push/unsubscribe",
json={"endpoint": "https://push.example.com/unknown"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
finally:
app.dependency_overrides.clear()
class TestListSubscriptions:
"""Tests for GET /api/push/subscriptions."""
def test_requires_authentication(self, client):
"""Returns 401 when not authenticated."""
app.dependency_overrides.clear()
response = client.get("/api/push/subscriptions")
assert response.status_code == 401
def test_returns_user_subscriptions(self, client, mock_auth_result):
"""Returns list of user's subscriptions."""
async def override_auth():
return mock_auth_result
app.dependency_overrides[require_auth] = override_auth
try:
with patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_service.return_value.list_subscriptions = AsyncMock(return_value=[
{
"endpoint": "https://push.example.com/endpoint1",
"device_name": "Device 1",
"created_at": "2024-01-01T00:00:00Z",
"last_used_at": None
},
{
"endpoint": "https://push.example.com/endpoint2",
"device_name": "Device 2",
"created_at": "2024-01-02T00:00:00Z",
"last_used_at": "2024-01-03T00:00:00Z"
}
])
response = client.get("/api/push/subscriptions")
assert response.status_code == 200
data = response.json()
assert data["count"] == 2
assert len(data["subscriptions"]) == 2
assert data["subscriptions"][0]["device_name"] == "Device 1"
finally:
app.dependency_overrides.clear()
class TestTestNotification:
"""Tests for POST /api/push/test."""
def test_requires_admin(self, client, mock_auth_result):
"""Returns 403 when not admin."""
# Use a non-admin auth result
async def override_admin():
from fastapi import HTTPException
raise HTTPException(status_code=403, detail="Admin access required")
app.dependency_overrides[require_admin] = override_admin
try:
response = client.post(
"/api/push/test",
json={"title": "Test", "body": "Test message"}
)
assert response.status_code == 403
finally:
app.dependency_overrides.clear()
def test_returns_503_when_disabled(self, client, mock_admin_auth_result):
"""Returns 503 when push notifications disabled."""
async def override_admin():
return mock_admin_auth_result
app.dependency_overrides[require_admin] = override_admin
try:
with patch('backend.api.routes.push.get_settings') as mock_settings:
mock_settings.return_value.enable_push_notifications = False
response = client.post(
"/api/push/test",
json={"title": "Test", "body": "Test message"}
)
assert response.status_code == 503
finally:
app.dependency_overrides.clear()
def test_sends_test_notification(self, client, mock_admin_auth_result):
"""Successfully sends test notification to admin."""
async def override_admin():
return mock_admin_auth_result
app.dependency_overrides[require_admin] = override_admin
try:
with patch('backend.api.routes.push.get_settings') as mock_settings, \
patch('backend.api.routes.push.get_push_notification_service') as mock_service:
mock_settings.return_value.enable_push_notifications = True
mock_service.return_value.send_push = AsyncMock(return_value=2)
response = client.post(
"/api/push/test",
json={"title": "Test Title", "body": "Test Body"}
)
assert response.status_code == 200
data = response.json()
assert data["status"] == "success"
assert data["sent_count"] == 2
mock_service.return_value.send_push.assert_called_once_with(
user_email="admin@nomadkaraoke.com",
title="Test Title",
body="Test Body",
url="/app/",
tag="test"
)
finally:
app.dependency_overrides.clear()
-396
View File
@@ -1,396 +0,0 @@
"""
Unit tests for RateLimitService.
Tests rate limiting logic for jobs, YouTube uploads, and beta enrollments.
"""
import pytest
from unittest.mock import Mock, MagicMock, patch
from datetime import datetime, timezone
# Mock Google Cloud before imports
import sys
sys.modules['google.cloud.firestore'] = MagicMock()
sys.modules['google.cloud.storage'] = MagicMock()
class TestRateLimitService:
"""Test RateLimitService functionality."""
@pytest.fixture
def mock_db(self):
"""Create a mock Firestore client."""
mock = MagicMock()
return mock
@pytest.fixture
def mock_settings(self):
"""Create mock settings."""
settings = Mock()
settings.enable_rate_limiting = True
settings.rate_limit_jobs_per_day = 5
settings.rate_limit_youtube_uploads_per_day = 10
settings.rate_limit_beta_ip_per_day = 1
return settings
@pytest.fixture
def rate_limit_service(self, mock_db, mock_settings):
"""Create RateLimitService instance with mocks."""
with patch('backend.services.rate_limit_service.settings', mock_settings):
from backend.services.rate_limit_service import RateLimitService
service = RateLimitService(db=mock_db)
return service
# =========================================================================
# User Job Rate Limiting Tests
# =========================================================================
def test_check_user_job_limit_under_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test that user under limit is allowed."""
# Mock: user has 2 jobs today
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 2}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("user@example.com")
assert allowed is True
assert remaining == 3 # 5 - 2
assert "3 jobs remaining" in message
def test_check_user_job_limit_at_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test that user at limit is blocked."""
# Mock: user has 5 jobs today (at limit)
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 5}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("user@example.com")
assert allowed is False
assert remaining == 0
assert "Daily job limit reached" in message
def test_check_user_job_limit_over_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test that user over limit is blocked."""
# Mock: user somehow has 7 jobs (over limit)
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 7}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("user@example.com")
assert allowed is False
assert remaining == 0
def test_check_user_job_limit_no_jobs_yet(self, rate_limit_service, mock_db, mock_settings):
"""Test user with no jobs today is allowed."""
# Mock: no document exists
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("newuser@example.com")
assert allowed is True
assert remaining == 5
def test_check_user_job_limit_with_admin_bypass(self, rate_limit_service, mock_db, mock_settings):
"""Test that admins bypass job limits."""
# Mock: user at limit
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 10}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit(
"admin@example.com", is_admin=True
)
assert allowed is True
assert remaining == -1 # Unlimited for admins
def test_check_user_job_limit_with_override_bypass(self, rate_limit_service, mock_db, mock_settings):
"""Test that users with bypass override are allowed."""
# Mock: user at limit
mock_job_doc = Mock()
mock_job_doc.exists = True
mock_job_doc.to_dict.return_value = {"count": 10}
# Mock: user has bypass override
mock_override_doc = Mock()
mock_override_doc.exists = True
mock_override_doc.to_dict.return_value = {"bypass_job_limit": True}
def mock_get(*args, **kwargs):
# Return different docs based on collection
return mock_override_doc if "overrides" in str(mock_db.collection.call_args) else mock_job_doc
mock_db.collection.return_value.document.return_value.get.side_effect = [mock_override_doc, mock_job_doc]
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("vip@example.com")
assert allowed is True
def test_check_user_job_limit_with_custom_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test that custom limit from override is respected."""
# Mock: user has 8 jobs
mock_job_doc = Mock()
mock_job_doc.exists = True
mock_job_doc.to_dict.return_value = {"count": 8}
# Mock: user has custom limit of 20
mock_override_doc = Mock()
mock_override_doc.exists = True
mock_override_doc.to_dict.return_value = {"bypass_job_limit": False, "custom_daily_job_limit": 20}
mock_db.collection.return_value.document.return_value.get.side_effect = [mock_override_doc, mock_job_doc]
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("custom@example.com")
assert allowed is True
assert remaining == 12 # 20 - 8
def test_check_user_job_limit_rate_limiting_disabled(self, rate_limit_service, mock_settings):
"""Test that rate limiting can be disabled."""
mock_settings.enable_rate_limiting = False
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_user_job_limit("user@example.com")
assert allowed is True
assert remaining == -1
assert "Rate limiting disabled" in message
# =========================================================================
# YouTube Upload Rate Limiting Tests
# =========================================================================
def test_check_youtube_limit_under_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test YouTube upload under limit is allowed."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 3}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_youtube_upload_limit()
assert allowed is True
assert remaining == 7 # 10 - 3
def test_check_youtube_limit_at_limit(self, rate_limit_service, mock_db, mock_settings):
"""Test YouTube upload at limit is blocked."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 10}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_youtube_upload_limit()
assert allowed is False
assert remaining == 0
assert "Daily YouTube upload limit reached" in message
def test_check_youtube_limit_no_uploads_yet(self, rate_limit_service, mock_db, mock_settings):
"""Test YouTube upload with no uploads today is allowed."""
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_youtube_upload_limit()
assert allowed is True
assert remaining == 10
def test_check_youtube_limit_zero_configured(self, rate_limit_service, mock_db, mock_settings):
"""Test that zero limit means unlimited YouTube uploads."""
mock_settings.rate_limit_youtube_uploads_per_day = 0
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_youtube_upload_limit()
assert allowed is True
assert remaining == -1
assert "No YouTube upload limit" in message
# =========================================================================
# Beta Enrollment IP Rate Limiting Tests
# =========================================================================
def test_check_beta_ip_limit_first_enrollment(self, rate_limit_service, mock_db, mock_settings):
"""Test first enrollment from IP is allowed."""
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_beta_ip_limit("192.168.1.1")
assert allowed is True
assert remaining == 1
def test_check_beta_ip_limit_already_enrolled(self, rate_limit_service, mock_db, mock_settings):
"""Test second enrollment from same IP is blocked."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {"count": 1}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
allowed, remaining, message = rate_limit_service.check_beta_ip_limit("192.168.1.1")
assert allowed is False
assert remaining == 0
assert "Too many beta enrollments" in message
# =========================================================================
# Recording Tests
# =========================================================================
def test_record_job_creation(self, rate_limit_service, mock_db, mock_settings):
"""Test recording a job creation."""
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
with patch('backend.services.rate_limit_service.firestore') as mock_firestore:
# Mock the transactional decorator
mock_firestore.transactional = lambda f: f
rate_limit_service.record_job_creation("user@example.com", "job123")
# Verify transaction was used
mock_db.transaction.assert_called()
def test_record_youtube_upload(self, rate_limit_service, mock_db, mock_settings):
"""Test recording a YouTube upload."""
mock_transaction = Mock()
mock_db.transaction.return_value = mock_transaction
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
with patch('backend.services.rate_limit_service.settings', mock_settings):
with patch('backend.services.rate_limit_service.firestore') as mock_firestore:
mock_firestore.transactional = lambda f: f
rate_limit_service.record_youtube_upload("job123", "user@example.com")
mock_db.transaction.assert_called()
def test_record_skipped_when_disabled(self, rate_limit_service, mock_db, mock_settings):
"""Test that recording is skipped when rate limiting is disabled."""
mock_settings.enable_rate_limiting = False
with patch('backend.services.rate_limit_service.settings', mock_settings):
rate_limit_service.record_job_creation("user@example.com", "job123")
# Should not call Firestore
mock_db.collection.assert_not_called()
# =========================================================================
# Override Management Tests
# =========================================================================
def test_get_user_override_exists(self, rate_limit_service, mock_db):
"""Test getting an existing user override."""
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"email": "vip@example.com",
"bypass_job_limit": True,
"reason": "VIP user"
}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
override = rate_limit_service.get_user_override("vip@example.com")
assert override is not None
assert override["bypass_job_limit"] is True
def test_get_user_override_not_exists(self, rate_limit_service, mock_db):
"""Test getting a non-existent user override."""
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
override = rate_limit_service.get_user_override("regular@example.com")
assert override is None
def test_set_user_override(self, rate_limit_service, mock_db):
"""Test setting a user override."""
rate_limit_service.set_user_override(
user_email="vip@example.com",
bypass_job_limit=True,
reason="Special access",
admin_email="admin@example.com"
)
mock_db.collection.return_value.document.return_value.set.assert_called_once()
def test_remove_user_override_exists(self, rate_limit_service, mock_db):
"""Test removing an existing user override."""
mock_doc = Mock()
mock_doc.exists = True
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
result = rate_limit_service.remove_user_override("vip@example.com", "admin@example.com")
assert result is True
mock_db.collection.return_value.document.return_value.delete.assert_called_once()
def test_remove_user_override_not_exists(self, rate_limit_service, mock_db):
"""Test removing a non-existent user override."""
mock_doc = Mock()
mock_doc.exists = False
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
result = rate_limit_service.remove_user_override("regular@example.com", "admin@example.com")
assert result is False
class TestRateLimitExceededError:
"""Test RateLimitExceededError exception."""
def test_exception_attributes(self):
"""Test exception has all expected attributes."""
from backend.exceptions import RateLimitExceededError
exc = RateLimitExceededError(
message="Limit exceeded",
limit_type="job",
remaining_seconds=3600,
current_count=5,
limit_value=5
)
assert exc.message == "Limit exceeded"
assert exc.limit_type == "job"
assert exc.remaining_seconds == 3600
assert exc.current_count == 5
assert exc.limit_value == 5
def test_exception_str(self):
"""Test exception string representation."""
from backend.exceptions import RateLimitExceededError
exc = RateLimitExceededError(message="Test message")
assert str(exc) == "Test message"
-392
View File
@@ -1,392 +0,0 @@
"""
Unit tests for rate limits admin API endpoints.
Tests the rate limit statistics, blocklist management, and user override endpoints.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from fastapi.testclient import TestClient
from fastapi import FastAPI
from datetime import datetime, timezone
from backend.api.routes.rate_limits import router
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult, UserType
# Create a test app with the rate_limits router
app = FastAPI()
app.include_router(router, prefix="/api")
def get_mock_admin():
"""Override for require_admin dependency."""
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="Admin access granted",
user_email="admin@example.com",
is_admin=True,
)
# Override the require_admin dependency
app.dependency_overrides[require_admin] = get_mock_admin
@pytest.fixture
def client():
"""Create a test client."""
return TestClient(app)
@pytest.fixture
def mock_settings():
"""Create mock settings."""
settings = Mock()
settings.enable_rate_limiting = True
settings.rate_limit_jobs_per_day = 5
settings.rate_limit_youtube_uploads_per_day = 10
settings.rate_limit_beta_ip_per_day = 1
return settings
class TestGetRateLimitStats:
"""Tests for GET /api/admin/rate-limits/stats endpoint."""
def test_returns_stats(self, client, mock_settings):
"""Test successful stats retrieval."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls, \
patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs, \
patch('backend.api.routes.rate_limits.settings', mock_settings):
# Setup rate limit service mock
mock_rls = Mock()
mock_rls.get_youtube_uploads_today.return_value = 3
mock_rls.get_all_overrides.return_value = {"user1@example.com": {}}
mock_get_rls.return_value = mock_rls
# Setup email validation service mock
mock_evs = Mock()
mock_evs.get_blocklist_stats.return_value = {
"disposable_domains_count": 100,
"blocked_emails_count": 5,
"blocked_ips_count": 2,
"default_disposable_domains_count": 130,
}
mock_get_evs.return_value = mock_evs
response = client.get(
"/api/admin/rate-limits/stats",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["jobs_per_day_limit"] == 5
assert data["youtube_uploads_per_day_limit"] == 10
assert data["youtube_uploads_today"] == 3
assert data["youtube_uploads_remaining"] == 7
assert data["disposable_domains_count"] == 100
assert data["total_overrides"] == 1
class TestGetUserRateLimitStatus:
"""Tests for GET /api/admin/rate-limits/users/{email} endpoint."""
def test_returns_user_status(self, client, mock_settings):
"""Test successful user status retrieval."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls, \
patch('backend.api.routes.rate_limits.settings', mock_settings):
mock_rls = Mock()
mock_rls.get_user_job_count_today.return_value = 2
mock_rls.get_user_override.return_value = None
mock_get_rls.return_value = mock_rls
response = client.get(
"/api/admin/rate-limits/users/user@example.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["email"] == "user@example.com"
assert data["jobs_today"] == 2
assert data["jobs_limit"] == 5
assert data["jobs_remaining"] == 3
assert data["has_bypass"] is False
def test_returns_user_with_bypass(self, client, mock_settings):
"""Test user with bypass override."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls, \
patch('backend.api.routes.rate_limits.settings', mock_settings):
mock_rls = Mock()
mock_rls.get_user_job_count_today.return_value = 10
mock_rls.get_user_override.return_value = {
"bypass_job_limit": True,
"reason": "VIP user"
}
mock_get_rls.return_value = mock_rls
response = client.get(
"/api/admin/rate-limits/users/vip@example.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["has_bypass"] is True
assert data["bypass_reason"] == "VIP user"
assert data["jobs_remaining"] == -1 # Unlimited
class TestBlocklistEndpoints:
"""Tests for blocklist management endpoints."""
def test_get_blocklists(self, client):
"""Test getting all blocklists."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs, \
patch('backend.services.firestore_service.get_firestore_client') as mock_get_db:
mock_evs = Mock()
mock_evs.get_blocklist_config.return_value = {
"disposable_domains": {"tempmail.com", "mailinator.com"},
"blocked_emails": {"spammer@example.com"},
"blocked_ips": {"192.168.1.100"},
}
mock_get_evs.return_value = mock_evs
# Mock Firestore for metadata
mock_db = Mock()
mock_doc = Mock()
mock_doc.exists = True
mock_doc.to_dict.return_value = {
"updated_at": datetime.now(timezone.utc),
"updated_by": "admin@example.com"
}
mock_db.collection.return_value.document.return_value.get.return_value = mock_doc
mock_get_db.return_value = mock_db
response = client.get(
"/api/admin/rate-limits/blocklists",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert "tempmail.com" in data["disposable_domains"]
assert "spammer@example.com" in data["blocked_emails"]
def test_add_disposable_domain(self, client):
"""Test adding a disposable domain."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs:
mock_evs = Mock()
mock_evs.add_disposable_domain.return_value = True
mock_get_evs.return_value = mock_evs
response = client.post(
"/api/admin/rate-limits/blocklists/disposable-domains",
json={"domain": "newtemp.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["success"] is True
mock_evs.add_disposable_domain.assert_called_once_with("newtemp.com", "admin@example.com")
def test_add_disposable_domain_invalid(self, client):
"""Test adding invalid domain."""
response = client.post(
"/api/admin/rate-limits/blocklists/disposable-domains",
json={"domain": "invalid"}, # No dot
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 400
def test_remove_disposable_domain(self, client):
"""Test removing a disposable domain."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs:
mock_evs = Mock()
mock_evs.remove_disposable_domain.return_value = True
mock_get_evs.return_value = mock_evs
response = client.delete(
"/api/admin/rate-limits/blocklists/disposable-domains/tempmail.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
mock_evs.remove_disposable_domain.assert_called_once()
def test_remove_disposable_domain_not_found(self, client):
"""Test removing non-existent domain."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs:
mock_evs = Mock()
mock_evs.remove_disposable_domain.return_value = False
mock_get_evs.return_value = mock_evs
response = client.delete(
"/api/admin/rate-limits/blocklists/disposable-domains/notfound.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
def test_add_blocked_email(self, client):
"""Test adding a blocked email."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs:
mock_evs = Mock()
mock_evs.add_blocked_email.return_value = True
mock_get_evs.return_value = mock_evs
response = client.post(
"/api/admin/rate-limits/blocklists/blocked-emails",
json={"email": "spammer@example.com"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
def test_add_blocked_ip(self, client):
"""Test adding a blocked IP."""
with patch('backend.api.routes.rate_limits.get_email_validation_service') as mock_get_evs:
mock_evs = Mock()
mock_evs.add_blocked_ip.return_value = True
mock_get_evs.return_value = mock_evs
response = client.post(
"/api/admin/rate-limits/blocklists/blocked-ips",
json={"ip_address": "192.168.1.100"},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
class TestUserOverrideEndpoints:
"""Tests for user override management endpoints."""
def test_get_all_overrides(self, client):
"""Test getting all user overrides."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls:
mock_rls = Mock()
mock_rls.get_all_overrides.return_value = {
"vip@example.com": {
"bypass_job_limit": True,
"custom_daily_job_limit": None,
"reason": "VIP user",
"created_by": "admin@example.com",
"created_at": datetime.now(timezone.utc),
}
}
mock_get_rls.return_value = mock_rls
response = client.get(
"/api/admin/rate-limits/overrides",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
data = response.json()
assert data["total"] == 1
assert len(data["overrides"]) == 1
assert data["overrides"][0]["email"] == "vip@example.com"
def test_set_user_override(self, client):
"""Test setting a user override."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls:
mock_rls = Mock()
mock_get_rls.return_value = mock_rls
response = client.put(
"/api/admin/rate-limits/overrides/user@example.com",
json={
"bypass_job_limit": True,
"reason": "Special access granted"
},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
mock_rls.set_user_override.assert_called_once_with(
user_email="user@example.com",
bypass_job_limit=True,
custom_daily_job_limit=None,
reason="Special access granted",
admin_email="admin@example.com",
)
def test_set_user_override_with_custom_limit(self, client):
"""Test setting a user override with custom limit."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls:
mock_rls = Mock()
mock_get_rls.return_value = mock_rls
response = client.put(
"/api/admin/rate-limits/overrides/user@example.com",
json={
"bypass_job_limit": False,
"custom_daily_job_limit": 20,
"reason": "High volume user"
},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
mock_rls.set_user_override.assert_called_once()
def test_set_user_override_missing_reason(self, client):
"""Test setting override without reason fails."""
response = client.put(
"/api/admin/rate-limits/overrides/user@example.com",
json={
"bypass_job_limit": True,
"reason": "ab" # Too short
},
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 400
def test_remove_user_override(self, client):
"""Test removing a user override."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls:
mock_rls = Mock()
mock_rls.remove_user_override.return_value = True
mock_get_rls.return_value = mock_rls
response = client.delete(
"/api/admin/rate-limits/overrides/user@example.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 200
def test_remove_user_override_not_found(self, client):
"""Test removing non-existent override."""
with patch('backend.api.routes.rate_limits.get_rate_limit_service') as mock_get_rls:
mock_rls = Mock()
mock_rls.remove_user_override.return_value = False
mock_get_rls.return_value = mock_rls
response = client.delete(
"/api/admin/rate-limits/overrides/notfound@example.com",
headers={"Authorization": "Bearer admin-token"}
)
assert response.status_code == 404
+1 -2
View File
@@ -149,8 +149,7 @@ class TestUploadSubmissionRequest:
# CDG/TXT disabled by default (requires style config)
assert request.enable_cdg is False
assert request.enable_txt is False
# YouTube upload default is None (use server default)
assert request.enable_youtube_upload is None
assert request.enable_youtube_upload is False
class TestStartReviewRequest:
+1 -1
View File
@@ -204,7 +204,7 @@ class TestAddLyricsEndpoint:
def test_add_lyrics_uses_correction_operations(self):
"""Verify CorrectionOperations.add_lyrics_source is available."""
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
from lyrics_transcriber.correction.operations import CorrectionOperations
# The method should exist
assert hasattr(CorrectionOperations, 'add_lyrics_source')
-119
View File
@@ -1,119 +0,0 @@
"""Tests for SpaCy preloader service."""
import pytest
from unittest.mock import patch, MagicMock
from backend.services.spacy_preloader import (
preload_spacy_model,
get_preloaded_model,
is_model_preloaded,
clear_preloaded_models,
)
class TestSpacyPreloader:
"""Tests for SpaCy preloading functionality."""
def setup_method(self):
"""Clear preloaded models before each test."""
clear_preloaded_models()
def teardown_method(self):
"""Clear preloaded models after each test."""
clear_preloaded_models()
def test_preload_spacy_model_loads_and_stores(self):
"""GIVEN no preloaded models
WHEN preload_spacy_model is called
THEN model should be loaded and stored in singleton."""
mock_nlp = MagicMock()
with patch("spacy.load", return_value=mock_nlp) as mock_load:
preload_spacy_model("en_core_web_sm")
mock_load.assert_called_once_with("en_core_web_sm")
assert is_model_preloaded("en_core_web_sm")
assert get_preloaded_model("en_core_web_sm") is mock_nlp
def test_preload_is_idempotent(self):
"""GIVEN a model already preloaded
WHEN preload_spacy_model is called again
THEN model should not be reloaded."""
mock_nlp = MagicMock()
with patch("spacy.load", return_value=mock_nlp) as mock_load:
preload_spacy_model("en_core_web_sm")
preload_spacy_model("en_core_web_sm") # Second call
# Should only load once
assert mock_load.call_count == 1
def test_get_preloaded_model_returns_none_if_not_loaded(self):
"""GIVEN no preloaded models
WHEN get_preloaded_model is called
THEN should return None."""
assert get_preloaded_model("en_core_web_sm") is None
assert not is_model_preloaded("en_core_web_sm")
def test_preload_different_models(self):
"""GIVEN no preloaded models
WHEN preload_spacy_model is called with different model names
THEN each model should be loaded and stored separately."""
mock_nlp_sm = MagicMock(name="en_core_web_sm")
mock_nlp_md = MagicMock(name="en_core_web_md")
def mock_load(model_name):
if model_name == "en_core_web_sm":
return mock_nlp_sm
elif model_name == "en_core_web_md":
return mock_nlp_md
raise ValueError(f"Unknown model: {model_name}")
with patch("spacy.load", side_effect=mock_load):
preload_spacy_model("en_core_web_sm")
preload_spacy_model("en_core_web_md")
assert is_model_preloaded("en_core_web_sm")
assert is_model_preloaded("en_core_web_md")
assert get_preloaded_model("en_core_web_sm") is mock_nlp_sm
assert get_preloaded_model("en_core_web_md") is mock_nlp_md
def test_clear_preloaded_models(self):
"""GIVEN preloaded models
WHEN clear_preloaded_models is called
THEN all models should be removed."""
mock_nlp = MagicMock()
with patch("spacy.load", return_value=mock_nlp):
preload_spacy_model("en_core_web_sm")
assert is_model_preloaded("en_core_web_sm")
clear_preloaded_models()
assert not is_model_preloaded("en_core_web_sm")
assert get_preloaded_model("en_core_web_sm") is None
def test_preload_failure_raises_exception(self):
"""GIVEN spacy.load raises an exception
WHEN preload_spacy_model is called
THEN exception should be propagated."""
with patch(
"spacy.load",
side_effect=OSError("Model not found"),
):
with pytest.raises(OSError, match="Model not found"):
preload_spacy_model("nonexistent_model")
# Model should not be marked as preloaded
assert not is_model_preloaded("nonexistent_model")
def test_preload_uses_default_model_name(self):
"""GIVEN no model name specified
WHEN preload_spacy_model is called without arguments
THEN should use en_core_web_sm as default."""
mock_nlp = MagicMock()
with patch("spacy.load", return_value=mock_nlp) as mock_load:
preload_spacy_model()
mock_load.assert_called_once_with("en_core_web_sm")
assert is_model_preloaded("en_core_web_sm")
-205
View File
@@ -1,205 +0,0 @@
"""
Tests for StripeService.
Tests cover:
- Made-for-you checkout session URL generation
- Credit purchase checkout session URL generation
- Package validation
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
import os
class TestStripeServiceUrls:
"""Tests for checkout session URL generation."""
@pytest.fixture
def stripe_service(self):
"""Create a StripeService instance with mocked Stripe."""
# Set required env vars
with patch.dict(os.environ, {
'STRIPE_SECRET_KEY': 'sk_test_fake',
'FRONTEND_URL': 'https://gen.nomadkaraoke.com',
}):
# Import here to pick up env vars
from backend.services.stripe_service import StripeService
service = StripeService()
return service
def test_made_for_you_success_url_uses_frontend_url(self, stripe_service):
"""Test that made-for-you checkout uses frontend_url, not hardcoded domain."""
# Mock stripe.checkout.Session.create to capture the params
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
success, url, message = stripe_service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
)
# Verify the call was made
assert mock_create.called
call_kwargs = mock_create.call_args[1]
# Verify success_url uses frontend_url and correct path
success_url = call_kwargs['success_url']
assert success_url.startswith('https://gen.nomadkaraoke.com')
assert '/order/success' in success_url
assert 'session_id=' in success_url
def test_made_for_you_success_url_not_hardcoded_to_marketing_site(self, stripe_service):
"""Test that success_url is NOT the old hardcoded marketing site URL."""
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
stripe_service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
)
call_kwargs = mock_create.call_args[1]
success_url = call_kwargs['success_url']
# The bug was: success_url = "https://nomadkaraoke.com/order/success/"
# (note: nomadkaraoke.com WITHOUT the 'gen.' prefix - that's the marketing site)
# This should NOT be the case anymore - should use gen.nomadkaraoke.com
assert not success_url.startswith('https://nomadkaraoke.com/')
# Should use the frontend URL (gen.nomadkaraoke.com)
assert success_url.startswith('https://gen.nomadkaraoke.com')
def test_made_for_you_respects_custom_frontend_url(self):
"""Test that made-for-you checkout uses custom FRONTEND_URL if set."""
with patch.dict(os.environ, {
'STRIPE_SECRET_KEY': 'sk_test_fake',
'FRONTEND_URL': 'https://custom.example.com',
}):
from backend.services.stripe_service import StripeService
service = StripeService()
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
)
call_kwargs = mock_create.call_args[1]
success_url = call_kwargs['success_url']
assert success_url.startswith('https://custom.example.com')
def test_credit_purchase_success_url_uses_payment_success(self, stripe_service):
"""Test that credit purchase checkout uses /payment/success path."""
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
success, url, message = stripe_service.create_checkout_session(
package_id='1_credit',
user_email='user@example.com',
)
call_kwargs = mock_create.call_args[1]
success_url = call_kwargs['success_url']
# Credit purchases go to /payment/success
assert '/payment/success' in success_url
assert success_url.startswith('https://gen.nomadkaraoke.com')
def test_made_for_you_uses_order_success_path(self, stripe_service):
"""Test that made-for-you uses /order/success path (not /payment/success)."""
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
stripe_service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
)
call_kwargs = mock_create.call_args[1]
success_url = call_kwargs['success_url']
# Made-for-you orders go to /order/success (different messaging)
assert '/order/success' in success_url
# Should NOT go to /payment/success
assert '/payment/success' not in success_url
class TestStripeServiceMetadata:
"""Tests for checkout session metadata."""
@pytest.fixture
def stripe_service(self):
"""Create a StripeService instance with mocked Stripe."""
with patch.dict(os.environ, {
'STRIPE_SECRET_KEY': 'sk_test_fake',
'FRONTEND_URL': 'https://gen.nomadkaraoke.com',
}):
from backend.services.stripe_service import StripeService
return StripeService()
def test_made_for_you_includes_order_type_metadata(self, stripe_service):
"""Test that made-for-you checkout includes order_type in metadata."""
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
stripe_service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
notes='Special request',
)
call_kwargs = mock_create.call_args[1]
metadata = call_kwargs['metadata']
assert metadata['order_type'] == 'made_for_you'
assert metadata['customer_email'] == 'customer@example.com'
assert metadata['artist'] == 'Test Artist'
assert metadata['title'] == 'Test Song'
assert metadata['notes'] == 'Special request'
def test_made_for_you_truncates_long_notes(self, stripe_service):
"""Test that notes longer than 500 chars are truncated."""
with patch('stripe.checkout.Session.create') as mock_create:
mock_session = MagicMock()
mock_session.id = 'cs_test_123'
mock_session.url = 'https://checkout.stripe.com/test'
mock_create.return_value = mock_session
long_notes = 'x' * 600
stripe_service.create_made_for_you_checkout_session(
customer_email='customer@example.com',
artist='Test Artist',
title='Test Song',
notes=long_notes,
)
call_kwargs = mock_create.call_args[1]
metadata = call_kwargs['metadata']
assert len(metadata['notes']) == 500
-295
View File
@@ -1,295 +0,0 @@
"""
Unit tests for template service.
"""
import pytest
from unittest.mock import Mock, patch, MagicMock
from backend.services.template_service import (
TemplateService,
get_template_service,
DEFAULT_JOB_COMPLETION_TEMPLATE,
DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE,
DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE,
)
class TestTemplateRendering:
"""Tests for template rendering functionality."""
def test_render_template_basic(self):
"""Test basic template variable replacement."""
service = TemplateService()
template = "Hello {name}, your job {job_id} is ready!"
variables = {"name": "Alice", "job_id": "123"}
result = service.render_template(template, variables)
assert result == "Hello Alice, your job 123 is ready!"
def test_render_template_missing_variables(self):
"""Test that missing variables are replaced with empty strings."""
service = TemplateService()
template = "Hello {name}, your {thing} is {status}!"
variables = {"name": "Bob"}
result = service.render_template(template, variables)
assert result == "Hello Bob, your is !"
def test_render_template_none_values(self):
"""Test that None values are handled correctly."""
service = TemplateService()
template = "Hello {name}!"
variables = {"name": None}
result = service.render_template(template, variables)
assert result == "Hello !"
def test_render_template_removes_feedback_section_when_no_url(self):
"""Test that feedback section is removed when feedback_url is not provided."""
service = TemplateService()
template = """Thanks for your order!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Have a great day!"""
variables = {"feedback_url": None}
result = service.render_template(template, variables)
assert "feedback" not in result.lower()
assert "Thanks for your order!" in result
assert "Have a great day!" in result
def test_render_template_keeps_feedback_section_when_url_provided(self):
"""Test that feedback section is kept when feedback_url is provided."""
service = TemplateService()
template = """Thanks!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Bye!"""
variables = {"feedback_url": "https://example.com/feedback"}
result = service.render_template(template, variables)
assert "https://example.com/feedback" in result
assert "feedback" in result.lower()
class TestJobCompletionTemplate:
"""Tests for job completion template rendering."""
def test_render_job_completion_all_fields(self):
"""Test rendering job completion with all fields."""
service = TemplateService()
# Mock _fetch_template_from_gcs to return None (use default)
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion(
name="Alice",
youtube_url="https://youtube.com/watch?v=123",
dropbox_url="https://dropbox.com/folder/abc",
artist="Test Artist",
title="Test Song",
job_id="job-123",
feedback_url="https://example.com/feedback",
)
assert "Alice" in result
assert "https://youtube.com/watch?v=123" in result
assert "https://dropbox.com/folder/abc" in result
assert "https://example.com/feedback" in result
def test_render_job_completion_defaults(self):
"""Test rendering job completion with default values."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion()
assert "there" in result # Default name
assert "[YouTube URL not available]" in result
assert "[Dropbox URL not available]" in result
def test_render_job_completion_no_feedback_url(self):
"""Test rendering job completion without feedback URL removes section."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_job_completion(
name="Bob",
youtube_url="https://youtube.com/123",
dropbox_url="https://dropbox.com/abc",
)
assert "Bob" in result
# Feedback section should be removed
assert "really appreciate your feedback" not in result
class TestActionNeededTemplates:
"""Tests for action-needed template rendering."""
def test_render_action_needed_lyrics(self):
"""Test rendering lyrics review reminder."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_action_needed_lyrics(
name="Charlie",
artist="Test Artist",
title="Test Song",
review_url="https://example.com/review",
)
assert "Charlie" in result
assert "Test Artist" in result
assert "Test Song" in result
assert "https://example.com/review" in result
def test_render_action_needed_instrumental(self):
"""Test rendering instrumental selection reminder."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.render_action_needed_instrumental(
name="Diana",
artist="Test Artist",
title="Test Song",
instrumental_url="https://example.com/instrumental",
)
assert "Diana" in result
assert "Test Artist" in result
assert "Test Song" in result
assert "https://example.com/instrumental" in result
class TestGCSFetching:
"""Tests for GCS template fetching."""
def test_fetch_template_from_gcs_success(self):
"""Test successful template fetch from GCS."""
service = TemplateService()
# Mock the bucket and blob
mock_blob = Mock()
mock_blob.exists.return_value = True
mock_blob.download_as_text.return_value = "Custom template {name}"
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("test.txt")
assert result == "Custom template {name}"
mock_bucket.blob.assert_called_once_with("templates/test.txt")
def test_fetch_template_from_gcs_not_found(self):
"""Test template fetch when file doesn't exist in GCS."""
service = TemplateService()
mock_blob = Mock()
mock_blob.exists.return_value = False
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("nonexistent.txt")
assert result is None
def test_fetch_template_from_gcs_error(self):
"""Test template fetch handles errors gracefully."""
service = TemplateService()
mock_bucket = Mock()
mock_bucket.blob.side_effect = Exception("GCS error")
with patch.object(service, "_bucket", mock_bucket):
result = service._fetch_template_from_gcs("test.txt")
assert result is None
def test_get_job_completion_template_fallback(self):
"""Test that default template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_job_completion_template()
assert result == DEFAULT_JOB_COMPLETION_TEMPLATE
def test_get_action_needed_lyrics_template_fallback(self):
"""Test that default lyrics template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_action_needed_lyrics_template()
assert result == DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE
def test_get_action_needed_instrumental_template_fallback(self):
"""Test that default instrumental template is used when GCS fetch fails."""
service = TemplateService()
with patch.object(service, "_fetch_template_from_gcs", return_value=None):
result = service.get_action_needed_instrumental_template()
assert result == DEFAULT_ACTION_NEEDED_INSTRUMENTAL_TEMPLATE
class TestTemplateUpload:
"""Tests for template upload functionality."""
def test_upload_template_success(self):
"""Test successful template upload."""
service = TemplateService()
mock_blob = Mock()
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service.upload_template("test.txt", "Test content")
assert result is True
mock_bucket.blob.assert_called_once_with("templates/test.txt")
mock_blob.upload_from_string.assert_called_once_with(
"Test content", content_type="text/plain"
)
def test_upload_template_error(self):
"""Test template upload handles errors."""
service = TemplateService()
mock_blob = Mock()
mock_blob.upload_from_string.side_effect = Exception("Upload failed")
mock_bucket = Mock()
mock_bucket.blob.return_value = mock_blob
with patch.object(service, "_bucket", mock_bucket):
result = service.upload_template("test.txt", "Test content")
assert result is False
class TestGlobalInstance:
"""Tests for global instance management."""
def test_get_template_service_returns_same_instance(self):
"""Test that get_template_service returns singleton."""
# Reset global
import backend.services.template_service as ts
ts._template_service = None
service1 = get_template_service()
service2 = get_template_service()
assert service1 is service2
-350
View File
@@ -1,350 +0,0 @@
"""
Unit tests for tenant API routes.
Tests the /api/tenant/config endpoints.
Note: We need to patch get_tenant_service in both the routes module AND the
middleware module since the middleware runs first and also calls get_tenant_service.
"""
import pytest
from unittest.mock import MagicMock, patch
from fastapi.testclient import TestClient
from backend.models.tenant import (
TenantConfig,
TenantBranding,
TenantFeatures,
TenantDefaults,
TenantAuth,
TenantPublicConfig,
)
# Sample tenant config for mocking
SAMPLE_VOCALSTAR_CONFIG = TenantConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
is_active=True,
branding=TenantBranding(
logo_url="https://example.com/logo.png",
primary_color="#ffff00",
secondary_color="#006CF9",
site_title="Vocal Star Karaoke Generator",
),
features=TenantFeatures(
audio_search=False,
youtube_url=False,
youtube_upload=False,
dropbox_upload=False,
gdrive_upload=False,
theme_selection=False,
),
defaults=TenantDefaults(
theme_id="vocalstar",
locked_theme="vocalstar",
distribution_mode="download_only",
),
auth=TenantAuth(
allowed_email_domains=["vocal-star.com", "vocalstarmusic.com"],
require_email_domain=True,
sender_email="vocalstar@nomadkaraoke.com",
),
)
INACTIVE_CONFIG = TenantConfig(
id="inactive",
name="Inactive Tenant",
subdomain="inactive.nomadkaraoke.com",
is_active=False,
)
class TestTenantConfigEndpoint:
"""Tests for GET /api/tenant/config endpoint."""
@pytest.fixture
def mock_tenant_service(self):
"""Mock the tenant service in both routes and middleware."""
service = MagicMock()
# Patch in both places: routes AND middleware
with patch("backend.api.routes.tenant.get_tenant_service", return_value=service), \
patch("backend.middleware.tenant.get_tenant_service", return_value=service):
yield service
@pytest.fixture
def client(self, mock_tenant_service):
"""Create test client with mocked tenant service."""
from backend.main import app
return TestClient(app)
def test_get_config_default(self, client, mock_tenant_service):
"""Test returns default config when no tenant detected."""
mock_tenant_service.tenant_exists.return_value = False
response = client.get("/api/tenant/config")
assert response.status_code == 200
data = response.json()
assert data["tenant"] is None
assert data["is_default"] is True
def test_get_config_with_query_param(self, client, mock_tenant_service):
"""Test tenant detection via query parameter."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
assert data["tenant"]["id"] == "vocalstar"
assert data["tenant"]["name"] == "Vocal Star"
assert data["is_default"] is False
def test_get_config_with_header(self, client, mock_tenant_service):
"""Test tenant detection via X-Tenant-ID header."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get(
"/api/tenant/config",
headers={"X-Tenant-ID": "vocalstar"},
)
assert response.status_code == 200
data = response.json()
assert data["tenant"]["id"] == "vocalstar"
assert data["is_default"] is False
def test_get_config_query_param_takes_priority(self, client, mock_tenant_service):
"""Test query param takes priority over header."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get(
"/api/tenant/config?tenant=vocalstar",
headers={"X-Tenant-ID": "other"},
)
assert response.status_code == 200
# Should use vocalstar from query param
mock_tenant_service.get_public_config.assert_called_with("vocalstar")
def test_get_config_tenant_not_found(self, client, mock_tenant_service):
"""Test returns default when tenant not found."""
mock_tenant_service.get_public_config.return_value = None
response = client.get("/api/tenant/config?tenant=nonexistent")
assert response.status_code == 200
data = response.json()
assert data["tenant"] is None
assert data["is_default"] is True
def test_get_config_inactive_tenant(self, client, mock_tenant_service):
"""Test returns default when tenant is inactive."""
inactive_public = TenantPublicConfig.from_config(INACTIVE_CONFIG)
mock_tenant_service.get_public_config.return_value = inactive_public
response = client.get("/api/tenant/config?tenant=inactive")
assert response.status_code == 200
data = response.json()
assert data["tenant"] is None
assert data["is_default"] is True
def test_get_config_includes_branding(self, client, mock_tenant_service):
"""Test response includes full branding config."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
branding = data["tenant"]["branding"]
assert branding["primary_color"] == "#ffff00"
assert branding["secondary_color"] == "#006CF9"
assert branding["site_title"] == "Vocal Star Karaoke Generator"
def test_get_config_includes_features(self, client, mock_tenant_service):
"""Test response includes feature flags."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
features = data["tenant"]["features"]
assert features["audio_search"] is False
assert features["file_upload"] is True
assert features["youtube_upload"] is False
def test_get_config_includes_defaults(self, client, mock_tenant_service):
"""Test response includes default settings."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
defaults = data["tenant"]["defaults"]
assert defaults["theme_id"] == "vocalstar"
assert defaults["locked_theme"] == "vocalstar"
assert defaults["distribution_mode"] == "download_only"
def test_get_config_includes_allowed_domains(self, client, mock_tenant_service):
"""Test response includes allowed email domains."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
assert "vocal-star.com" in data["tenant"]["allowed_email_domains"]
assert "vocalstarmusic.com" in data["tenant"]["allowed_email_domains"]
class TestTenantConfigByIdEndpoint:
"""Tests for GET /api/tenant/config/{tenant_id} endpoint."""
@pytest.fixture
def mock_tenant_service(self):
"""Mock the tenant service in both routes and middleware."""
service = MagicMock()
with patch("backend.api.routes.tenant.get_tenant_service", return_value=service), \
patch("backend.middleware.tenant.get_tenant_service", return_value=service):
yield service
@pytest.fixture
def client(self, mock_tenant_service):
"""Create test client with mocked tenant service."""
from backend.main import app
return TestClient(app)
def test_get_config_by_id_success(self, client, mock_tenant_service):
"""Test getting config by explicit ID."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config/vocalstar")
assert response.status_code == 200
data = response.json()
assert data["tenant"]["id"] == "vocalstar"
assert data["is_default"] is False
def test_get_config_by_id_not_found(self, client, mock_tenant_service):
"""Test returns default when tenant ID not found."""
mock_tenant_service.get_public_config.return_value = None
response = client.get("/api/tenant/config/nonexistent")
assert response.status_code == 200
data = response.json()
assert data["tenant"] is None
assert data["is_default"] is True
def test_get_config_by_id_inactive(self, client, mock_tenant_service):
"""Test returns default when tenant is inactive."""
inactive_public = TenantPublicConfig.from_config(INACTIVE_CONFIG)
mock_tenant_service.get_public_config.return_value = inactive_public
response = client.get("/api/tenant/config/inactive")
assert response.status_code == 200
data = response.json()
assert data["tenant"] is None
assert data["is_default"] is True
class TestTenantAssetEndpoint:
"""Tests for GET /api/tenant/asset/{tenant_id}/{asset_name} endpoint."""
@pytest.fixture
def mock_tenant_service(self):
"""Mock the tenant service in both routes and middleware."""
service = MagicMock()
with patch("backend.api.routes.tenant.get_tenant_service", return_value=service), \
patch("backend.middleware.tenant.get_tenant_service", return_value=service):
yield service
@pytest.fixture
def client(self, mock_tenant_service):
"""Create test client with mocked tenant service."""
from backend.main import app
return TestClient(app)
def test_get_asset_redirects(self, client, mock_tenant_service):
"""Test asset endpoint redirects to signed URL."""
mock_tenant_service.get_asset_url.return_value = "https://storage.googleapis.com/signed-url"
response = client.get(
"/api/tenant/asset/vocalstar/logo.png",
follow_redirects=False,
)
# Should redirect (307 Temporary Redirect is FastAPI default)
assert response.status_code == 307
assert "storage.googleapis.com" in response.headers["location"]
def test_get_asset_not_found(self, client, mock_tenant_service):
"""Test returns 404 when asset not found."""
mock_tenant_service.get_asset_url.return_value = None
response = client.get("/api/tenant/asset/vocalstar/missing.png")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
class TestTenantConfigResponseSchema:
"""Tests for response schema validation."""
@pytest.fixture
def mock_tenant_service(self):
"""Mock the tenant service in both routes and middleware."""
service = MagicMock()
with patch("backend.api.routes.tenant.get_tenant_service", return_value=service), \
patch("backend.middleware.tenant.get_tenant_service", return_value=service):
yield service
@pytest.fixture
def client(self, mock_tenant_service):
"""Create test client with mocked tenant service."""
from backend.main import app
return TestClient(app)
def test_response_has_required_fields(self, client, mock_tenant_service):
"""Test response always has required fields."""
mock_tenant_service.tenant_exists.return_value = False
response = client.get("/api/tenant/config")
assert response.status_code == 200
data = response.json()
assert "tenant" in data
assert "is_default" in data
def test_tenant_config_excludes_sensitive_fields(self, client, mock_tenant_service):
"""Test public config doesn't include sensitive auth data."""
public_config = TenantPublicConfig.from_config(SAMPLE_VOCALSTAR_CONFIG)
mock_tenant_service.get_public_config.return_value = public_config
response = client.get("/api/tenant/config?tenant=vocalstar")
assert response.status_code == 200
data = response.json()
tenant = data["tenant"]
# Should NOT have full auth object
assert "auth" not in tenant
# Should NOT have sensitive defaults
assert tenant["defaults"].get("brand_prefix") is None
# Should have allowed_email_domains at top level
assert "allowed_email_domains" in tenant
-345
View File
@@ -1,345 +0,0 @@
"""
Unit tests for TenantMiddleware.
Tests tenant detection from headers, query params, and subdomains,
as well as request state attachment.
"""
import pytest
from unittest.mock import MagicMock, patch, AsyncMock
import os
from starlette.requests import Request
from starlette.datastructures import Headers
from starlette.responses import Response
from backend.middleware.tenant import (
TenantMiddleware,
get_tenant_from_request,
get_tenant_config_from_request,
NON_TENANT_SUBDOMAINS,
)
from backend.models.tenant import TenantConfig, TenantFeatures
# Sample tenant config for mocking
SAMPLE_CONFIG = TenantConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
is_active=True,
features=TenantFeatures(audio_search=False),
)
INACTIVE_CONFIG = TenantConfig(
id="inactive",
name="Inactive",
subdomain="inactive.nomadkaraoke.com",
is_active=False,
)
class MockRequest:
"""Mock Request object for testing."""
def __init__(self, headers=None, query_params=None):
self.headers = headers or {}
self.query_params = query_params or {}
self.url = MagicMock()
self.url.path = "/api/test"
self.state = MagicMock()
class TestTenantMiddleware:
"""Tests for TenantMiddleware class."""
@pytest.fixture
def middleware(self):
"""Create middleware instance."""
app = MagicMock()
return TenantMiddleware(app)
@pytest.fixture
def mock_tenant_service(self):
"""Create mock tenant service."""
with patch("backend.middleware.tenant.get_tenant_service") as mock:
service = MagicMock()
mock.return_value = service
yield service
# Tests for _extract_tenant_from_host()
def test_extract_tenant_from_host_standard_subdomain(self, middleware, mock_tenant_service):
"""Test extracting tenant from vocalstar.nomadkaraoke.com."""
mock_tenant_service.tenant_exists.return_value = True
tenant_id = middleware._extract_tenant_from_host("vocalstar.nomadkaraoke.com")
assert tenant_id == "vocalstar"
def test_extract_tenant_from_host_gen_subdomain(self, middleware, mock_tenant_service):
"""Test extracting tenant from vocalstar.gen.nomadkaraoke.com."""
mock_tenant_service.tenant_exists.return_value = True
tenant_id = middleware._extract_tenant_from_host("vocalstar.gen.nomadkaraoke.com")
assert tenant_id == "vocalstar"
def test_extract_tenant_from_host_with_port(self, middleware, mock_tenant_service):
"""Test extracting tenant when host includes port."""
mock_tenant_service.tenant_exists.return_value = True
tenant_id = middleware._extract_tenant_from_host("vocalstar.nomadkaraoke.com:443")
assert tenant_id == "vocalstar"
def test_extract_tenant_from_host_case_insensitive(self, middleware, mock_tenant_service):
"""Test host parsing is case insensitive."""
mock_tenant_service.tenant_exists.return_value = True
tenant_id = middleware._extract_tenant_from_host("VOCALSTAR.NomadKaraoke.COM")
assert tenant_id == "vocalstar"
def test_extract_tenant_from_host_empty(self, middleware):
"""Test returns None for empty host."""
tenant_id = middleware._extract_tenant_from_host("")
assert tenant_id is None
def test_extract_tenant_from_host_localhost(self, middleware):
"""Test returns None for localhost."""
tenant_id = middleware._extract_tenant_from_host("localhost:3000")
assert tenant_id is None
def test_extract_tenant_from_host_non_nomad_domain(self, middleware):
"""Test returns None for non-nomadkaraoke.com domains."""
tenant_id = middleware._extract_tenant_from_host("example.com")
assert tenant_id is None
def test_extract_tenant_from_host_base_domain(self, middleware):
"""Test returns None for nomadkaraoke.com without subdomain."""
tenant_id = middleware._extract_tenant_from_host("nomadkaraoke.com")
assert tenant_id is None
@pytest.mark.parametrize("subdomain", NON_TENANT_SUBDOMAINS)
def test_extract_tenant_from_host_non_tenant_subdomains(self, middleware, subdomain):
"""Test returns None for known non-tenant subdomains."""
host = f"{subdomain}.nomadkaraoke.com"
tenant_id = middleware._extract_tenant_from_host(host)
assert tenant_id is None
def test_extract_tenant_from_host_tenant_not_exists(self, middleware, mock_tenant_service):
"""Test returns None when tenant doesn't exist in GCS."""
mock_tenant_service.tenant_exists.return_value = False
tenant_id = middleware._extract_tenant_from_host("nonexistent.nomadkaraoke.com")
assert tenant_id is None
# Tests for _extract_tenant_id()
def test_extract_tenant_id_from_header(self, middleware):
"""Test X-Tenant-ID header takes priority."""
request = MockRequest(
headers={"X-Tenant-ID": "vocalstar", "Host": "other.nomadkaraoke.com"},
query_params={"tenant": "different"},
)
tenant_id = middleware._extract_tenant_id(request)
assert tenant_id == "vocalstar"
def test_extract_tenant_id_header_normalized(self, middleware):
"""Test header value is normalized (lowercased, stripped)."""
request = MockRequest(headers={"X-Tenant-ID": " VOCALSTAR "})
tenant_id = middleware._extract_tenant_id(request)
assert tenant_id == "vocalstar"
@patch.dict(os.environ, {"ENV": ""})
def test_extract_tenant_id_from_query_param_dev(self, middleware):
"""Test query param works in non-production."""
request = MockRequest(query_params={"tenant": "vocalstar"})
tenant_id = middleware._extract_tenant_id(request)
assert tenant_id == "vocalstar"
@patch.dict(os.environ, {"ENV": "production"})
def test_extract_tenant_id_query_param_disabled_in_prod(self, middleware, mock_tenant_service):
"""Test query param is ignored in production."""
# Need to reimport to pick up new env value
import importlib
import backend.middleware.tenant as tenant_module
# Save original value
original_is_prod = tenant_module.IS_PRODUCTION
# Temporarily set to True for this test
tenant_module.IS_PRODUCTION = True
try:
mock_tenant_service.tenant_exists.return_value = False
request = MockRequest(
query_params={"tenant": "vocalstar"},
headers={"Host": "gen.nomadkaraoke.com"},
)
tenant_id = middleware._extract_tenant_id(request)
# Should not use query param, and gen subdomain is non-tenant
assert tenant_id is None
finally:
# Restore
tenant_module.IS_PRODUCTION = original_is_prod
def test_extract_tenant_id_from_host(self, middleware, mock_tenant_service):
"""Test falls back to host header subdomain detection."""
mock_tenant_service.tenant_exists.return_value = True
request = MockRequest(headers={"Host": "vocalstar.nomadkaraoke.com"})
tenant_id = middleware._extract_tenant_id(request)
assert tenant_id == "vocalstar"
def test_extract_tenant_id_no_tenant(self, middleware, mock_tenant_service):
"""Test returns None when no tenant detected."""
mock_tenant_service.tenant_exists.return_value = False
request = MockRequest(headers={"Host": "gen.nomadkaraoke.com"})
tenant_id = middleware._extract_tenant_id(request)
assert tenant_id is None
# Tests for dispatch()
@pytest.mark.asyncio
async def test_dispatch_attaches_tenant_to_state(self, middleware, mock_tenant_service):
"""Test middleware attaches tenant info to request.state."""
mock_tenant_service.get_tenant_config.return_value = SAMPLE_CONFIG
mock_tenant_service.tenant_exists.return_value = True
request = MockRequest(headers={"X-Tenant-ID": "vocalstar"})
response = Response(content="OK")
call_next = AsyncMock(return_value=response)
result = await middleware.dispatch(request, call_next)
assert request.state.tenant_id == "vocalstar"
assert request.state.tenant_config == SAMPLE_CONFIG
@pytest.mark.asyncio
async def test_dispatch_adds_header_to_response(self, middleware, mock_tenant_service):
"""Test middleware adds X-Tenant-ID header to response."""
mock_tenant_service.get_tenant_config.return_value = SAMPLE_CONFIG
mock_tenant_service.tenant_exists.return_value = True
request = MockRequest(headers={"X-Tenant-ID": "vocalstar"})
response = Response(content="OK")
call_next = AsyncMock(return_value=response)
result = await middleware.dispatch(request, call_next)
assert result.headers.get("X-Tenant-ID") == "vocalstar"
@pytest.mark.asyncio
async def test_dispatch_no_tenant(self, middleware, mock_tenant_service):
"""Test middleware handles no tenant gracefully."""
mock_tenant_service.tenant_exists.return_value = False
request = MockRequest(headers={"Host": "gen.nomadkaraoke.com"})
response = Response(content="OK")
call_next = AsyncMock(return_value=response)
result = await middleware.dispatch(request, call_next)
assert request.state.tenant_id is None
assert request.state.tenant_config is None
assert "X-Tenant-ID" not in result.headers
@pytest.mark.asyncio
async def test_dispatch_inactive_tenant(self, middleware, mock_tenant_service):
"""Test middleware treats inactive tenant as no tenant."""
mock_tenant_service.get_tenant_config.return_value = INACTIVE_CONFIG
request = MockRequest(headers={"X-Tenant-ID": "inactive"})
response = Response(content="OK")
call_next = AsyncMock(return_value=response)
result = await middleware.dispatch(request, call_next)
# Inactive tenant should be treated as default
assert request.state.tenant_id is None
assert request.state.tenant_config is None
@pytest.mark.asyncio
async def test_dispatch_calls_next(self, middleware, mock_tenant_service):
"""Test middleware calls the next handler."""
mock_tenant_service.tenant_exists.return_value = False
request = MockRequest(headers={"Host": "gen.nomadkaraoke.com"})
response = Response(content="OK")
call_next = AsyncMock(return_value=response)
await middleware.dispatch(request, call_next)
call_next.assert_called_once_with(request)
class TestHelperFunctions:
"""Tests for helper functions."""
def test_get_tenant_from_request(self):
"""Test get_tenant_from_request helper."""
request = MagicMock()
request.state.tenant_id = "vocalstar"
assert get_tenant_from_request(request) == "vocalstar"
def test_get_tenant_from_request_none(self):
"""Test get_tenant_from_request when no tenant."""
request = MagicMock()
del request.state.tenant_id # Simulate missing attribute
assert get_tenant_from_request(request) is None
def test_get_tenant_config_from_request(self):
"""Test get_tenant_config_from_request helper."""
request = MagicMock()
request.state.tenant_config = SAMPLE_CONFIG
config = get_tenant_config_from_request(request)
assert config == SAMPLE_CONFIG
assert config.features.audio_search is False
def test_get_tenant_config_from_request_none(self):
"""Test get_tenant_config_from_request when no config."""
request = MagicMock()
del request.state.tenant_config
assert get_tenant_config_from_request(request) is None
class TestNonTenantSubdomains:
"""Tests for NON_TENANT_SUBDOMAINS constant."""
def test_gen_is_non_tenant(self):
"""Test 'gen' is in non-tenant subdomains."""
assert "gen" in NON_TENANT_SUBDOMAINS
def test_api_is_non_tenant(self):
"""Test 'api' is in non-tenant subdomains."""
assert "api" in NON_TENANT_SUBDOMAINS
def test_www_is_non_tenant(self):
"""Test 'www' is in non-tenant subdomains."""
assert "www" in NON_TENANT_SUBDOMAINS
def test_admin_is_non_tenant(self):
"""Test 'admin' is in non-tenant subdomains."""
assert "admin" in NON_TENANT_SUBDOMAINS
-406
View File
@@ -1,406 +0,0 @@
"""
Unit tests for tenant data models.
Tests the TenantConfig, TenantBranding, TenantFeatures, TenantDefaults,
TenantAuth, and TenantPublicConfig models.
"""
import pytest
from datetime import datetime
from backend.models.tenant import (
TenantConfig,
TenantBranding,
TenantFeatures,
TenantDefaults,
TenantAuth,
TenantPublicConfig,
TenantConfigResponse,
)
class TestTenantBranding:
"""Tests for TenantBranding model."""
def test_default_values(self):
"""Test that TenantBranding has sensible defaults."""
branding = TenantBranding()
assert branding.logo_url is None
assert branding.logo_height == 40
assert branding.primary_color == "#ff5bb8"
assert branding.secondary_color == "#8b5cf6"
assert branding.accent_color is None
assert branding.background_color is None
assert branding.favicon_url is None
assert branding.site_title == "Karaoke Generator"
assert branding.tagline is None
def test_custom_values(self):
"""Test TenantBranding with custom values."""
branding = TenantBranding(
logo_url="https://example.com/logo.png",
logo_height=60,
primary_color="#ffff00",
secondary_color="#0000ff",
accent_color="#ff0000",
background_color="#000000",
favicon_url="https://example.com/favicon.ico",
site_title="Custom Karaoke",
tagline="Make music magic",
)
assert branding.logo_url == "https://example.com/logo.png"
assert branding.logo_height == 60
assert branding.primary_color == "#ffff00"
assert branding.site_title == "Custom Karaoke"
class TestTenantFeatures:
"""Tests for TenantFeatures model."""
def test_default_features_enabled(self):
"""Test that most features are enabled by default."""
features = TenantFeatures()
# Input methods
assert features.audio_search is True
assert features.file_upload is True
assert features.youtube_url is True
# Distribution
assert features.youtube_upload is True
assert features.dropbox_upload is True
assert features.gdrive_upload is True
# Customization
assert features.theme_selection is True
assert features.color_overrides is True
# Output formats
assert features.enable_cdg is True
assert features.enable_4k is True
# Admin is disabled by default
assert features.admin_access is False
def test_restricted_features(self):
"""Test a restricted feature set like Vocal Star would have."""
features = TenantFeatures(
audio_search=False,
youtube_url=False,
youtube_upload=False,
dropbox_upload=False,
gdrive_upload=False,
theme_selection=False,
color_overrides=False,
)
assert features.audio_search is False
assert features.file_upload is True # Still allowed
assert features.youtube_url is False
assert features.youtube_upload is False
class TestTenantDefaults:
"""Tests for TenantDefaults model."""
def test_default_values(self):
"""Test TenantDefaults has sensible defaults."""
defaults = TenantDefaults()
assert defaults.theme_id is None
assert defaults.locked_theme is None
assert defaults.distribution_mode == "all"
assert defaults.brand_prefix is None
assert defaults.youtube_description_template is None
def test_locked_theme(self):
"""Test locked_theme setting."""
defaults = TenantDefaults(
theme_id="vocalstar",
locked_theme="vocalstar",
distribution_mode="download_only",
brand_prefix="VSTAR",
)
assert defaults.locked_theme == "vocalstar"
assert defaults.distribution_mode == "download_only"
assert defaults.brand_prefix == "VSTAR"
class TestTenantAuth:
"""Tests for TenantAuth model."""
def test_default_values(self):
"""Test TenantAuth defaults."""
auth = TenantAuth()
assert auth.allowed_email_domains == []
assert auth.require_email_domain is True
assert auth.fixed_token_ids == []
assert auth.sender_email is None
def test_restricted_domains(self):
"""Test restricted email domains configuration."""
auth = TenantAuth(
allowed_email_domains=["vocal-star.com", "vocalstarmusic.com"],
require_email_domain=True,
sender_email="vocalstar@nomadkaraoke.com",
)
assert "vocal-star.com" in auth.allowed_email_domains
assert auth.require_email_domain is True
assert auth.sender_email == "vocalstar@nomadkaraoke.com"
class TestTenantConfig:
"""Tests for TenantConfig model."""
@pytest.fixture
def basic_config(self):
"""Create a basic tenant config for testing."""
return TenantConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
)
@pytest.fixture
def full_config(self):
"""Create a fully configured tenant."""
return TenantConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
is_active=True,
branding=TenantBranding(
logo_url="https://example.com/logo.png",
primary_color="#ffff00",
secondary_color="#006CF9",
site_title="Vocal Star Karaoke Generator",
),
features=TenantFeatures(
audio_search=False,
youtube_url=False,
youtube_upload=False,
dropbox_upload=False,
gdrive_upload=False,
theme_selection=False,
),
defaults=TenantDefaults(
theme_id="vocalstar",
locked_theme="vocalstar",
distribution_mode="download_only",
),
auth=TenantAuth(
allowed_email_domains=["vocal-star.com", "vocalstarmusic.com"],
require_email_domain=True,
sender_email="vocalstar@nomadkaraoke.com",
),
)
def test_required_fields(self):
"""Test that id, name, and subdomain are required."""
config = TenantConfig(
id="test",
name="Test Tenant",
subdomain="test.nomadkaraoke.com",
)
assert config.id == "test"
assert config.name == "Test Tenant"
assert config.subdomain == "test.nomadkaraoke.com"
def test_default_is_active(self, basic_config):
"""Test that tenants are active by default."""
assert basic_config.is_active is True
def test_nested_models_have_defaults(self, basic_config):
"""Test that nested models are created with defaults."""
assert basic_config.branding is not None
assert basic_config.features is not None
assert basic_config.defaults is not None
assert basic_config.auth is not None
# Tests for get_sender_email()
def test_get_sender_email_custom(self, full_config):
"""Test get_sender_email returns custom sender when configured."""
assert full_config.get_sender_email() == "vocalstar@nomadkaraoke.com"
def test_get_sender_email_default_pattern(self, basic_config):
"""Test get_sender_email returns default pattern when not configured."""
# basic_config has no custom sender_email
assert basic_config.get_sender_email() == "vocalstar@nomadkaraoke.com"
def test_get_sender_email_different_tenant(self):
"""Test sender email follows tenant ID pattern."""
config = TenantConfig(
id="customtenant",
name="Custom",
subdomain="custom.nomadkaraoke.com",
)
assert config.get_sender_email() == "customtenant@nomadkaraoke.com"
# Tests for is_email_allowed()
def test_is_email_allowed_matching_domain(self, full_config):
"""Test email is allowed when domain matches."""
assert full_config.is_email_allowed("user@vocal-star.com") is True
assert full_config.is_email_allowed("user@vocalstarmusic.com") is True
def test_is_email_allowed_non_matching_domain(self, full_config):
"""Test email is rejected when domain doesn't match and require_email_domain=True."""
assert full_config.is_email_allowed("user@gmail.com") is False
assert full_config.is_email_allowed("user@other.com") is False
def test_is_email_allowed_case_insensitive(self, full_config):
"""Test email domain matching is case insensitive."""
assert full_config.is_email_allowed("User@VOCAL-STAR.COM") is True
assert full_config.is_email_allowed("User@VocalStarMusic.com") is True
def test_is_email_allowed_no_domain_restrictions(self, basic_config):
"""Test any email allowed when no domain restrictions."""
# basic_config has empty allowed_email_domains
assert basic_config.is_email_allowed("anyone@gmail.com") is True
assert basic_config.is_email_allowed("user@anything.com") is True
def test_is_email_allowed_require_domain_false(self):
"""Test non-matching emails allowed when require_email_domain=False."""
config = TenantConfig(
id="flexible",
name="Flexible Tenant",
subdomain="flexible.nomadkaraoke.com",
auth=TenantAuth(
allowed_email_domains=["preferred.com"],
require_email_domain=False, # Don't require matching
),
)
# Matching domain still works
assert config.is_email_allowed("user@preferred.com") is True
# Non-matching also allowed since require_email_domain=False
assert config.is_email_allowed("user@other.com") is True
def test_is_email_allowed_partial_domain_no_match(self, full_config):
"""Test partial domain matches don't work (must be exact suffix)."""
# "star.com" should not match "vocal-star.com"
assert full_config.is_email_allowed("user@star.com") is False
# Subdomain of allowed domain should work
assert full_config.is_email_allowed("user@sub.vocal-star.com") is False
class TestTenantPublicConfig:
"""Tests for TenantPublicConfig model."""
@pytest.fixture
def full_config(self):
"""Create a fully configured tenant for conversion testing."""
return TenantConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
is_active=True,
branding=TenantBranding(
logo_url="https://example.com/logo.png",
primary_color="#ffff00",
),
features=TenantFeatures(
audio_search=False,
admin_access=True, # This should be included in public config
),
defaults=TenantDefaults(
theme_id="vocalstar",
locked_theme="vocalstar",
distribution_mode="download_only",
brand_prefix="VSTAR", # This should NOT be in public config
youtube_description_template="Custom template", # This should NOT be in public config
),
auth=TenantAuth(
allowed_email_domains=["vocal-star.com"],
require_email_domain=True,
fixed_token_ids=["secret-token-123"], # This should NOT be in public config
sender_email="vocalstar@nomadkaraoke.com", # This should NOT be in public config
),
)
def test_from_config_basic_fields(self, full_config):
"""Test that basic fields are copied correctly."""
public = TenantPublicConfig.from_config(full_config)
assert public.id == "vocalstar"
assert public.name == "Vocal Star"
assert public.subdomain == "vocalstar.nomadkaraoke.com"
assert public.is_active is True
def test_from_config_branding_included(self, full_config):
"""Test that branding is fully included."""
public = TenantPublicConfig.from_config(full_config)
assert public.branding.logo_url == "https://example.com/logo.png"
assert public.branding.primary_color == "#ffff00"
def test_from_config_features_included(self, full_config):
"""Test that features are fully included."""
public = TenantPublicConfig.from_config(full_config)
assert public.features.audio_search is False
assert public.features.admin_access is True
def test_from_config_allowed_email_domains_included(self, full_config):
"""Test that allowed_email_domains is included for frontend validation."""
public = TenantPublicConfig.from_config(full_config)
assert "vocal-star.com" in public.allowed_email_domains
def test_from_config_sensitive_auth_excluded(self, full_config):
"""Test that sensitive auth fields are not in public config."""
public = TenantPublicConfig.from_config(full_config)
# TenantPublicConfig only has allowed_email_domains from auth
# It should NOT have fixed_token_ids or sender_email
assert not hasattr(public, 'auth')
# allowed_email_domains is a top-level field
assert hasattr(public, 'allowed_email_domains')
def test_from_config_defaults_partially_included(self, full_config):
"""Test that only safe defaults are included."""
public = TenantPublicConfig.from_config(full_config)
# These should be included
assert public.defaults.theme_id == "vocalstar"
assert public.defaults.locked_theme == "vocalstar"
assert public.defaults.distribution_mode == "download_only"
# brand_prefix and youtube_description_template should NOT be included
# The from_config method creates a new TenantDefaults without these
assert public.defaults.brand_prefix is None
assert public.defaults.youtube_description_template is None
class TestTenantConfigResponse:
"""Tests for TenantConfigResponse model."""
def test_default_response(self):
"""Test default response indicates no tenant."""
response = TenantConfigResponse()
assert response.tenant is None
assert response.is_default is True
def test_tenant_response(self):
"""Test response with tenant config."""
public_config = TenantPublicConfig(
id="vocalstar",
name="Vocal Star",
subdomain="vocalstar.nomadkaraoke.com",
is_active=True,
branding=TenantBranding(),
features=TenantFeatures(),
defaults=TenantDefaults(),
)
response = TenantConfigResponse(tenant=public_config, is_default=False)
assert response.tenant is not None
assert response.tenant.id == "vocalstar"
assert response.is_default is False
-418
View File
@@ -1,418 +0,0 @@
"""
Unit tests for TenantService.
Tests config loading from GCS, caching behavior, subdomain resolution,
and email domain validation.
"""
import pytest
from datetime import datetime, timedelta
from unittest.mock import MagicMock, patch
from backend.services.tenant_service import TenantService, get_tenant_service
from backend.models.tenant import TenantConfig, TenantPublicConfig
# Sample tenant config data that would come from GCS
SAMPLE_VOCALSTAR_CONFIG = {
"id": "vocalstar",
"name": "Vocal Star",
"subdomain": "vocalstar.nomadkaraoke.com",
"is_active": True,
"branding": {
"logo_url": "https://example.com/logo.png",
"logo_height": 50,
"primary_color": "#ffff00",
"secondary_color": "#006CF9",
"site_title": "Vocal Star Karaoke Generator",
},
"features": {
"audio_search": False,
"file_upload": True,
"youtube_url": False,
"youtube_upload": False,
"dropbox_upload": False,
"gdrive_upload": False,
"theme_selection": False,
},
"defaults": {
"theme_id": "vocalstar",
"locked_theme": "vocalstar",
"distribution_mode": "download_only",
},
"auth": {
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
"require_email_domain": True,
"sender_email": "vocalstar@nomadkaraoke.com",
},
}
SAMPLE_INACTIVE_CONFIG = {
"id": "inactive",
"name": "Inactive Tenant",
"subdomain": "inactive.nomadkaraoke.com",
"is_active": False,
"branding": {},
"features": {},
"defaults": {},
"auth": {},
}
class TestTenantService:
"""Tests for TenantService class."""
@pytest.fixture
def mock_storage(self):
"""Create a mock StorageService."""
storage = MagicMock()
storage.file_exists.return_value = False
storage.download_json.return_value = {}
storage.generate_signed_url.return_value = "https://signed-url.example.com"
return storage
@pytest.fixture
def tenant_service(self, mock_storage):
"""Create a TenantService with mock storage."""
return TenantService(storage=mock_storage)
# Tests for get_tenant_config()
def test_get_tenant_config_success(self, tenant_service, mock_storage):
"""Test successful config loading from GCS."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
config = tenant_service.get_tenant_config("vocalstar")
assert config is not None
assert config.id == "vocalstar"
assert config.name == "Vocal Star"
assert config.features.audio_search is False
mock_storage.download_json.assert_called_once_with("tenants/vocalstar/config.json")
def test_get_tenant_config_not_found(self, tenant_service, mock_storage):
"""Test returns None when tenant config doesn't exist."""
mock_storage.file_exists.return_value = False
config = tenant_service.get_tenant_config("nonexistent")
assert config is None
mock_storage.download_json.assert_not_called()
def test_get_tenant_config_gcs_error(self, tenant_service, mock_storage):
"""Test handles GCS errors gracefully."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.side_effect = Exception("GCS error")
config = tenant_service.get_tenant_config("vocalstar")
assert config is None
# Tests for caching behavior
def test_config_is_cached(self, tenant_service, mock_storage):
"""Test that configs are cached after first load."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# First call loads from GCS
config1 = tenant_service.get_tenant_config("vocalstar")
# Second call should use cache
config2 = tenant_service.get_tenant_config("vocalstar")
assert config1 is config2
# download_json should only be called once
assert mock_storage.download_json.call_count == 1
def test_force_refresh_bypasses_cache(self, tenant_service, mock_storage):
"""Test that force_refresh reloads from GCS."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# First call
tenant_service.get_tenant_config("vocalstar")
# Force refresh
tenant_service.get_tenant_config("vocalstar", force_refresh=True)
# download_json should be called twice
assert mock_storage.download_json.call_count == 2
def test_cache_expires_after_ttl(self, tenant_service, mock_storage):
"""Test that cache expires after TTL."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# Load config
tenant_service.get_tenant_config("vocalstar")
# Simulate cache expiration by manipulating cache time
tenant_service._cache_times["vocalstar"] = datetime.now() - timedelta(seconds=400)
# Next call should reload
tenant_service.get_tenant_config("vocalstar")
assert mock_storage.download_json.call_count == 2
# Tests for get_tenant_by_subdomain()
def test_get_tenant_by_subdomain_cached(self, tenant_service, mock_storage):
"""Test subdomain lookup uses cache."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# First load by ID to populate cache
tenant_service.get_tenant_config("vocalstar")
# Now lookup by subdomain should use cached mapping
config = tenant_service.get_tenant_by_subdomain("vocalstar.nomadkaraoke.com")
assert config is not None
assert config.id == "vocalstar"
def test_get_tenant_by_subdomain_fallback_resolution(self, tenant_service, mock_storage):
"""Test subdomain resolution when not in cache."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# Lookup by subdomain without prior cache
config = tenant_service.get_tenant_by_subdomain("vocalstar.nomadkaraoke.com")
assert config is not None
assert config.id == "vocalstar"
def test_get_tenant_by_subdomain_not_found(self, tenant_service, mock_storage):
"""Test returns None for unknown subdomain."""
mock_storage.file_exists.return_value = False
config = tenant_service.get_tenant_by_subdomain("unknown.nomadkaraoke.com")
assert config is None
def test_get_tenant_by_subdomain_case_insensitive(self, tenant_service, mock_storage):
"""Test subdomain lookup is case insensitive."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
config = tenant_service.get_tenant_by_subdomain("VOCALSTAR.NomadKaraoke.COM")
assert config is not None
assert config.id == "vocalstar"
# Tests for _resolve_subdomain_to_tenant_id()
def test_resolve_subdomain_3_parts(self, tenant_service, mock_storage):
"""Test resolving tenant.nomadkaraoke.com pattern."""
mock_storage.file_exists.return_value = True
tenant_id = tenant_service._resolve_subdomain_to_tenant_id("vocalstar.nomadkaraoke.com")
assert tenant_id == "vocalstar"
def test_resolve_subdomain_4_parts(self, tenant_service, mock_storage):
"""Test resolving tenant.gen.nomadkaraoke.com pattern."""
mock_storage.file_exists.return_value = True
tenant_id = tenant_service._resolve_subdomain_to_tenant_id("vocalstar.gen.nomadkaraoke.com")
assert tenant_id == "vocalstar"
def test_resolve_subdomain_too_short(self, tenant_service, mock_storage):
"""Test returns None for subdomains with fewer than 3 parts."""
tenant_id = tenant_service._resolve_subdomain_to_tenant_id("nomadkaraoke.com")
assert tenant_id is None
def test_resolve_subdomain_tenant_not_exists(self, tenant_service, mock_storage):
"""Test returns None when tenant config doesn't exist in GCS."""
mock_storage.file_exists.return_value = False
tenant_id = tenant_service._resolve_subdomain_to_tenant_id("nonexistent.nomadkaraoke.com")
assert tenant_id is None
# Tests for get_public_config()
def test_get_public_config_success(self, tenant_service, mock_storage):
"""Test getting public config."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
public_config = tenant_service.get_public_config("vocalstar")
assert public_config is not None
assert isinstance(public_config, TenantPublicConfig)
assert public_config.id == "vocalstar"
def test_get_public_config_not_found(self, tenant_service, mock_storage):
"""Test returns None when tenant not found."""
mock_storage.file_exists.return_value = False
public_config = tenant_service.get_public_config("nonexistent")
assert public_config is None
def test_get_public_config_by_subdomain(self, tenant_service, mock_storage):
"""Test getting public config by subdomain."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
public_config = tenant_service.get_public_config_by_subdomain("vocalstar.nomadkaraoke.com")
assert public_config is not None
assert public_config.id == "vocalstar"
# Tests for tenant_exists()
def test_tenant_exists_true(self, tenant_service, mock_storage):
"""Test returns True when config exists."""
mock_storage.file_exists.return_value = True
assert tenant_service.tenant_exists("vocalstar") is True
mock_storage.file_exists.assert_called_with("tenants/vocalstar/config.json")
def test_tenant_exists_false(self, tenant_service, mock_storage):
"""Test returns False when config doesn't exist."""
mock_storage.file_exists.return_value = False
assert tenant_service.tenant_exists("nonexistent") is False
# Tests for is_email_allowed_for_tenant()
def test_is_email_allowed_for_tenant_valid(self, tenant_service, mock_storage):
"""Test email allowed for valid domain."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
assert tenant_service.is_email_allowed_for_tenant("vocalstar", "user@vocal-star.com") is True
def test_is_email_allowed_for_tenant_invalid(self, tenant_service, mock_storage):
"""Test email not allowed for invalid domain."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
assert tenant_service.is_email_allowed_for_tenant("vocalstar", "user@gmail.com") is False
def test_is_email_allowed_for_tenant_not_found(self, tenant_service, mock_storage):
"""Test returns False when tenant not found."""
mock_storage.file_exists.return_value = False
assert tenant_service.is_email_allowed_for_tenant("nonexistent", "user@any.com") is False
# Tests for get_tenant_sender_email()
def test_get_tenant_sender_email_custom(self, tenant_service, mock_storage):
"""Test returns custom sender email when configured."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
sender = tenant_service.get_tenant_sender_email("vocalstar")
assert sender == "vocalstar@nomadkaraoke.com"
def test_get_tenant_sender_email_default(self, tenant_service, mock_storage):
"""Test returns default sender when tenant not found."""
mock_storage.file_exists.return_value = False
sender = tenant_service.get_tenant_sender_email("nonexistent")
# Should match DEFAULT_SENDER_EMAIL constant (consistent with EmailService)
assert sender == "gen@nomadkaraoke.com"
# Tests for invalidate_cache()
def test_invalidate_cache_specific_tenant(self, tenant_service, mock_storage):
"""Test invalidating cache for specific tenant."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# Load into cache
tenant_service.get_tenant_config("vocalstar")
assert "vocalstar" in tenant_service._config_cache
# Invalidate
tenant_service.invalidate_cache("vocalstar")
assert "vocalstar" not in tenant_service._config_cache
assert "vocalstar" not in tenant_service._cache_times
def test_invalidate_cache_specific_tenant_clears_subdomain_map(self, tenant_service, mock_storage):
"""Test invalidating specific tenant also clears its subdomain map entry."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# Load into cache (this also populates subdomain map)
tenant_service.get_tenant_config("vocalstar")
assert "vocalstar.nomadkaraoke.com" in tenant_service._subdomain_map
assert tenant_service._subdomain_map["vocalstar.nomadkaraoke.com"] == "vocalstar"
# Invalidate specific tenant
tenant_service.invalidate_cache("vocalstar")
# Subdomain map entry should also be removed
assert "vocalstar.nomadkaraoke.com" not in tenant_service._subdomain_map
def test_invalidate_cache_all(self, tenant_service, mock_storage):
"""Test invalidating all caches."""
mock_storage.file_exists.return_value = True
mock_storage.download_json.return_value = SAMPLE_VOCALSTAR_CONFIG
# Load into cache
tenant_service.get_tenant_config("vocalstar")
# Invalidate all
tenant_service.invalidate_cache()
assert len(tenant_service._config_cache) == 0
assert len(tenant_service._cache_times) == 0
assert len(tenant_service._subdomain_map) == 0
# Tests for get_asset_url()
def test_get_asset_url_success(self, tenant_service, mock_storage):
"""Test getting signed URL for existing asset."""
mock_storage.file_exists.return_value = True
url = tenant_service.get_asset_url("vocalstar", "logo.png")
assert url == "https://signed-url.example.com"
mock_storage.file_exists.assert_called_with("tenants/vocalstar/logo.png")
mock_storage.generate_signed_url.assert_called_with("tenants/vocalstar/logo.png", expiration_minutes=60)
def test_get_asset_url_not_found(self, tenant_service, mock_storage):
"""Test returns None when asset doesn't exist."""
mock_storage.file_exists.return_value = False
url = tenant_service.get_asset_url("vocalstar", "missing.png")
assert url is None
mock_storage.generate_signed_url.assert_not_called()
class TestGetTenantServiceSingleton:
"""Tests for the get_tenant_service() singleton function."""
def test_returns_same_instance(self):
"""Test that get_tenant_service returns singleton."""
# Reset the singleton
import backend.services.tenant_service as module
module._tenant_service = None
with patch.object(TenantService, '__init__', return_value=None):
service1 = get_tenant_service()
service2 = get_tenant_service()
assert service1 is service2
def test_thread_safe_initialization(self):
"""Test that singleton initialization is thread-safe."""
import threading
import backend.services.tenant_service as module
# Reset the singleton
module._tenant_service = None
instances = []
def get_instance():
with patch.object(TenantService, '__init__', return_value=None):
instances.append(get_tenant_service())
threads = [threading.Thread(target=get_instance) for _ in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
# All instances should be the same
assert all(i is instances[0] for i in instances)
-522
View File
@@ -1,522 +0,0 @@
"""
Comprehensive tests for Unicode and special character handling.
This test file verifies that artist/title with Unicode characters (curly quotes,
em dashes, non-ASCII characters) are properly sanitized throughout the codebase
to prevent:
1. HTTP header encoding failures (Content-Disposition, email subjects)
2. API query failures (Google Drive)
3. Filename encoding issues (Modal API, filesystem)
The root cause was job d49efab1 which had title "Mama Says (You Can't Back Down)"
with a curly apostrophe (U+2019) that caused the audio separation to fail because
the Modal API couldn't handle the non-latin-1 character in HTTP headers.
"""
import pytest
from karaoke_gen.utils import (
sanitize_filename,
normalize_text,
UNICODE_REPLACEMENTS,
TEXT_NORMALIZATIONS,
APOSTROPHE_REPLACEMENTS,
DOUBLE_QUOTE_REPLACEMENTS,
DASH_REPLACEMENTS,
WHITESPACE_REPLACEMENTS,
)
class TestSanitizeFilename:
"""Test the sanitize_filename function handles all edge cases."""
def test_curly_single_quotes(self):
"""Test that curly single quotes are converted to straight quotes."""
# The exact character that caused job d49efab1 to fail
assert sanitize_filename("Can't") == "Can't"
assert sanitize_filename("It's") == "It's"
# Left single quote
assert sanitize_filename("'Hello'") == "'Hello'"
def test_curly_double_quotes(self):
"""Test that curly double quotes are converted to underscores.
Note: Double quotes are filesystem-unsafe so they become underscores."""
# Curly quotes first become straight, then straight quotes become underscore
assert sanitize_filename("\u201cHello\u201d") == "_Hello_"
def test_em_dash(self):
"""Test that em dashes are converted to regular hyphens."""
assert sanitize_filename("Artist — Title") == "Artist - Title"
assert sanitize_filename("Song—Name") == "Song-Name"
def test_en_dash(self):
"""Test that en dashes are converted to regular hyphens."""
assert sanitize_filename("1990–2000") == "1990-2000"
def test_ellipsis(self):
"""Test that horizontal ellipsis is converted to three dots."""
# Note: The ellipsis becomes "..." but trailing dots are stripped
assert sanitize_filename("Wait\u2026") == "Wait"
# But in the middle of a string they're preserved
assert sanitize_filename("Wait\u2026Here") == "Wait...Here"
def test_non_breaking_space(self):
"""Test that non-breaking spaces are converted to regular spaces."""
# U+00A0 is non-breaking space
assert sanitize_filename("Hello\u00A0World") == "Hello World"
def test_filesystem_unsafe_characters(self):
"""Test that filesystem-unsafe characters are replaced."""
assert sanitize_filename("file/name") == "file_name"
assert sanitize_filename("file\\name") == "file_name"
assert sanitize_filename("file:name") == "file_name"
assert sanitize_filename("file*name") == "file_name"
assert sanitize_filename("file?name") == "file_name"
assert sanitize_filename('file"name') == "file_name"
assert sanitize_filename("file<name") == "file_name"
assert sanitize_filename("file>name") == "file_name"
assert sanitize_filename("file|name") == "file_name"
def test_trailing_periods_and_spaces(self):
"""Test that trailing periods and spaces are removed."""
assert sanitize_filename("filename.") == "filename"
assert sanitize_filename("filename...") == "filename"
assert sanitize_filename("filename ") == "filename"
assert sanitize_filename("filename . ") == "filename"
def test_leading_periods_and_spaces(self):
"""Test that leading periods and spaces are removed."""
assert sanitize_filename(".filename") == "filename"
assert sanitize_filename("...filename") == "filename"
assert sanitize_filename(" filename") == "filename"
assert sanitize_filename(" . filename") == "filename"
def test_multiple_underscores_collapsed(self):
"""Test that multiple consecutive underscores are collapsed to one."""
assert sanitize_filename("file___name") == "file_name"
# Multiple unsafe chars in a row
assert sanitize_filename("file?*:name") == "file_name"
def test_multiple_spaces_collapsed(self):
"""Test that multiple consecutive spaces are collapsed to one."""
assert sanitize_filename("file name") == "file name"
def test_none_input(self):
"""Test that None input returns None."""
assert sanitize_filename(None) is None
def test_empty_string(self):
"""Test that empty string returns empty string."""
assert sanitize_filename("") == ""
def test_real_world_examples(self):
"""Test real-world examples that have caused issues."""
# The exact title from job d49efab1
assert sanitize_filename("Mama Says (You Can't Back Down)") == "Mama Says (You Can't Back Down)"
# Broadway cast with smart quotes
assert sanitize_filename("Footloose (Broadway Cast) \u2014 \u201cFinal Song\u201d") == "Footloose (Broadway Cast) - _Final Song_"
# Japanese artist with em dash
assert sanitize_filename("宇多田ヒカル — First Love") == "宇多田ヒカル - First Love"
# Korean title (should pass through unchanged)
assert sanitize_filename("아이유 - 좋은 날") == "아이유 - 좋은 날"
# Mixed content
assert sanitize_filename("L'Arc~en~Ciel - Driver's High") == "L'Arc~en~Ciel - Driver's High"
def test_combination_of_issues(self):
"""Test strings with multiple problematic characters."""
result = sanitize_filename("It\u2019s \u201cMy\u201d Song \u2014 Volume\u20261")
# Curly ' -> ', curly " -> _ (via filesystem check), em dash -> -, ellipsis -> ...
assert result == "It's _My_ Song - Volume...1"
def test_unicode_replacements_dict_complete(self):
"""Verify all expected Unicode characters are in the replacements dict."""
# Curly quotes
assert "\u2018" in UNICODE_REPLACEMENTS # LEFT SINGLE QUOTATION MARK
assert "\u2019" in UNICODE_REPLACEMENTS # RIGHT SINGLE QUOTATION MARK
assert "\u201A" in UNICODE_REPLACEMENTS # SINGLE LOW-9 QUOTATION MARK
assert "\u201B" in UNICODE_REPLACEMENTS # SINGLE HIGH-REVERSED-9 QUOTATION MARK
assert "\u201C" in UNICODE_REPLACEMENTS # LEFT DOUBLE QUOTATION MARK
assert "\u201D" in UNICODE_REPLACEMENTS # RIGHT DOUBLE QUOTATION MARK
assert "\u201E" in UNICODE_REPLACEMENTS # DOUBLE LOW-9 QUOTATION MARK
assert "\u201F" in UNICODE_REPLACEMENTS # DOUBLE HIGH-REVERSED-9 QUOTATION MARK
# Dashes
assert "\u2013" in UNICODE_REPLACEMENTS # EN DASH
assert "\u2014" in UNICODE_REPLACEMENTS # EM DASH
# Other
assert "\u2026" in UNICODE_REPLACEMENTS # HORIZONTAL ELLIPSIS
assert "\u00A0" in UNICODE_REPLACEMENTS # NON-BREAKING SPACE
class TestSanitizationInContext:
"""Test that sanitization is applied correctly in various contexts.
These tests verify that the fixes we applied work in the actual
code paths that use artist/title data.
"""
def test_artist_title_format_for_audio_separation(self):
"""Verify artist-title formatting is sanitized for Modal API."""
# Simulate what audio_worker.py does
artist = "Footloose (Broadway Cast)"
title = "Mama Says (You Can\u2019t Back Down)" # Curly apostrophe U+2019
safe_artist = sanitize_filename(artist) if artist else "Unknown"
safe_title = sanitize_filename(title) if title else "Unknown"
artist_title = f"{safe_artist} - {safe_title}"
# The curly apostrophe should be converted to straight
assert "'" in artist_title # Straight apostrophe
assert "\u2019" not in artist_title # No curly apostrophe (U+2019)
def test_email_subject_safe_for_latin1(self):
"""Verify email subjects can be encoded to latin-1 after sanitization."""
artist = "Café del Mar" # Has accented e
title = "It\u2019s a \u201cBeautiful\u201d Day \u2014 Remix"
safe_artist = sanitize_filename(artist) if artist else None
safe_title = sanitize_filename(title) if title else None
subject = f"{safe_artist} - {safe_title}"
# Accented characters should pass through (they're valid latin-1)
assert "é" in subject
# But curly quotes and em dash should be converted
assert "'" in subject # Straight apostrophe
assert "-" in subject # Regular hyphen
# No curly apostrophe (U+2019)
assert "\u2019" not in subject
def test_content_disposition_safe(self):
"""Verify filenames are safe for HTTP Content-Disposition headers."""
artist = "Artist\u2019s \u201cName\u201d"
title = "Song\u2014Title"
safe_artist = sanitize_filename(artist) if artist else None
safe_title = sanitize_filename(title) if title else None
filename = f"{safe_artist} - {safe_title} (Final Karaoke).mp4"
# Should be able to encode to latin-1 for HTTP headers
try:
filename.encode('latin-1')
can_encode = True
except UnicodeEncodeError:
can_encode = False
# Standard ASCII chars and latin-1 safe accented chars should work
# Note: The test above with 'é' passes latin-1; smart quotes do not
assert can_encode
def test_google_drive_query_safe(self):
"""Verify filenames don't break Google Drive API queries."""
base_name = "Artist\u2019s \u201cSong\u201d"
safe_base_name = sanitize_filename(base_name) if base_name else base_name
filename = f"NOMAD-1234 - {safe_base_name}.mp4"
# Google Drive queries use single quotes - our sanitized string
# should have straight single quotes, not curly
# The query escaping handles straight quotes: '
escaped = filename.replace("'", "\\'")
query = f"name='{escaped}'"
# Should be a valid query string (no curly quotes to break syntax)
assert "\u2019" not in query # No curly single quote (U+2019)
assert "\u2018" not in query # No curly single quote (U+2018)
def test_dropbox_path_safe(self):
"""Verify Dropbox paths don't have problematic characters."""
artist = "Don\u2019t Stop"
title = "Believin\u2019"
safe_artist = sanitize_filename(artist) if artist else "Unknown"
safe_title = sanitize_filename(title) if title else "Unknown"
folder_name = f"NOMAD-1234 - {safe_artist} - {safe_title}"
remote_path = f"/Karaoke/{folder_name}"
# Path should not have curly quotes (U+2018 and U+2019)
assert "\u2018" not in remote_path
assert "\u2019" not in remote_path
# But should have the converted straight apostrophe
assert "'" in remote_path
class TestInternationalCharacters:
"""Test that international/non-ASCII characters are handled properly.
The goal is NOT to strip all non-ASCII, but to convert problematic
Unicode to safe equivalents while preserving legitimate international text.
"""
def test_japanese_preserved(self):
"""Japanese characters should pass through unchanged."""
assert sanitize_filename("宇多田ヒカル") == "宇多田ヒカル"
assert sanitize_filename("君が代") == "君が代"
def test_korean_preserved(self):
"""Korean characters should pass through unchanged."""
assert sanitize_filename("방탄소년단") == "방탄소년단"
assert sanitize_filename("좋은 날") == "좋은 날"
def test_chinese_preserved(self):
"""Chinese characters should pass through unchanged."""
assert sanitize_filename("周杰倫") == "周杰倫"
assert sanitize_filename("青花瓷") == "青花瓷"
def test_cyrillic_preserved(self):
"""Cyrillic characters should pass through unchanged."""
assert sanitize_filename("Тату") == "Тату"
assert sanitize_filename("Не верь, не бойся") == "Не верь, не бойся"
def test_arabic_preserved(self):
"""Arabic characters should pass through unchanged."""
assert sanitize_filename("فيروز") == "فيروز"
def test_accented_latin_preserved(self):
"""Accented Latin characters should pass through unchanged."""
assert sanitize_filename("Café") == "Café"
assert sanitize_filename("Señorita") == "Señorita"
assert sanitize_filename("Naïve") == "Naïve"
assert sanitize_filename("Björk") == "Björk"
def test_mixed_script_preserved(self):
"""Mixed script text should work correctly."""
# Japanese artist, English title
assert sanitize_filename("宇多田ヒカル - First Love") == "宇多田ヒカル - First Love"
# K-pop with English
assert sanitize_filename("BTS 방탄소년단 - Dynamite") == "BTS 방탄소년단 - Dynamite"
class TestEdgeCasesAndRegression:
"""Edge cases and regression tests for specific bugs."""
def test_job_d49efab1_exact_title(self):
"""Exact reproduction of job d49efab1 failure case.
The job had:
- artist: "Footloose (Broadway Cast)" (from display_artist override)
- title: "Mama Says (You Can't Back Down)" (with curly apostrophe U+2019)
This caused Stage 2 audio separation to fail because the filename
with the curly apostrophe couldn't be encoded in HTTP headers.
"""
artist = "Footloose (Broadway Cast)"
title = "Mama Says (You Can\u2019t Back Down)" # Explicit U+2019
safe_artist = sanitize_filename(artist)
safe_title = sanitize_filename(title)
artist_title = f"{safe_artist} - {safe_title}"
# The result should use straight apostrophe
assert "Can't" in artist_title
assert "\u2019" not in artist_title # No curly apostrophe
# Should be HTTP-header safe
try:
artist_title.encode('latin-1')
header_safe = True
except UnicodeEncodeError:
header_safe = False
assert header_safe
def test_double_sanitization_idempotent(self):
"""Sanitizing twice should give the same result as once."""
original = "It\u2019s \u201cMy\u201d Song \u2014 Test\u2026"
once = sanitize_filename(original)
twice = sanitize_filename(once)
assert once == twice
def test_only_problematic_chars_string(self):
"""Test string made entirely of problematic characters."""
result = sanitize_filename("\u2018\u2019\u201c\u201d\u2014\u2026")
# Should become: ''""--... then quotes -> underscores, collapses
# Actually: ' ' " " - ... -> underscores for filesystem chars
assert result # Should not be empty
def test_very_long_filename(self):
"""Test that very long filenames are handled."""
long_name = "A" * 1000
result = sanitize_filename(long_name)
assert result == long_name # No truncation in sanitize_filename itself
def test_special_musical_characters(self):
"""Test musical symbols and special characters."""
# These should pass through as they're not in our replacement list
assert "♪" in sanitize_filename("♪ Intro ♪")
assert "♫" in sanitize_filename("♫ Music ♫")
assert "♯" in sanitize_filename("C♯ Minor")
assert "♭" in sanitize_filename("B♭ Major")
class TestNormalizeText:
"""Test the normalize_text function for data consistency normalization."""
def test_curly_single_quotes_normalized(self):
"""Test that curly single quotes are converted to straight apostrophe."""
assert normalize_text("Can\u2019t") == "Can't"
assert normalize_text("It\u2018s") == "It's"
assert normalize_text("\u201Aquote\u201B") == "'quote'"
def test_curly_double_quotes_normalized(self):
"""Test that curly double quotes are converted to straight double quotes."""
assert normalize_text("\u201CHello\u201D") == '"Hello"'
assert normalize_text("\u201Equote\u201F") == '"quote"'
def test_backticks_normalized(self):
"""Test that backticks and similar marks are converted to apostrophe."""
assert normalize_text("code`here") == "code'here"
assert normalize_text("acute\u00B4accent") == "acute'accent"
assert normalize_text("prime\u2032mark") == "prime'mark"
def test_dashes_normalized(self):
"""Test that various dashes are converted to hyphen-minus."""
# EN DASH
assert normalize_text("1990\u20132000") == "1990-2000"
# EM DASH
assert normalize_text("word\u2014word") == "word-word"
# MINUS SIGN
assert normalize_text("a\u2212b") == "a-b"
# FIGURE DASH
assert normalize_text("phone\u2012number") == "phone-number"
def test_whitespace_normalized(self):
"""Test that various whitespace characters are normalized."""
# NON-BREAKING SPACE
assert normalize_text("hello\u00A0world") == "hello world"
# EM SPACE
assert normalize_text("hello\u2003world") == "hello world"
# IDEOGRAPHIC SPACE (CJK full-width)
assert normalize_text("hello\u3000world") == "hello world"
# ZERO WIDTH SPACE (removed entirely)
assert normalize_text("hello\u200Bworld") == "helloworld"
def test_ellipsis_normalized(self):
"""Test that ellipsis character is converted to three dots."""
assert normalize_text("Wait\u2026") == "Wait..."
assert normalize_text("Loading\u2026please wait") == "Loading...please wait"
def test_multiple_spaces_collapsed(self):
"""Test that multiple spaces are collapsed to one."""
assert normalize_text("hello world") == "hello world"
# After normalizing multiple whitespace chars
assert normalize_text("hello\u00A0\u00A0\u00A0world") == "hello world"
def test_leading_trailing_whitespace_stripped(self):
"""Test that leading/trailing whitespace is stripped."""
assert normalize_text(" hello ") == "hello"
assert normalize_text("\u00A0hello\u00A0") == "hello"
def test_none_input(self):
"""Test that None input returns None."""
assert normalize_text(None) is None
def test_non_string_input(self):
"""Test that non-string input is returned unchanged."""
assert normalize_text(123) == 123
assert normalize_text(["list"]) == ["list"]
def test_international_characters_preserved(self):
"""Test that international characters are NOT normalized away."""
assert normalize_text("日本語") == "日本語"
assert normalize_text("한국어") == "한국어"
assert normalize_text("Café") == "Café"
assert normalize_text("Björk") == "Björk"
def test_real_world_examples(self):
"""Test real-world examples with mixed content."""
# Job d49efab1's title
assert normalize_text("Mama Says (You Can\u2019t Back Down)") == "Mama Says (You Can't Back Down)"
# Broadway cast with smart quotes and em dash
assert normalize_text("Footloose \u2014 \u201CFinal Song\u201D") == 'Footloose - "Final Song"'
# Japanese with em dash
assert normalize_text("宇多田ヒカル \u2014 First Love") == "宇多田ヒカル - First Love"
def test_idempotent(self):
"""Test that normalizing twice gives the same result."""
original = "It\u2019s \u201CMy\u201D Song \u2014 Test\u2026"
once = normalize_text(original)
twice = normalize_text(once)
assert once == twice
def test_text_normalizations_dict_complete(self):
"""Verify TEXT_NORMALIZATIONS includes all expected categories."""
# Check apostrophe-like characters
for char in APOSTROPHE_REPLACEMENTS:
assert char in TEXT_NORMALIZATIONS
# Check double quote-like characters
for char in DOUBLE_QUOTE_REPLACEMENTS:
assert char in TEXT_NORMALIZATIONS
# Check dash-like characters
for char in DASH_REPLACEMENTS:
assert char in TEXT_NORMALIZATIONS
# Check whitespace characters
for char in WHITESPACE_REPLACEMENTS:
assert char in TEXT_NORMALIZATIONS
class TestModelValidatorNormalization:
"""Test that model validators correctly normalize input text."""
def test_job_create_normalizes_artist_title(self):
"""Test that JobCreate model normalizes artist and title fields."""
from backend.models.job import JobCreate
job = JobCreate(
artist="Don\u2019t Stop",
title="Believin\u2019 \u2014 Live",
)
assert job.artist == "Don't Stop"
assert job.title == "Believin' - Live"
def test_job_create_preserves_international_chars(self):
"""Test that JobCreate preserves international characters."""
from backend.models.job import JobCreate
job = JobCreate(
artist="宇多田ヒカル",
title="First Love",
)
assert job.artist == "宇多田ヒカル"
assert job.title == "First Love"
def test_job_create_handles_none(self):
"""Test that JobCreate handles None values correctly."""
from backend.models.job import JobCreate
job = JobCreate(
artist=None,
title="Some Title",
)
assert job.artist is None
assert job.title == "Some Title"
def test_audio_search_normalizes_fields(self):
"""Test that AudioSearchRequest normalizes text fields."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="Artist\u2019s Name",
title="Song \u2014 Remix",
display_artist="Display\u2019s Artist",
display_title="Display\u2019s Title",
)
assert request.artist == "Artist's Name"
assert request.title == "Song - Remix"
assert request.display_artist == "Display's Artist"
assert request.display_title == "Display's Title"
def test_audio_search_preserves_international_chars(self):
"""Test that AudioSearchRequest preserves international characters."""
from backend.api.routes.audio_search import AudioSearchRequest
request = AudioSearchRequest(
artist="방탄소년단",
title="Dynamite",
)
assert request.artist == "방탄소년단"
assert request.title == "Dynamite"
@@ -425,111 +425,6 @@ class TestVideoWorkerOrchestratorEncoding:
assert "Encoding failed" in str(exc_info.value)
@pytest.mark.asyncio
async def test_run_encoding_gce_downloads_files(self):
"""Test that GCE encoding downloads files from GCS to local directory.
This test verifies the fix for YouTube upload failure when using GCE encoding.
GCE encoding returns GCS blob paths, which need to be downloaded locally
before YouTube upload can access them.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# GCE backend returns GCS blob paths (not local paths)
mock_backend = MagicMock()
mock_backend.name = "gce" # Important: must be "gce" to trigger download
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path="jobs/test-job/finals/output_4k_lossless.mp4",
lossy_4k_mp4_path="jobs/test-job/finals/output_4k_lossy.mp4",
lossy_720p_mp4_path="jobs/test-job/finals/output_720p.mp4",
lossless_mkv_path="jobs/test-job/finals/output_4k.mkv",
encoding_time_seconds=60.0,
encoding_backend="gce",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was called for each output file
assert mock_storage.download_file.call_count == 4
# Verify the result paths were updated to local paths
assert orchestrator.result.final_video.startswith(temp_dir)
assert orchestrator.result.final_video_mkv.startswith(temp_dir)
assert orchestrator.result.final_video_lossy.startswith(temp_dir)
assert orchestrator.result.final_video_720p.startswith(temp_dir)
@pytest.mark.asyncio
async def test_run_encoding_local_does_not_download(self):
"""Test that local encoding does NOT trigger GCS download.
Local encoding produces files directly in the output directory,
so no download is needed.
"""
with tempfile.TemporaryDirectory() as temp_dir:
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path=os.path.join(temp_dir, "title.mov"),
karaoke_video_path=os.path.join(temp_dir, "karaoke.mov"),
instrumental_audio_path=os.path.join(temp_dir, "audio.flac"),
output_dir=temp_dir,
)
# Mock storage service
mock_storage = MagicMock()
mock_storage.download_file = MagicMock()
orchestrator = VideoWorkerOrchestrator(
config,
storage=mock_storage,
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
from backend.services.encoding_interface import EncodingOutput
# Local backend returns local paths directly
mock_backend = MagicMock()
mock_backend.name = "local" # Local backend
mock_backend.encode = AsyncMock(return_value=EncodingOutput(
success=True,
lossless_4k_mp4_path=os.path.join(temp_dir, "lossless.mp4"),
lossy_4k_mp4_path=os.path.join(temp_dir, "lossy.mp4"),
lossy_720p_mp4_path=os.path.join(temp_dir, "720p.mp4"),
lossless_mkv_path=os.path.join(temp_dir, "lossless.mkv"),
encoding_time_seconds=120.0,
encoding_backend="local",
))
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# Verify download_file was NOT called for local encoding
mock_storage.download_file.assert_not_called()
class TestVideoWorkerOrchestratorOrganization:
"""Test organization stage."""
@@ -845,192 +740,3 @@ class TestCreateOrchestratorConfigFromJob:
)
assert config.instrumental_audio_path == "/tmp/test/Test Artist - Test Title (Instrumental User).mp3"
def test_create_config_passes_instrumental_selection(self):
"""Test that instrumental_selection is passed through to OrchestratorConfig.
This is a REGRESSION TEST for the bug where orchestrator -> GCE encoding
path did not pass instrumental_selection, causing GCE worker to default
to 'clean' even when user selected 'with_backing'.
The bug was:
- PR #271 fixed GCE worker to READ instrumental_selection from config
- But the orchestrator path (encoding_interface.py) was never updated to SEND it
- The legacy path (video_worker.py _encode_via_gce) was already correct
- So the bug only manifested when USE_NEW_ORCHESTRATOR=true (the default)
See: fix(gce): Respect user's instrumental selection in GCE encoding worker (#271)
"""
job = MagicMock()
job.job_id = "test-123"
job.artist = "Test Artist"
job.title = "Test Title"
job.state_data = {"instrumental_selection": "with_backing"} # User selected backing vocals
job.enable_cdg = False
job.enable_txt = False
job.enable_youtube_upload = False
job.brand_prefix = None
job.discord_webhook_url = None
job.youtube_description_template = None
job.dropbox_path = None
job.gdrive_folder_id = None
job.keep_brand_code = None
job.existing_instrumental_gcs_path = None
config = create_orchestrator_config_from_job(
job=job,
temp_dir="/tmp/test",
)
# CRITICAL: instrumental_selection must be passed to OrchestratorConfig
# If this fails, the GCE worker will default to 'clean' and ignore user's selection
assert config.instrumental_selection == "with_backing", \
"instrumental_selection must be passed from job.state_data to OrchestratorConfig"
# Also verify the instrumental path uses "Backing" not "Clean"
assert "Backing" in config.instrumental_audio_path, \
"When with_backing is selected, instrumental path should contain 'Backing'"
class TestInstrumentalSelectionEndToEnd:
"""End-to-end tests for instrumental selection flow.
These tests verify that instrumental_selection flows correctly from:
job.state_data -> OrchestratorConfig -> EncodingInput -> GCE encoding_config
This test class was added after discovering that PR #271 only fixed the
GCE worker (receiving side) but not the orchestrator (sending side),
causing the bug to persist in production where USE_NEW_ORCHESTRATOR=true.
"""
def test_encoding_input_has_instrumental_selection_field(self):
"""Test that EncodingInput dataclass includes instrumental_selection.
Without this field, the orchestrator cannot pass the selection to
the encoding backend.
"""
from backend.services.encoding_interface import EncodingInput
# Test with explicit selection
input_with_backing = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
instrumental_selection="with_backing",
)
assert input_with_backing.instrumental_selection == "with_backing"
# Test default value
input_default = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert input_default.instrumental_selection == "clean", \
"Default instrumental_selection should be 'clean' for backward compatibility"
def test_orchestrator_config_has_instrumental_selection_field(self):
"""Test that OrchestratorConfig includes instrumental_selection."""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
instrumental_selection="with_backing",
)
assert config.instrumental_selection == "with_backing"
# Test default
config_default = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
)
assert config_default.instrumental_selection == "clean"
def test_gce_encoding_config_includes_instrumental_selection(self):
"""Test that GCEEncodingBackend passes instrumental_selection to encoding_config.
This is the CRITICAL test that would have caught the bug in PR #271.
The GCE worker reads config.get("instrumental_selection", "clean"),
so if we don't send it, it defaults to 'clean' regardless of user selection.
"""
from backend.services.encoding_interface import EncodingInput, GCEEncodingBackend
backend = GCEEncodingBackend(dry_run=True)
# Create input with 'with_backing' selection
encoding_input = EncodingInput(
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
artist="Test Artist",
title="Test Title",
instrumental_selection="with_backing",
options={
"job_id": "test-123",
"input_gcs_path": "gs://bucket/jobs/test-123/",
"output_gcs_path": "gs://bucket/jobs/test-123/finals/",
},
)
# We can't easily test the actual encoding_config dict without mocking
# the service, but we can verify the input has the right value
assert encoding_input.instrumental_selection == "with_backing"
# The fix ensures GCEEncodingBackend.encode() includes this in encoding_config:
# encoding_config = {
# ...
# "instrumental_selection": input_config.instrumental_selection,
# }
@pytest.mark.asyncio
async def test_orchestrator_passes_instrumental_selection_to_encoding(self):
"""Test full flow: orchestrator creates EncodingInput with instrumental_selection.
This integration test verifies the complete path:
OrchestratorConfig.instrumental_selection -> EncodingInput.instrumental_selection
"""
config = OrchestratorConfig(
job_id="test-job",
artist="Test Artist",
title="Test Title",
title_video_path="/path/title.mov",
karaoke_video_path="/path/karaoke.mov",
instrumental_audio_path="/path/audio.flac",
output_dir="/output",
instrumental_selection="with_backing",
)
orchestrator = VideoWorkerOrchestrator(config)
# Capture the EncodingInput that gets passed to the backend
captured_input = None
async def capture_encode(encoding_input):
nonlocal captured_input
captured_input = encoding_input
from backend.services.encoding_interface import EncodingOutput
return EncodingOutput(
success=True,
lossless_4k_mp4_path="/output/lossless.mp4",
encoding_time_seconds=1.0,
encoding_backend="mock",
)
with patch.object(orchestrator, "_get_encoding_backend") as mock_get:
mock_backend = MagicMock()
mock_backend.name = "mock"
mock_backend.encode = capture_encode
mock_get.return_value = mock_backend
await orchestrator._run_encoding()
# CRITICAL ASSERTION: instrumental_selection must be passed through
assert captured_input is not None, "encode() should have been called"
assert captured_input.instrumental_selection == "with_backing", \
"Orchestrator must pass instrumental_selection to EncodingInput"
-131
View File
@@ -1,131 +0,0 @@
"""Tests for the worker registry module."""
import asyncio
import pytest
from backend.workers.registry import WorkerRegistry
@pytest.fixture
def registry():
"""Create a fresh WorkerRegistry for each test."""
return WorkerRegistry()
class TestWorkerRegistry:
"""Test cases for WorkerRegistry."""
@pytest.mark.asyncio
async def test_register_adds_worker(self, registry):
"""Register should add worker to active workers."""
await registry.register("job-123", "audio")
assert registry.has_active_workers()
active = registry.get_active_workers()
assert "job-123" in active
assert "audio" in active["job-123"]
@pytest.mark.asyncio
async def test_unregister_removes_worker(self, registry):
"""Unregister should remove worker from active workers."""
await registry.register("job-123", "audio")
await registry.unregister("job-123", "audio")
assert not registry.has_active_workers()
assert "job-123" not in registry.get_active_workers()
@pytest.mark.asyncio
async def test_multiple_workers_same_job(self, registry):
"""Multiple workers can be registered for the same job."""
await registry.register("job-123", "audio")
await registry.register("job-123", "lyrics")
active = registry.get_active_workers()
assert "job-123" in active
assert active["job-123"] == {"audio", "lyrics"}
@pytest.mark.asyncio
async def test_partial_unregister_keeps_other_workers(self, registry):
"""Unregistering one worker should keep others active."""
await registry.register("job-123", "audio")
await registry.register("job-123", "lyrics")
await registry.unregister("job-123", "audio")
assert registry.has_active_workers()
active = registry.get_active_workers()
assert "job-123" in active
assert active["job-123"] == {"lyrics"}
@pytest.mark.asyncio
async def test_multiple_jobs(self, registry):
"""Workers from different jobs are tracked separately."""
await registry.register("job-123", "audio")
await registry.register("job-456", "lyrics")
active = registry.get_active_workers()
assert "job-123" in active
assert "job-456" in active
@pytest.mark.asyncio
async def test_unregister_nonexistent_worker(self, registry):
"""Unregistering a worker that doesn't exist should not error."""
await registry.unregister("nonexistent-job", "audio")
assert not registry.has_active_workers()
@pytest.mark.asyncio
async def test_wait_for_completion_no_workers(self, registry):
"""Wait should return immediately when no workers are active."""
result = await registry.wait_for_completion(timeout=1.0)
assert result is True
@pytest.mark.asyncio
async def test_wait_for_completion_workers_finish(self, registry):
"""Wait should complete when workers finish."""
await registry.register("job-123", "audio")
# Create a task that unregisters after a delay
async def unregister_after_delay():
await asyncio.sleep(0.2)
await registry.unregister("job-123", "audio")
asyncio.create_task(unregister_after_delay())
result = await registry.wait_for_completion(timeout=5.0)
assert result is True
@pytest.mark.asyncio
async def test_wait_for_completion_timeout(self, registry):
"""Wait should return False on timeout."""
await registry.register("job-123", "audio")
# Short timeout, worker never unregisters
result = await registry.wait_for_completion(timeout=0.1)
assert result is False
@pytest.mark.asyncio
async def test_concurrent_registration(self, registry):
"""Concurrent registrations should be thread-safe."""
job_ids = [f"job-{i}" for i in range(10)]
async def register_worker(job_id):
await registry.register(job_id, "audio")
# Register all workers concurrently
await asyncio.gather(*[register_worker(job_id) for job_id in job_ids])
active = registry.get_active_workers()
assert len(active) == 10
for job_id in job_ids:
assert job_id in active
@pytest.mark.asyncio
async def test_get_active_workers_returns_copy(self, registry):
"""get_active_workers should return a copy, not the internal state."""
await registry.register("job-123", "audio")
active = registry.get_active_workers()
active["job-123"].add("fake-worker")
# Internal state should not be affected
internal = registry.get_active_workers()
assert "fake-worker" not in internal.get("job-123", set())
+2 -2
View File
@@ -560,7 +560,7 @@ class TestRenderVideoWorkerCountdownPadding:
def test_countdown_processor_has_process_method(self):
"""Test that CountdownProcessor has the process method (main API)."""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
from lyrics_transcriber.output.countdown_processor import CountdownProcessor
countdown_processor = CountdownProcessor(cache_dir="/tmp")
assert hasattr(countdown_processor, 'process')
@@ -568,7 +568,7 @@ class TestRenderVideoWorkerCountdownPadding:
def test_countdown_processor_constants(self):
"""Test that CountdownProcessor has expected constants."""
from karaoke_gen.lyrics_transcriber.output.countdown_processor import CountdownProcessor
from lyrics_transcriber.output.countdown_processor import CountdownProcessor
# Verify the constants exist (these control countdown behavior)
assert hasattr(CountdownProcessor, 'COUNTDOWN_TEXT')
-27
View File
@@ -1,27 +0,0 @@
"""
Utilities for identifying test data.
Test data is generated by automated E2E tests and should be filterable
in admin dashboards to show only real user data by default.
"""
# Email domains used by automated testing frameworks
TEST_EMAIL_DOMAINS = [
"inbox.testmail.app", # Used by E2E happy path tests
]
def is_test_email(email: str) -> bool:
"""
Check if an email address belongs to automated test data.
Args:
email: Email address to check
Returns:
True if the email matches a test email domain pattern
"""
if not email:
return False
email_lower = email.lower()
return any(email_lower.endswith(f"@{domain}") for domain in TEST_EMAIL_DOMAINS)
+3 -16
View File
@@ -26,7 +26,6 @@ from backend.services.job_manager import JobManager
from backend.services.storage_service import StorageService
from backend.config import get_settings
from backend.workers.worker_logging import create_job_logger, setup_job_logging, job_logging_context
from backend.workers.registry import worker_registry
from backend.services.tracing import job_span, add_span_event, add_span_attribute
from backend.services.metrics import metrics
@@ -265,11 +264,7 @@ async def process_audio_separation(job_id: str) -> bool:
logger.info(f"[job:{job_id}] WORKER_START worker=audio")
job_log.info("=== AUDIO WORKER STARTED ===")
job_log.info(f"Job ID: {job_id}")
# Register with worker registry to prevent premature container shutdown
# This ensures Cloud Run waits for this worker to complete before terminating
await worker_registry.register(job_id, "audio")
# Set up log capture for AudioProcessor
log_handler = setup_job_logging(job_id, "audio", *AUDIO_WORKER_LOGGERS)
job_log.info(f"Log handler attached for {len(AUDIO_WORKER_LOGGERS)} loggers")
@@ -349,12 +344,7 @@ async def process_audio_separation(job_id: str) -> bool:
job_log.info(f"Stored effective model names: clean={effective_model_names['clean_instrumental_model']}")
# Format artist-title for file naming (matches CLI behavior)
# Sanitize to handle Unicode characters (curly quotes, em dashes, etc.)
# that cause HTTP header encoding issues with the remote API
from karaoke_gen.utils import sanitize_filename
safe_artist = sanitize_filename(job.artist) if job.artist else "Unknown"
safe_title = sanitize_filename(job.title) if job.title else "Unknown"
artist_title = f"{safe_artist} - {safe_title}"
artist_title = f"{job.artist} - {job.title}"
# Run audio separation (calls Modal API internally)
# This returns a dict with paths to all separated stems
@@ -416,16 +406,13 @@ async def process_audio_separation(job_id: str) -> bool:
return False
finally:
# Unregister from worker registry to signal completion
await worker_registry.unregister(job_id, "audio")
# Remove log handler to avoid duplicate logging on future runs
for logger_name in AUDIO_WORKER_LOGGERS:
try:
logging.getLogger(logger_name).removeHandler(log_handler)
except Exception:
pass
# Cleanup temporary directory
if os.path.exists(temp_dir):
shutil.rmtree(temp_dir)
Loaded 100 of 977 files, more files were not shown because too many files have changed in this diff. Show more