Compare commits

..
Author SHA1 Message Date
Andrew Beveridge d0a657610b Added admin API script 2026-01-04 20:32:46 -05:00
5639 changed files with 263682 additions and 1053047 deletions

No files matched your search

@@ -1,109 +0,0 @@
# Plan: Admin Feedback Visibility
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1454-feedback-email-debug
**Status:** Draft
## Overview
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
1. **Admin UI page** to browse and review all submitted feedback
2. **Email notification** to admin@nomadkaraoke.com on each new submission
## Requirements
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
- [ ] Sortable by date (newest first by default) and filterable by search (email)
- [ ] Star ratings rendered visually (not just numbers)
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
- [ ] Email includes all feedback content (ratings, text fields, user email)
- [ ] Test data filtering works (exclude_test toggle)
- [ ] Sidebar navigation includes new "Feedback" item
## Technical Approach
### Admin API Endpoint
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
### Admin UI Page
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
### Email Notification
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
## Implementation Steps
### Step 1: Backend — Admin feedback list endpoint
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
2. [ ] Add `GET /api/admin/feedback` endpoint with:
- Pagination: `limit=50`, `offset=0`
- Search by user email
- `exclude_test` filter (reuse `is_test_email`)
- Sort by `created_at` descending
- Return: items, total, offset, limit, has_more
3. [ ] Add aggregate stats in response: average ratings, total count
### Step 2: Backend — Email notification on submission
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
- User email
- All 4 star ratings
- All text fields
- Would recommend / would use again
- Submitted timestamp
- Link to admin feedback page
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
### Step 3: Frontend — Admin API client
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
### Step 4: Frontend — Admin feedback page
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
- Search by email
- Pagination (50 per page)
- Refresh button
- Test data toggle (via `useAdminSettings`)
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
### Step 5: Testing
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
11. [ ] Backend unit test for `send_feedback_notification` email method
12. [ ] Frontend component test for feedback page (render, empty state)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
## Testing Strategy
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
- **Integration**: Submit feedback → verify email service called
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
## Open Questions
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
## Rollback Plan
- Backend: Remove endpoint from admin.py, revert email call in users.py
- Frontend: Delete feedback page, remove sidebar item
- No data migration needed — Firestore collection unchanged
-393
View File
@@ -1,393 +0,0 @@
# Plan: Admin Job Details Page
**Created:** 2026-01-09
**Branch:** feat/sess-20260109-1738-admin-job-details
**Status:** In Progress (Phase 1 Complete)
## Overview
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
## Requirements
### Must Have
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
- [ ] Download links for all existing files (generate signed URLs)
- [ ] View worker logs with filtering by level/worker
- [ ] Restart job from beginning (reset to PENDING)
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
### Should Have
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
- [ ] View/edit raw JSON fields (state_data, file_urls)
- [ ] Quick actions: send completion email, clear cache, impersonate user
### Nice to Have
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
- [ ] Diff view when editing fields
- [ ] Audit log of admin changes
## Technical Approach
### Routing Fix (COMPLETED)
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
- Uses `useSearchParams()` to detect `?id=` parameter
- Shows detail view when `selectedJobId` is present, list view otherwise
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
- Deleted old `/admin/jobs/detail/` directory
### Backend Enhancements
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
### Frontend Structure
The page will have collapsible sections:
1. **Header**: Job ID, status badge, artist/title, quick actions
2. **Overview Cards**: User, dates, progress, source info
3. **Stage Timeline**: Visual breakdown of each stage with times
4. **Files**: Downloadable links for all GCS files
5. **Logs**: Filterable worker logs
6. **Configuration**: Editable job settings
7. **Raw Data**: JSON views of state_data, file_urls, full job
### Job Reset Logic
Reset to specific states requires:
- Clearing downstream state_data keys
- Resetting status
- Optionally clearing associated files
- Adding timeline event for the reset
Valid reset targets:
| Target State | Clears | Use Case |
|--------------|--------|----------|
| PENDING | Everything except input | Full restart |
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
## Implementation Steps
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
### Phase 1: Fix Routing (COMPLETED)
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
2. [x] Combined list and detail views in single component
3. [x] Use `useSearchParams()` to detect `?id=` parameter
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
6. [x] Verified: All 9 E2E tests pass, build succeeds
### Phase 2: Backend - File Downloads with Signed URLs
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
10. [ ] Traverse job.file_urls recursively
11. [ ] Generate signed URLs for each GCS path
12. [ ] Return structured response with file metadata
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
### Phase 3: Frontend - Enhanced File Display
15. [ ] Create "Files" accordion section with grouped downloads
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
17. [ ] Show file name, type, download button
18. [ ] Handle missing/unavailable files gracefully
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
20. [ ] Verify: `npm run test:e2e` passes
### Phase 4: Backend - Job Update Endpoint
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
23. [ ] Define editable fields whitelist
24. [ ] Validate field values (e.g., status must be valid enum)
25. [ ] Log admin changes for audit trail
26. [ ] Verify: Backend tests pass
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
### Phase 5: Frontend - Editable Fields
28. [ ] Create "Configuration" section with editable fields
29. [ ] Use inline edit pattern (click to edit, enter to save)
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
31. [ ] Show validation errors inline
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
33. [ ] Verify: `npm run test:e2e` passes
### Phase 6: Backend - Job Reset Endpoint
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
36. [ ] Accept `target_state` parameter
37. [ ] Implement reset logic for each target state
38. [ ] Clear appropriate state_data keys
39. [ ] Optionally trigger workers after reset
40. [ ] Verify: Backend tests pass
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
### Phase 7: Frontend - Reset Actions
42. [ ] Add "Reset Job" dropdown/menu in header
43. [ ] Show available reset targets based on current state
44. [ ] Confirm dialog explaining what will be cleared
45. [ ] Show success/error toast after reset
46. [ ] **E2E Test**: Add reset confirmation dialog test
47. [ ] Verify: `npm run test:e2e` passes
### Phase 8: Enhanced Stage Timeline
48. [ ] Parse timeline events to calculate stage durations
49. [ ] Display stages as cards or timeline items
50. [ ] Show duration for each stage
51. [ ] Highlight current/blocking stages
52. [ ] **E2E Test**: Add timeline display test
### Phase 9: Final Verification
53. [ ] Run full backend test suite: `make test-backend`
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
55. [ ] Manual production testing checklist (see Testing Strategy)
56. [ ] Verify CI passes on PR
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
## Testing Strategy
Based on `docs/TESTING.md` guidelines, this feature requires:
### Test Types Required
| Type | Location | Purpose |
|------|----------|---------|
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
| Manual Production E2E | N/A | Final validation on staging/prod |
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
```python
def test_get_job_files_returns_signed_urls_for_all_files():
"""Test that all files in file_urls get signed URLs."""
def test_get_job_files_handles_nested_file_urls():
"""Test recursive traversal of stems.instrumental_clean, etc."""
def test_get_job_files_handles_missing_files_gracefully():
"""Test files that don't exist in GCS are omitted or marked."""
def test_get_job_files_requires_admin():
"""Test non-admin gets 403."""
def test_get_job_files_returns_404_for_missing_job():
"""Test invalid job_id returns 404."""
```
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
```python
def test_update_job_allows_editable_fields():
"""Test artist, title, user_email, theme_id can be updated."""
def test_update_job_rejects_non_editable_fields():
"""Test job_id, created_at, etc. cannot be updated."""
def test_update_job_validates_status_enum():
"""Test invalid status value returns 422."""
def test_update_job_validates_theme_id_exists():
"""Test non-existent theme_id returns 422."""
def test_update_job_requires_admin():
"""Test non-admin gets 403."""
def test_update_job_returns_updated_job():
"""Test response includes updated job data."""
```
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
```python
def test_reset_to_pending_clears_all_state_data():
"""Test full reset clears everything except input."""
def test_reset_to_audio_selection_clears_search_results():
"""Test audio selection reset clears audio_search_results."""
def test_reset_to_awaiting_review_clears_review_data():
"""Test review reset clears screens, videos, review data."""
def test_reset_to_instrumental_selection_clears_finals():
"""Test instrumental reset clears encoding results."""
def test_reset_adds_timeline_event():
"""Test reset adds admin action to timeline."""
def test_reset_invalid_target_returns_422():
"""Test invalid target_state returns validation error."""
def test_reset_requires_admin():
"""Test non-admin gets 403."""
def test_reset_from_complete_to_pending_works():
"""Test can reset completed job back to start."""
```
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
```typescript
import { test, expect } from '@playwright/test';
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
test.describe('Admin Job Detail Page', () => {
test.beforeEach(async ({ page }) => {
await setAdminAuthToken(page);
});
test('navigates from jobs list to detail page', async ({ page }) => {
// Mock jobs list and single job endpoints
await page.route('**/api/admin/jobs', route => route.fulfill({
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
}));
await page.route('**/api/jobs/test123', route => route.fulfill({
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
}));
await page.goto('/admin/jobs');
await page.click('text=test123');
await expect(page).toHaveURL('/admin/jobs/test123');
await expect(page.getByText('Job test123')).toBeVisible();
});
test('displays all job metadata sections', async ({ page }) => {
// Test overview cards, timeline, logs accordion, etc.
});
test('files section shows downloadable links', async ({ page }) => {
// Mock files endpoint, verify download buttons appear
});
test('edit field inline and save', async ({ page }) => {
// Click edit, change value, press Enter, verify PATCH called
});
test('reset job shows confirmation dialog', async ({ page }) => {
// Click reset, verify dialog, confirm, verify POST called
});
test('handles API errors gracefully', async ({ page }) => {
// Mock 500 response, verify error toast shown
});
});
```
### Test File Organization
```
backend/tests/
├── test_admin_job_detail.py # NEW - all backend tests for this feature
├── test_admin_api.py # Existing admin tests (keep separate)
└── conftest.py # Shared fixtures
frontend/e2e/
├── fixtures/
│ ├── test-helper.ts # Add setAdminAuthToken helper
│ └── mock-responses.ts # Add admin job detail mocks
└── regression/
└── admin-job-detail.spec.ts # NEW - E2E regression tests
```
### Mocking Guidelines
**Backend - What to mock:**
- `StorageService.generate_signed_url()` - return predictable test URLs
- `StorageService.file_exists()` - control which files "exist"
- Firestore operations via existing `mock_firestore` fixture
**Backend - What NOT to mock:**
- `JobManager` logic - test the real reset/update logic
- Validation logic - test real Pydantic validation
- Authorization checks - test real `require_admin` dependency
**Frontend - What to mock:**
- All API responses via Playwright route interception
- Time (if testing token expiry)
**Frontend - What NOT to mock:**
- React component behavior
- User interactions (click, type, etc.)
### CI Requirements
This feature will be validated by existing CI checks:
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
- `frontend-build` - Must pass (verifies new page compiles)
### Running Tests Locally
```bash
# Backend tests only (for this feature)
poetry run pytest backend/tests/test_admin_job_detail.py -v
# All backend tests
make test-backend
# Frontend E2E regression (includes new tests)
cd frontend && npm run test:e2e
# Single E2E test file
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
# With browser visible for debugging
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
```
### Manual Production Testing Checklist
After deployment, manually verify on production:
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
- [ ] All file download links work (valid signed URLs)
- [ ] Edit artist/title → refresh → changes persisted
- [ ] Reset job to pending → job status shows "pending"
- [ ] Logs section loads and filters work
- [ ] Error states handled (invalid job ID shows friendly error)
## Open Questions
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
- **Recommendation**: Delete it. No external links depend on it.
2. [ ] Should job reset delete GCS files or just clear state_data references?
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
3. [ ] Should edits be immediately saved or require explicit "Save" button?
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
## Rollback Plan
If issues arise:
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
2. Backend: New endpoints are additive, won't affect existing functionality
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
## Notes
- The existing detail page has good foundational code - we're enhancing rather than replacing
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
- Job reset is powerful - consider adding confirmation dialogs and logging
-120
View File
@@ -1,120 +0,0 @@
# Plan: Admin Lyrics Review Dashboard
**Created:** 2026-03-05
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
**Status:** Draft
## Overview
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
## Requirements
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
- [ ] Admin page showing list of recently reviewed jobs with key metrics
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
- [ ] Edit log replay/stepper UI to walk through user changes one by one
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
## Technical Approach
### Backend: 2 new admin endpoints
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
- Filter by status (completed jobs, or optionally all post-review statuses)
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
- Support pagination, search, exclude_test filtering
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
- Returns immutable copy of:
- Original corrections (`corrections.json` from GCS)
- Updated corrections (`corrections_updated.json` from GCS)
- Edit log (`edit_log_{session_id}.json` from GCS)
- Annotations (`annotations.json` from GCS)
- Job metadata (artist, title, user_email, timestamps)
- Audio URL (signed, for playback in review UI)
- This is a **read-only** endpoint - no job state mutations
- Uses `require_admin` dependency (not `require_review_auth`)
- Does NOT transition job status or modify any state
### Frontend: New admin page + Read-only review mode
1. **`/admin/edit-reviews/page.tsx`** - List page
- Table of recently reviewed jobs
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
- "View Review" button → opens read-only lyrics review
- "View Edit Log" button → opens edit replay view
- "View Raw Data" button → shows raw JSON
2. **Read-only review viewer** - Reuse LyricsAnalyzer
- Load data from the new admin endpoint (not the regular review endpoint)
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
- Header shows "Admin Review - Read Only" indicator
- No submit/save buttons, no state transitions
- Audio playback still works (via signed URL)
3. **Edit log replay** - New component
- Timeline/stepper showing each edit entry
- For each entry: operation type, text before → text after, timestamp
- Step forward/backward through edits
- Show feedback reason if user provided one
- Summary stats: total edits, edits with feedback, correction types breakdown
4. **Raw data viewer** - JSON viewer tabs
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
- Tab 2: Annotations
- Tab 3: Corrections diff (original vs updated)
### Key safety guarantee
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
## Implementation Steps
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
9. [ ] **Tests: Backend unit tests** for new admin endpoints
10. [ ] **Tests: Frontend component tests** for new page
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
## Testing Strategy
- Unit tests for backend endpoints (mock Firestore + GCS)
- Verify admin auth is required on all new endpoints
- Verify no write operations occur on read-only endpoints
- Frontend component tests for key interactions
## Open Questions
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
## Rollback Plan
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
-151
View File
@@ -1,151 +0,0 @@
# Plan: AI-Powered Credit Gating
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1716-ai-credit-gating
**Status:** Implemented
## Overview
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
## Requirements
- [ ] Welcome credit grants go through AI evaluation before granting
- [ ] Feedback credit grants go through AI evaluation before granting
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
- [ ] Approved users get credits as before (seamless experience)
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
- [ ] Rejection email invites them to reply if it's a mistake
- [ ] Decision is logged to Firestore for audit trail
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
- [ ] Admin can override decisions via existing admin credit grant endpoint
## Technical Approach
### Architecture
```
User verifies email / submits feedback
↓
Collect abuse signals (sync, fast):
- User's fingerprint + IP
- Other accounts with same fingerprint (from correlations)
- Other accounts with same IP
- IP geolocation info
- Recent signup count from same IP/fingerprint (last 24h)
- User agent string
- For feedback: the feedback content itself
↓
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
- Present all signals
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
↓
If "grant": grant credits normally, log decision
If "deny": skip credit grant, send rejection email, log decision
If AI call fails: grant credits (fail-open), log error
```
### Key Design Decisions
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
## Implementation Steps
### Step 1: Add google-generativeai dependency
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
- Verify it works with Vertex AI auth in the nomadkaraoke project
### Step 2: Create credit evaluation service
- New file: `backend/services/credit_evaluation_service.py`
- `CreditEvaluationService` class with:
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
- Builds prompt with signals
- Calls Gemini for decision
- Logs decision to Firestore
- Returns structured result
### Step 3: Build the Gemini prompt
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
- Present all signals in a structured format
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
### Step 4: Create rejection email template
- Add `send_credit_denied_email(email, reasoning)` to EmailService
- Friendly-but-firm tone
- CC andrew@beveridge.uk
- Invite reply if it's a mistake
- Use existing email template infrastructure (branding, styling)
### Step 5: Integrate into welcome credit flow
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
- If denied: send rejection email, return False
- If error: log warning, grant anyway (fail-open)
### Step 6: Integrate into feedback credit flow
- Modify feedback endpoint in users.py
- Before granting credits: evaluate with feedback content included
- If denied: return different response message, send rejection email
- If error: grant anyway
### Step 7: Add Firestore collection for audit trail
- Collection: `credit_evaluations`
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
- TTL: keep indefinitely (small volume, high audit value)
### Step 8: Tests
- Unit tests for signal collection
- Unit tests for prompt building
- Unit tests for decision parsing (grant/deny/error cases)
- Integration test for the evaluation service with mocked Gemini
- Integration test for welcome credit flow with evaluation
- Integration test for feedback credit flow with evaluation
- Test fail-open behavior
### Step 9: Configuration
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
- Add to config.py
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `pyproject.toml` | Modify | Add google-generativeai dependency |
| `backend/config.py` | Modify | Add credit evaluation config vars |
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
| `backend/services/email_service.py` | Modify | Add credit denied email template |
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
## Testing Strategy
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
- **Integration tests**: Full evaluation flow with mocked Gemini responses
- **E2E tests**: Manual verification in production with test accounts
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
## Open Questions
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
## Rollback Plan
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
@@ -1,150 +0,0 @@
# Plan: Block Disposable Email Registration
**Created:** 2026-03-19
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
**Status:** Draft
## Overview
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
## Context
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
- Each new account gets 2 free welcome credits automatically
## Requirements
- [x] Reject disposable email domains at the magic link endpoint (backend)
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
- [x] Add `emailhook.site` to the default disposable domains list
- [x] Add backend unit tests for the integration
- [x] Add frontend test coverage for the new error state
## Design Decision: Explicit Rejection vs Silent Failure
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
1. We *want* the user to know their disposable email won't work (so they can use a real one)
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
3. The friendly message serves our goal of converting them to real email users
## Implementation Steps
### Step 1: Backend — Add disposable email check to magic link endpoint
**File:** `backend/api/routes/users.py`
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
```python
# Check for disposable email domains
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
raise HTTPException(
status_code=422,
detail="disposable_email_not_allowed"
)
```
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
### Step 2: Backend — Also check email and IP blocklists
In the same location, add checks for explicitly blocked emails and IPs:
```python
if email_validation.is_email_blocked(email):
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
# Silent reject — return generic success to prevent enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Blocked IP signup attempt: {ip_address}")
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
```
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
### Step 3: Backend — Add `emailhook.site` to default disposable domains
**File:** `backend/services/email_validation_service.py`
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
### Step 4: Frontend — Handle the disposable email error in AuthDialog
**File:** `frontend/components/auth/AuthDialog.tsx`
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
```tsx
const success = await sendMagicLink(email.trim().toLowerCase())
if (success) {
setStep("sent")
}
// If error contains our specific code, override with friendly message
// (the error state is already set by the auth store)
```
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
The friendly message:
> **We don't support disposable email addresses**
>
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
>
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
### Step 5: Backend tests
**File:** `backend/tests/test_email_validation_integration.py` (new)
Test cases:
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
2. Magic link with legitimate email proceeds normally
3. Magic link with blocked email returns silent success (no actual email sent)
4. Magic link with blocked IP returns silent success
### Step 6: Frontend tests
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
Test cases:
1. Submitting a disposable email shows the friendly rejection message
2. The friendly message contains the key copy (no spam promise, use real email)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
## Testing Strategy
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
## Rollback Plan
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
-236
View File
@@ -1,236 +0,0 @@
# Plan: Combined Lyrics + Instrumental Review Phase
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Overview
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
### Key Insight
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
### Benefits
1. **Better UX** - One interaction point instead of two
2. **Faster turnaround** - Users complete both reviews in one session
3. **Reduced drop-off** - Users less likely to abandon between phases
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
5. **Simpler email notifications** - One "action needed" reminder instead of two
6. **Cleaner codebase** - No conditional paths or legacy state handling
## Requirements
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
- [x] Functional: Single email notification for "review needed" instead of two separate ones
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
- [x] Functional: Admin "reset to review" action works with combined flow
- [x] Non-functional: No regression in existing test coverage
- [x] Non-functional: Clean removal of old code paths (no dead code)
## Technical Approach
### Strategy: "Combined Review - Clean Cutover"
**State Machine - Before:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
**State Machine - After:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
### Key Changes
1. **Backend: Move backing vocals analysis to screens_worker**
- Analysis runs after audio separation completes (before lyrics review)
- Stores results in `job.state_data['backing_vocals_analysis']`
- Remove analysis from render_video_worker entirely
2. **Backend: Replace review completion endpoint**
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
- Remove the old endpoint that didn't require instrumental
3. **Backend: Simplify render_video_worker**
- Always expect `instrumental_selection` in state_data
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
4. **Frontend: Single combined review page**
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
- Remove separate `/instrumental` route entirely
- Single "Complete Review" button submits both
5. **Backend: Remove instrumental-only endpoints**
- Remove `GET /api/jobs/{job_id}/instrumental-options`
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
- Remove `POST /api/jobs/{job_id}/select-instrumental`
- Keep custom instrumental creation for future advanced use
6. **Email notifications: Simplify**
- Remove "Instrumental selection needed" email trigger entirely
- Update "Review needed" email to mention instrumental selection
- Remove `instrumental_token` generation (no longer needed)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Move backing vocals analysis to screens_worker**
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
- Remove analysis code from render_video_worker.py entirely
2. [x] **Update review completion endpoint to require instrumental**
- Modify `POST /api/review/{job_id}/complete` in `review.py`
- Add required `instrumental_selection` field to request body
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
3. [x] **Add instrumental data to correction-data endpoint**
- Modify `GET /api/review/{job_id}/correction-data` to include:
- `instrumental_options`: List of {id, label, audio_url}
- `backing_vocals_analysis`: Analysis data from state_data
4. [x] **Simplify render_video_worker**
- Remove conditional check for instrumental pre-selection
- Always transition directly to `INSTRUMENTAL_SELECTED`
- Always trigger video worker immediately after render
5. [x] **Update state machine**
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
6. [x] **Remove instrumental-only endpoints**
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
- Remove `require_instrumental_auth` dependency
- Remove `instrumental_token` generation in job_manager.py
### Phase 2: Frontend Changes
7. [x] **Create combined review page**
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
- Instrumental selection UI appears after preview video section
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
8. [x] **Create InstrumentalSelectorEmbedded component**
- Created embedded version of InstrumentalSelector
- Accepts options, analysis, value, onChange props
- Supports compact mode for modal embedding
9. [x] **Update review completion flow**
- "Complete Review" button requires instrumental selection
- Validation error shown if no instrumental selected
- Single API call submits both lyrics and instrumental selection
10. [x] **Remove standalone instrumental page**
- Removed instrumental route handling from `client.tsx`
- Removed `getExpectedStates` case for "instrumental"
- InstrumentalSelector component stubbed out (deprecated)
- Kept `selectInstrumental` API for finalise-only jobs
11. [x] **Update lib/api.ts**
- Modified `completeReview()` to require instrumental selection
- Kept `selectInstrumental()` for finalise-only jobs
- Removed other instrumental-only functions
### Phase 3: Email & Notification Cleanup
12. [x] **Remove instrumental notification trigger**
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
- Updated docstrings to clarify finalise-only vs normal flow
- Instrumental reminder scheduling still works for finalise-only jobs
13. [x] **Update review email template**
- Update copy to mention "review lyrics and select instrumental"
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
### Phase 4: Testing & Cleanup
14. [x] **Update/remove tests for old flow**
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
- Updated worker tests to import from screens_worker (where analysis moved)
- Fixed conftest.py mocks to return AuthResult objects
15. [x] **Add tests for combined flow**
- Existing tests already cover combined completion endpoint
- Test infrastructure validates combined review flow works
- 1741 tests pass (only 1 pre-existing unrelated failure)
16. [x] **Update documentation**
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
17. [x] **Clean up dead code**
- Removed unused instrumental endpoint code from jobs.py
- Stubbed InstrumentalSelector component (deprecated)
- Kept select-instrumental endpoint for finalise-only jobs
## Files to Modify/Delete
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
## Testing Strategy
### Unit Tests
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
### Integration Tests
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
- `test_non_interactive_job`: CLI jobs skip review entirely as before
### E2E Tests
- `test_combined_review_ui`: User completes both reviews in single session
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
| CLI regression | Test `non_interactive` flag behavior explicitly |
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
## Migration Notes
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
1. Admin resets job to `AWAITING_REVIEW` via admin panel
2. User completes combined review
3. Job proceeds normally
This is acceptable because:
- Very few jobs should be in this state at any time
- Admin reset is already a supported operation
- Clean cutover is worth the minor migration friction
@@ -1,321 +0,0 @@
# Plan: Consolidate YouTube Download Paths
**Created:** 2026-01-20
**Status:** Planning
## Problem Statement
YouTube downloads currently have **3 different code paths** with different behaviors:
1. **Audio Search → Select YouTube** (`audio_search.py`)
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
- Runs on Cloud Run, blocked by YouTube bot detection
- Never uses remote flacfetch even when configured
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
- Works fine locally, not relevant to cloud issues
### Current Issues
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
3. **Inconsistent error handling** - Different paths handle errors differently
4. **Testing gaps** - No integration tests verifying the full flow
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
## Goals
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
5. **Remove dead code** - Clean up redundant implementations
## Proposed Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Entry Points │
├──────────────────────┬──────────────────────┬───────────────────────┤
│ Audio Search Select │ Direct URL Submit │ File Upload │
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
│ │ │
▼ ▼ │
┌──────────────────────────────────────────────┐ │
│ YouTubeDownloadService │ │
│ (NEW: backend/services/youtube_service.py) │ │
│ │ │
│ - download_by_url(url, job_id) → gcs_path │ │
│ - download_by_id(video_id, job_id) → gcs_path│ │
│ │ │
│ Internally: │ │
│ 1. Check if remote flacfetch enabled │ │
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
│ 3. If no: Use local yt_dlp with clear error │ │
│ messaging about bot detection risk │ │
└──────────────────┬───────────────────────────┘ │
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ GCS │
│ uploads/{job_id}/audio/ │
└─────────────────────────────────────────────────────────────────────┘
```
## Implementation Steps
### Phase 1: Create YouTubeDownloadService
**File:** `backend/services/youtube_download_service.py`
```python
class YouTubeDownloadService:
"""
Single point of entry for all YouTube downloads in the cloud backend.
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
on the flacfetch VM which has YouTube cookies and avoids bot detection.
When remote is not configured, falls back to local yt_dlp with a warning
that downloads may be blocked.
"""
def __init__(self):
self._flacfetch_client = get_flacfetch_client()
async def download(
self,
url: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download YouTube audio and upload to GCS.
Args:
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
job_id: Job ID for GCS path
artist: Optional artist name for filename
title: Optional title for filename
Returns:
GCS path (not gs:// prefix, just the path portion)
Raises:
YouTubeDownloadError: If download fails
"""
video_id = self._extract_video_id(url)
if self._flacfetch_client:
return await self._download_remote(video_id, job_id, artist, title)
else:
logger.warning(
"Remote flacfetch not configured. Attempting local download, "
"but this may fail due to YouTube bot detection on Cloud Run IPs."
)
return await self._download_local(url, job_id, artist, title)
async def download_by_id(
self,
video_id: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""Download by video ID instead of URL."""
url = f"https://www.youtube.com/watch?v={video_id}"
return await self.download(url, job_id, artist, title)
```
### Phase 2: Refactor Audio Search Path
**File:** `backend/api/routes/audio_search.py`
Changes to `_download_and_start_processing()`:
```python
# BEFORE: Complex branching with multiple paths
if is_torrent_source and is_remote_enabled:
# Torrent path...
elif is_remote_enabled and source_id and source_name:
# YouTube remote path (buggy source_id extraction)...
else:
# Local path with another YouTube branch...
# AFTER: Clean separation by source type
if is_torrent_source:
result = await _download_torrent(source_name, source_id, target_file, job_id)
elif source_name == 'YouTube':
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
elif source_name == 'Spotify':
result = await _download_spotify(source_id, job_id, artist, title)
else:
raise DownloadError(f"Unknown source: {source_name}")
```
### Phase 3: Refactor Direct URL Path
**File:** `backend/api/routes/file_upload.py`
Changes to `create_job_from_url()`:
```python
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
# This always runs on Cloud Run and gets blocked
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
if is_youtube_url(body.url):
youtube_service = get_youtube_download_service()
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
# Update job with audio already downloaded
job.input_media_gcs_path = gcs_path
# Skip audio download in worker
```
**File:** `backend/workers/audio_worker.py`
Changes to `download_audio()`:
```python
# BEFORE: Has YouTube download logic with download_from_url()
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
async def download_audio(job_id: str, job: Job) -> str:
"""Download audio from GCS (already uploaded by service layer)."""
if not job.input_media_gcs_path:
raise Exception("No audio path - this shouldn't happen for URL jobs")
# Download from GCS to local for processing
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
storage_service.download_file(job.input_media_gcs_path, local_path)
return local_path
```
### Phase 4: Clean Up Dead Code
Remove or consolidate:
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
### Phase 5: Comprehensive Testing
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
```python
class TestYouTubeDownloadService:
def test_download_uses_remote_when_configured(self):
"""When FLACFETCH_API_URL is set, should use remote client."""
def test_download_falls_back_to_local_when_remote_disabled(self):
"""When remote not configured, should use local yt_dlp."""
def test_download_extracts_video_id_from_various_url_formats(self):
"""Should handle youtube.com, youtu.be, shorts, etc."""
def test_download_returns_correct_gcs_path_format(self):
"""Should return path without gs:// prefix."""
def test_download_handles_remote_failure_gracefully(self):
"""If remote fails, should raise clear error, not fall back silently."""
```
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
```python
class TestYouTubeDownloadFlow:
@pytest.mark.integration
async def test_audio_search_youtube_selection_downloads_correctly(self):
"""Full flow: search → select YouTube → verify GCS upload."""
@pytest.mark.integration
async def test_direct_url_submission_downloads_correctly(self):
"""Full flow: submit YouTube URL → verify GCS upload."""
@pytest.mark.integration
async def test_youtube_download_with_remote_flacfetch(self):
"""Verify remote flacfetch integration works end-to-end."""
```
#### Flacfetch Tests (in flacfetch repo)
```python
class TestYouTubeDownloadByIdEndpoint:
def test_download_by_id_with_youtube_source(self):
"""POST /download-by-id with source_name=YouTube works."""
def test_download_by_id_uploads_to_gcs_when_requested(self):
"""When gcs_path provided, should upload and return gs:// path."""
def test_download_by_id_handles_bot_detection_error(self):
"""Should return clear error if YouTube blocks the request."""
```
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
## Migration Strategy
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
2. **Phase 3** changes the URL submission flow - may need careful rollout
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
4. **Phase 5** tests should be written alongside each phase
## Quick Fix vs Full Refactor
### Quick Fix (do now)
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
- This unblocks the "search → select YouTube" flow
### Full Refactor (this plan)
- Consolidate all YouTube downloads into `YouTubeDownloadService`
- Fix the "direct URL submission" flow to use remote flacfetch
- Remove dead code and simplify branching
- Add comprehensive tests
## Risk Assessment
| Risk | Mitigation |
|------|------------|
| Breaking existing flows | Comprehensive tests before deployment |
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
| YouTube bot detection | Remote flacfetch with cookies is the solution |
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
## Success Criteria
1. **One code path** for YouTube downloads in cloud backend
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
3. **All tests pass** with >80% coverage on new code
4. **Production verified** with real YouTube downloads
5. **No dead code** remaining in the codebase
## Open Questions
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
2. Should direct URL submission be synchronous (wait for download) or async (background)?
3. How do we handle YouTube cookies management for remote flacfetch?
## Related Documentation
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
@@ -1,111 +0,0 @@
# Plan: Credit Refund on Job Deletion and Cancellation
**Created:** 2026-03-04
**Branch:** feat/sess-20260304-1308-investigate-user-credits
**Status:** Implemented
## Overview
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
## Requirements
- [x] Job failures already refund credits (existing behavior, no change needed)
- [ ] Job cancellation refunds 1 credit
- [ ] Job deletion refunds 1 credit (if not already completed)
- [ ] No refund for completed jobs (user got their video)
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
- [ ] Refund failure must not block the cancel/delete operation
## Technical Approach
### Key Insight: Idempotency via Job Field
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
### What Should and Shouldn't Be Refunded
| Job Status at Time of Action | Delete | Cancel | Notes |
|------------------------------|--------|--------|-------|
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
### Refund Helper
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
1. Checks `job.credit_refunded` — skip if already refunded
2. Checks `job.user_email` — skip if no user
3. Checks if admin — skip if admin job
4. Calls `user_service.refund_credit()` with appropriate reason
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
6. Wraps everything in try/except so failures don't block the parent operation
## Implementation Steps
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
- Add `credit_refunded: bool = False` to the Job model
- No migration needed — Firestore is schemaless, defaults to False for existing docs
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
- Add `credit_refunded` check at the top
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
- The job object is already fetched at line 1017
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
- Need to fetch the job (already done when `delete_files=True`, but not always)
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
- For FAILED/CANCELLED: skip because refund was already issued during that transition
- For COMPLETE/PREP_COMPLETE: skip because user received their output
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
- The helper handles this, but also update the Firestore doc
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
- Test: cancel_job refunds credit for non-admin user
- Test: cancel_job skips refund for admin jobs
- Test: delete_job refunds credit for non-complete, non-terminal jobs
- Test: delete_job does NOT refund for completed jobs
- Test: delete_job does NOT double-refund for already-failed jobs
- Test: delete_job does NOT double-refund for already-cancelled jobs
- Test: refund failure doesn't block cancel/delete
- Test: credit_refunded flag prevents double refund
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
## Testing Strategy
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
## Open Questions
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
## Rollback Plan
- The `credit_refunded` field is additive (defaults to False for old jobs)
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
- Any refunds already issued are just bonus credits to users, no harm done
@@ -1,204 +0,0 @@
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
**Created:** 2026-01-19
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
**Status:** Draft
## Overview
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
This plan covers:
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
2. Archiving the GitHub repo
3. Deprecating the PyPI package (via README update published as final version)
4. Reorganizing vendored code to a permanent location in karaoke-gen
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
## Requirements
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
- [ ] python-lyrics-transcriber GitHub repo archived
- [ ] Final PyPI release with deprecation notice in README
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
- [ ] Tests pass after reorganization
- [ ] karaoke-gen pyproject.toml updated for new path
## Technical Approach
### New Path Structure
Move the vendored code to `libs/lyrics_transcriber/`:
```
karaoke-gen/
├── libs/
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
│ ├── __init__.py
│ ├── cli/
│ ├── core/
│ ├── correction/
│ ├── lyrics/
│ ├── output/
│ ├── review/
│ ├── storage/
│ ├── transcribers/
│ ├── types.py
│ └── utils/
├── karaoke_gen/
├── backend/
├── frontend/
└── ...
```
### Why `libs/lyrics_transcriber/` instead of alternatives?
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
### pyproject.toml Changes
```toml
packages = [
{ include = "karaoke_gen" },
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
{ include = "backend" }
]
include = [
"karaoke_gen/nextjs_frontend/out",
"karaoke_gen/nextjs_frontend/out/**/*",
"libs/lyrics_transcriber/output/fonts/*", # Updated path
"libs/lyrics_transcriber/output/cdgmaker/images/*",
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
]
```
### Files to Remove from Vendored Code
From current `lyrics_transcriber_temp/`:
- `pyproject.toml` - Package config for standalone release
- `poetry.lock` - Lock file for standalone package
- `README.md` - Will be replaced by deprecation notice
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
- `Dockerfile` - Standalone container build
- `.github/` - CI/CD for standalone package
- `.cursor/` - Cursor AI config
- `.specify/` - Specify config
- `.gitignore` - Not needed (karaoke-gen has its own)
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
- `scripts/` - Development scripts for standalone package
- `specs/` - Specification files
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
## Implementation Steps
### Phase 1: Update and Archive External Project (Manual Steps)
1. [ ] Clone python-lyrics-transcriber repo locally
2. [ ] Create deprecation notice README:
```markdown
# python-lyrics-transcriber (DEPRECATED)
**This project has been deprecated and archived.**
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
## Migration
For karaoke video generation with synchronized lyrics:
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
- Web app: https://gen.nomadkaraoke.com
## Historical Context
This library was originally developed as a standalone tool for creating synchronized
lyrics files. It has now been integrated into the karaoke-gen platform which provides
a complete end-to-end solution for karaoke video generation.
The last standalone version was 0.81.0. No further releases will be made to PyPI.
```
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
### Phase 2: Reorganize Vendored Code in karaoke-gen
5. [ ] Create `libs/` directory
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
8. [ ] Update `pyproject.toml` packages and include paths
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
### Phase 3: Verify and Test
10. [ ] Run `poetry install` to verify package configuration
11. [ ] Run `make test` to ensure all tests pass
12. [ ] Verify imports work:
```python
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
```
### Phase 4: Handle Tests from lyrics_transcriber
13. [ ] Evaluate lyrics_transcriber tests:
- **74 test files** in `lyrics_transcriber_temp/tests/`
- These test the internal lyrics_transcriber library functionality
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
- Skip integration/performance/manual tests that were for standalone development
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
| `pyproject.toml` | Modify | Update package paths |
| `scripts/benchmark_encoding.py` | Modify | Update import path |
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
## Testing Strategy
### Immediate Testing
- Run `make test` after reorganization - all existing tests must pass
- Verify Poetry can build/install the package correctly
### Test Migration Decision
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
- `unit/correction/` - Correction logic tests (important)
- `unit/lyrics/` - Lyrics provider tests
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
- `integration/` - Integration tests
- `performance/` - Performance benchmarks
**Recommendation:** Migrate a subset of critical unit tests, particularly:
- `test_corrector.py` - Core correction logic
- `test_anchor_sequence*.py` - Anchor sequence matching
- Handler tests in `handlers/`
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
## Open Questions
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
## Rollback Plan
If issues arise:
1. Git revert the consolidation commits
2. `lyrics_transcriber_temp/` can be restored from git history
3. The external python-lyrics-transcriber repo can be unarchived if needed
## Notes
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
- Only the internal project structure changes, not the API
- This is primarily a cleanup/organization task with some external repo admin work
@@ -1,110 +0,0 @@
# Plan: Disable Auto-Correction (Agentic and Heuristic)
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1914-disable-auto-correction
**Status:** Draft
## Overview
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
## Requirements
- [x] Disable agentic AI correction (LLM-based via Gemini)
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
- [x] Preserve the transcription → human review → video generation flow
- [x] Minimal code changes (configuration-based if possible)
- [x] Easy to re-enable in the future if correction quality improves
## Technical Approach
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
**Current behavior:**
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
**Proposed changes:**
### Option A: Add a new `skip_correction` config (Recommended)
Add a single boolean `skip_correction` setting that:
- When `true`: Skips the entire correction process, returning raw transcription
- When `false` (future): Falls back to existing agentic/heuristic behavior
This is cleaner than disabling both systems separately because:
1. Single config controls both
2. Clear intent ("skip correction" vs "disable two different systems")
3. Easy to understand for future maintainers
### Option B: Just disable both individually
- Set `use_agentic_ai` default to `false`
- Disable default heuristic handlers by setting `enabled_handlers=[]`
Option A is cleaner, but Option B requires fewer code changes.
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
## Implementation Steps
1. [x] **Add `skip_correction` config to backend/config.py**
- Add `skip_correction: bool` setting, default `true`
- Comment explaining this disables both agentic and heuristic correction
2. [x] **Update lyrics_worker.py to pass skip_correction**
- Pass the config value through to LyricsProcessor/LyricsTranscriber
3. [x] **Update corrector.py to support skip_correction**
- When `skip_correction=True`, return immediately with uncorrected segments
- No handler processing, no agentic AI
- Still need to do anchor/gap finding? No - skip entirely
4. [x] **Update LyricsProcessor to support skip_correction**
- Pass through to LyricsTranscriber
5. [x] **Update LyricsTranscriber controller to support skip_correction**
- Pass through to corrector or skip corrector.run() entirely
6. [x] **Update docs/README.md**
- Note that auto-correction is disabled
- Update "Agentic AI correction" status to "Disabled"
7. [x] **Run tests to verify no regressions**
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
| `docs/README.md` | Modify | Update status to reflect disabled correction |
## Testing Strategy
- **Unit tests**: No new tests needed - this is a config change
- **Existing tests**: Run `make test` to ensure no regressions
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
## Open Questions
None - the approach is straightforward.
## Rollback Plan
To re-enable auto-correction:
1. Set `SKIP_CORRECTION=false` environment variable, OR
2. Change the default back to `false` in `backend/config.py`
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
## Alternative Considered
Instead of adding a new config, we could:
1. Set `USE_AGENTIC_AI=false` (disables agentic)
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
@@ -1,99 +0,0 @@
# Plan: Encoding Worker Immutable Deployment Pattern
## Status: In Progress
## Problem Statement
The GCE encoding worker has recurring version mismatch issues because:
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
2. Wheel selection uses `sort -V` which can fail if image is outdated
3. CI only restarts service - doesn't verify version strictly
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
5. 51 old wheel files accumulate in GCS bucket
## Solution: Immutable Deployment Pattern
### Core Principles
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
3. **Version Manifest**: CI writes expected version, service MUST match
4. **Strict Verification**: CI FAILS if deployed version doesn't match
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
## Implementation Steps
### Phase 1: GCS Structure & Scripts
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
- [x] **Step 1.4**: Create README.md with documentation
### Phase 2: CI Workflow Updates
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
- [x] **Step 2.3**: Update CI to write version.txt manifest
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
### Phase 3: Packer Image Updates
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
### Phase 4: Documentation & Cleanup
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
- [x] **Step 4.2**: Update infrastructure/packer/README.md
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
## New GCS Structure
```
gs://karaoke-gen-storage-nomadkaraoke/
├── wheels/
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
│ └── karaoke_gen-{version}.whl # Versioned for rollback
│
└── encoding-worker/
├── bootstrap.sh # Reference copy
├── startup.sh # CI-managed, downloaded on boot
└── version.txt # Expected version number
```
## Files to Create/Modify
| File | Action | Purpose |
|------|--------|---------|
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
| `.github/workflows/ci.yml` | Modify | New deployment flow |
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
## Testing Strategy
1. **Unit test**: Verify startup.sh logic locally
2. **Integration test**: After merge, verify:
- CI uploads files to correct GCS paths
- Service starts with correct version
- Health endpoint returns expected version
3. **Manual verification**: Check VM logs after first deployment
## Rollback Plan
If issues occur:
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
2. Previous versioned wheels remain in GCS for manual rollback
3. Old Packer images remain for VM rollback
## Open Questions
None - design is complete.
## Notes
- Bootstrap.sh is ~10 lines and should never need changes
- All logic changes go through startup.sh in GCS
- Version verification is strict - CI fails on mismatch
@@ -1,166 +0,0 @@
# Plan: Fix YouTube Remote Download Missing GCS Path
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
**Status:** Complete
## Overview
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
## Root Cause
In `backend/api/routes/audio_search.py` lines 354-363:
```python
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
target_file=target_file,
download_url=download_url,
# <-- MISSING: gcs_path parameter
)
```
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
## Requirements
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
- [x] Backend should receive a `gs://` path from flacfetch
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
- [x] Maintain backwards compatibility (local-only deployments should still work)
- [x] Add test coverage for this code path
## Technical Approach
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
The current code structure has two branches:
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
- If remote disabled: download locally and upload to GCS from Cloud Run
## Implementation Steps
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
4. [x] Update logging to clarify the download path being used
5. [x] Bump version in `pyproject.toml`
6. [x] Run tests to verify no regressions
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
| `pyproject.toml` | Modify | Bump version |
## Detailed Code Changes
### audio_search.py (lines 349-393)
Current structure:
```python
else:
# Local download (YouTube or fallback) <-- WRONG: not always local!
temp_dir = tempfile.mkdtemp(...)
if source_id and source_name and is_remote_enabled:
result = audio_search_service.download_by_id(
...
output_dir=temp_dir, # <-- BUG: unused for remote
# Missing gcs_path!
)
```
Fixed structure:
```python
else:
# Non-torrent download (YouTube or fallback)
if is_remote_enabled and source_id and source_name:
# Remote download via flacfetch VM - upload directly to GCS
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination, # <-- FIX: Add this!
)
# Handle GCS path response (same logic as torrent branch)
if result.filepath.startswith("gs://"):
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: upload manually
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (remote disabled or no source_id)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# ... existing local download logic ...
```
## Testing Strategy
### Unit Tests
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
```python
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads via remote flacfetch include GCS path.
This is the same bug pattern as the torrent download bug fixed in 2025-12:
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
"""
# Setup: YouTube source with remote enabled
# Verify: download_by_id called with gcs_path parameter
```
### Manual Testing (post-deploy)
1. Create a job and select a YouTube audio source
2. Verify job completes successfully
3. Check GCS path is properly set in job data
## Open Questions
None - the fix is straightforward and follows the pattern established for torrent sources.
## Rollback Plan
This is a minor code change. If issues occur:
1. Revert the PR
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
## Related Documentation
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
-407
View File
@@ -1,407 +0,0 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -1,149 +0,0 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
@@ -1,218 +0,0 @@
# Plan: Job Email Notifications & Admin Message Tools
**Created:** 2026-01-06
**Branch:** feat/sess-20260106-0116-add-email-button
**Status:** Draft
## Overview
Add comprehensive email notification system for karaoke-gen that:
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
2. **Uses cloud-stored templates** for the job completion message
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
## Requirements
### Functional Requirements
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
- [ ] Store email templates in GCS (editable without code deploy)
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
### Non-Functional Requirements
- [ ] Email sending must not block job completion (async/background)
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
- [ ] Template changes should take effect without restart (no long-term caching)
## Technical Approach
### 1. Template Storage (GCS)
Store templates in `gs://karaoke-gen-storage/templates/`:
- `job-completion.txt` - Plain text template for job completion
- `job-completion.html` - HTML version (optional, for rich emails)
- `action-needed-lyrics.txt` - Reminder for lyrics review
- `action-needed-instrumental.txt` - Reminder for instrumental selection
**Template Variables:**
- `{name}` - User's display name or "there" if unknown
- `{youtube_url}` - YouTube video URL
- `{dropbox_url}` - Dropbox folder URL
- `{artist}` - Artist name
- `{title}` - Song title
- `{job_id}` - Job ID
- `{review_url}` - Lyrics review URL (for action-needed emails)
- `{instrumental_url}` - Instrumental selection URL
- `{feedback_url}` - Feedback form URL (for completion emails)
### 2. Email Service Extensions
Extend `backend/services/email_service.py`:
- Add `send_job_completion()` method with CC support
- Add `send_action_reminder()` method
- Add `render_template()` helper to fetch & render GCS templates
### 3. Auto-Email Triggers
**Job Completion:**
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
- Implementation: Call email service async in background task
**Action Reminders (5-min idle detection):**
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
- Record `blocking_state_entered_at` timestamp in `state_data`
- Schedule a Cloud Tasks task for 5 minutes later
- The scheduled task checks if job is still in blocking state and user hasn't started review
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
### 4. Admin UI (Frontend)
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
- "Copy Message" button - fetches rendered template, copies to clipboard
- "Send Email" button - opens dialog for email input, calls API to send
### 5. Backend API Endpoints
New endpoints in `backend/api/routes/admin.py`:
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
New internal endpoint for scheduled reminder:
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
## Implementation Steps
### Phase 1: Template Infrastructure
1. [ ] Upload initial template to GCS from user's existing template file
2. [ ] Create `template_service.py` to fetch/render templates from GCS
3. [ ] Add unit tests for template rendering
### Phase 2: Email Extensions
4. [ ] Extend `email_service.py` with CC support in `send_email()`
5. [ ] Add `send_job_completion()` method
6. [ ] Add `send_action_reminder()` method
7. [ ] Add unit tests for new email methods
### Phase 3: Auto-Email on Completion
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
9. [ ] Add integration test for completion email trigger
### Phase 4: Idle Reminder System
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
12. [ ] Schedule 5-minute delayed task when entering blocking states
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
14. [ ] Add tests for idle reminder flow
### Phase 5: Admin API
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
17. [ ] Add tests for admin endpoints
### Phase 6: Admin UI
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
19. [ ] Add "Copy Message" button (admin-only) with loading state
20. [ ] Add "Send Email" button (admin-only) with email input dialog
21. [ ] Add E2E test for admin buttons
### Phase 7: Documentation
22. [ ] Document template management in docs/DEVELOPMENT.md
23. [ ] Update docs/API.md with new endpoints
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
## Testing Strategy
**Unit Tests:**
- Template rendering with all variable combinations
- Email service CC support
- Completion/reminder email generation
- Idle detection logic
**Integration Tests:**
- Full flow: job completion → email sent
- Idle reminder: job enters blocking state → 5 min → reminder sent
- Admin endpoint authentication
**Manual Testing:**
- Verify emails render correctly in Gmail/Outlook
- Test admin buttons on completed job
- Test clipboard copy works across browsers
## Open Questions
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
## Rollback Plan
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
2. **Template fetch failures** - Fall back to hardcoded template in code
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
## Template Content
Initial template (from user's file, with feedback link added):
```
Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
```
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
## Updating Templates
To update email templates after deployment:
```bash
# Upload new template
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
# Templates are fetched fresh on each use (no caching in code)
# Changes take effect immediately
```
-279
View File
@@ -1,279 +0,0 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
@@ -1,239 +0,0 @@
# Plan: Song Lookup & Autocomplete + Community Version Detection
**Created:** 2026-03-02
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
**Status:** Implementation Complete
## Overview
Add two features to the karaoke-gen job creation flow:
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
## Requirements
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
- [ ] Community version check triggers when both artist and title are filled
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
- [ ] Community check does not block job submission — just informs the user
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
## Architecture Decisions
### How to access karaoke-decide catalog data
**Decision: Proxy through karaoke-gen backend**
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
| Option | Pros | Cons |
|--------|------|------|
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
### How to detect community karaoke versions
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
### Autocomplete UI component
**Decision: Use existing shadcn Command (cmdk) + Popover components**
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
- Keyboard navigation (arrow keys, enter to select)
- Debounced search-as-you-type
- Clean dropdown with loading states
- Works inline within forms (not modal)
## Technical Approach
### Data flow
```
User types in artist field
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
→ user selects "Fleetwood Mac" → field populated with canonical name
User types in title field
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
→ user selects "Dreams" → field populated with canonical title
Both fields filled → community check fires
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
→ if community tracks found with YouTube URLs, return them
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
```
### Caching strategy
| Endpoint | Cache TTL | Scope |
|----------|-----------|-------|
| Artist search proxy | 5 min | In-memory (backend) |
| Track search proxy | 5 min | In-memory (backend) |
| Community check | 1 hour | In-memory (backend) |
### karaoke-decide API endpoints used
These are all **public, no auth required**:
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
## Implementation Steps
### Phase 1: Backend — Catalog proxy + community check endpoints
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
- HTTP client (httpx async) calling karaoke-decide API
- In-memory TTL cache for responses (5 min)
- Configurable karaoke-decide base URL
- Graceful error handling (return empty results on failure)
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
- Add async support (httpx instead of requests)
- In-memory TTL cache (1 hour)
- Return only community tracks with YouTube URLs
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
- `POST /api/catalog/community-check` — scrape karaokenerds.com
- All endpoints require auth (same as job creation)
- Pydantic request/response models
4. [ ] **Register routes in main.py** — add `catalog.router` to app
5. [ ] **Add config** — `backend/config.py`
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
### Phase 2: Frontend — Autocomplete component
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
- Wraps an Input field with a Popover + Command dropdown
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
- Debounced search (300ms) triggered on input change
- Shows loading spinner while fetching
- Shows "No results" when search returns empty
- Keyboard navigation (up/down arrows, enter to select, escape to close)
- Clicking outside closes dropdown
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
- When user types without selecting, uses their raw text (autocomplete is optional)
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
### Phase 3: Frontend — Integrate autocomplete into job creation
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
- Fetch suggestions from `api.searchArtists(query)`
- On select: populate artist field with canonical name
- Show artist name + optional disambiguation in dropdown items
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
- On select: populate title field with canonical name
- Show track name + artist name + duration in dropdown items
### Phase 4: Frontend — Community version detection
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
- Shows when community karaoke version(s) found
- Displays: "A karaoke version of this song already exists!"
- Lists community tracks with brand name and YouTube link
- "Watch on YouTube" button for top community version
- Dismissible (X button or "Continue anyway" link)
- Styled with green border (matching kjbox community track styling)
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
- Debounced check (500ms) when both artist AND title fields have values
- Only triggers when values change (not on every keystroke)
- Shows `CommunityVersionBanner` between the form fields and the submit button
- Does NOT block form submission
- Clears when artist or title changes
### Phase 5: Testing
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
- Test proxy endpoints with mocked karaoke-decide responses
- Test community check with mocked karaokenerds HTML
- Test error handling (karaoke-decide down, karaokenerds down)
- Test caching behavior
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
- Test HTML parsing, community detection, YouTube URL cleanup
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
- Test suggestion fetching and display
- Test keyboard navigation
- Test selection behavior
- Test free-text input (no selection)
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
- Test autocomplete appears on typing
- Test selecting from autocomplete populates field
- Test community banner appears for known songs
- Test form submission works with and without autocomplete selection
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
| `backend/main.py` | Modify | Register catalog router |
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
## Decisions (Resolved)
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
## Rollback Plan
- Feature is entirely additive — no existing functionality is modified
- Autocomplete fields fall back to plain text Input if API calls fail
- Community check is non-blocking — if it fails, form works normally
- All new endpoints are behind the existing auth middleware
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
@@ -1,329 +0,0 @@
# Plan: Unified Combined Review for Local CLI
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Problem Statement
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
- Review code lives inside `correct_lyrics()` method in controller.py
- When auto-correction is disabled, `correct_lyrics()` is never called
- Users never get a chance to review lyrics
2. **Standalone instrumental UI still exists for CLI**
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
- CLI launches this standalone server after lyrics review
- The combined review flow we built only works for the web app
## Goals
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
4. **Keep it simple** - Minimal changes to achieve the goal
## Architecture Overview
### Current State (Broken)
```
CLI Flow:
controller.py::process()
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
→ ReviewServer starts (lyrics only)
→ User reviews
→ ReviewServer stops
→ gen_cli.py::run_instrumental_review()
→ InstrumentalReviewServer starts (instrumental only)
→ User selects instrumental
→ InstrumentalReviewServer stops
Cloud Web App Flow:
Job enters AWAITING_REVIEW
→ User visits /app/jobs/{id}/review
→ Combined UI: lyrics + instrumental
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
```
### Target State
```
CLI Flow (and Cloud CLI):
controller.py::process()
→ _run_human_review() [NEW - always called when review enabled]
→ ReviewServer starts (combined: lyrics + instrumental)
→ User reviews lyrics AND selects instrumental
→ ReviewServer stops, returns both results
→ Continue with selected instrumental
Cloud Web App Flow: (unchanged)
Same as before
```
## Key Insight
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
3. Call the review flow from the right place (not inside `correct_lyrics()`)
4. Delete `InstrumentalReviewServer` entirely
## Implementation Steps
### Phase 1: Fix ReviewServer to Support Instrumental Selection
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
```python
def __init__(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None, # NEW
backing_vocals_analysis: dict | None = None, # NEW
...
):
```
Update `_correction_data()` endpoint to include these in response:
```python
@app.get("/api/review/{job_id}/correction-data")
def _correction_data(job_id: str):
return {
"lyrics_data": self.correction_result.to_dict(),
"instrumental_options": self.instrumental_options, # NEW
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
}
```
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
```python
@app.post("/api/review/{job_id}/complete-review")
def _complete_review(job_id: str, request: ReviewCompleteRequest):
# Existing: store corrected lyrics
self.corrected_lyrics = request.corrected_lyrics
# NEW: store instrumental selection
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
self.review_completed = True
```
**1.3. Expose instrumental selection after review completes**
The calling code needs to get the selection. Update the return value or add an attribute:
```python
# After review loop completes:
return {
"corrected_lyrics": self.corrected_lyrics,
"instrumental_selection": self.instrumental_selection
}
```
### Phase 2: Fix controller.py to Call Review at the Right Time
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Create new method `_run_human_review()` that can be called independently:
```python
def _run_human_review(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None,
backing_vocals_analysis: dict | None = None
) -> tuple[CorrectionResult, str | None]:
"""Run human review server and return corrected result + instrumental selection."""
if not self.output_config.review_enabled:
return correction_result, None
server = ReviewServer(
correction_result=correction_result,
instrumental_options=instrumental_options,
backing_vocals_analysis=backing_vocals_analysis,
...
)
result = server.start() # Blocks until review complete
return result["corrected_lyrics"], result.get("instrumental_selection")
```
**2.2. Update `process()` to always call review when appropriate**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Current flow (simplified):
```python
def process(self):
self.transcribe()
self.separate_audio()
if self.output_config.run_correction:
self.correct_lyrics() # Review is inside here!
# ...
```
New flow:
```python
def process(self):
self.transcribe()
self.separate_audio()
# Build correction result (with or without auto-correction)
if self.output_config.run_correction:
correction_result = self._run_auto_correction()
else:
correction_result = self._create_uncorrected_result()
# Human review is separate from auto-correction
if self.output_config.review_enabled:
instrumental_options = self._get_instrumental_options()
backing_vocals_analysis = self._analyze_backing_vocals()
corrected, instrumental_selection = self._run_human_review(
correction_result,
instrumental_options,
backing_vocals_analysis
)
self.results.correction_result = corrected
self.results.instrumental_selection = instrumental_selection
```
### Phase 3: Update gen_cli.py to Use Combined Review
**3.1. Remove standalone instrumental review call**
File: `karaoke_gen/utils/gen_cli.py`
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
Before:
```python
# After lyrics processing
if not args.non_interactive:
instrumental_selection = run_instrumental_review(...)
```
After:
```python
# Instrumental selection comes from controller.results.instrumental_selection
instrumental_selection = controller.results.instrumental_selection
```
**3.2. Pass instrumental options to controller**
The controller needs instrumental options to display in review. These come from the separated audio stems:
```python
controller = LyricsTranscriber(
# ... existing params ...
)
# Before calling process(), set up instrumental options
# These are typically discovered during audio separation
```
### Phase 4: Delete Standalone InstrumentalReviewServer
**4.1. Delete files**
- Delete `karaoke_gen/instrumental_review/server.py`
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
**4.2. Clean up imports**
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
### Phase 5: Update Frontend for Local Mode
**5.1. Verify local mode detection works**
File: `frontend/lib/api.ts`
The frontend already has local mode detection based on hostname and port. Verify it works:
```typescript
const isLocalMode = () => {
if (typeof window === 'undefined') return false;
const hostname = window.location.hostname;
const port = parseInt(window.location.port, 10);
return hostname === 'localhost' && port >= 8000 && port <= 8770;
};
```
**5.2. Verify correction-data endpoint parsing**
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
### Phase 6: Testing
**6.1. Manual testing checklist**
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
- [ ] Combined UI shows both lyrics editor and instrumental selector
- [ ] Selecting instrumental and completing review works
- [ ] No standalone instrumental review page appears
- [ ] Cloud web app still works (no regression)
**6.2. Update unit tests**
- Remove tests for `InstrumentalReviewServer`
- Add tests for `ReviewServer` with instrumental options
- Add tests for `controller._run_human_review()`
## Files to Modify
| File | Action | Description |
|------|--------|-------------|
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
| Missing instrumental options in local mode | Fail clearly if options not provided |
| Review server port conflicts | Keep existing port allocation logic |
## Open Questions
1. **Where does backing vocals analysis run for local CLI?**
- Currently runs in `render_video_worker.py` for cloud
- For local, need to run it after audio separation, before review
- May need to call analysis function from controller
2. **How are instrumental stems exposed to local review?**
- Need to serve audio files from ReviewServer
- Or use file:// URLs (may have CORS issues)
- Existing approach: ReviewServer already serves static files
## Success Criteria
1. `SKIP_CORRECTION=true` no longer skips human review
2. Single combined review UI appears (no separate instrumental step)
3. All platforms (CLI, cloud CLI, web) use same review flow
4. No dead code remaining for standalone instrumental review
-317
View File
@@ -1,317 +0,0 @@
# Plan: White Label B2B Karaoke Portals
**Created:** 2026-01-08
**Branch:** feat/sess-20260108-0358-white-label-portal
**Status:** Draft
## Overview
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
**First customer: Vocal Star**
- Subdomain: `vocalstar.nomadkaraoke.com`
- Workflow: Upload original + instrumental audio (no audio search)
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
- Distribution: Download only (no Dropbox/GDrive/YouTube)
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
## Requirements
### Functional Requirements
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
- [ ] Per-tenant frontend branding (logo, colors, metadata)
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
- [ ] Per-tenant default theme (always use Vocal Star theme)
- [ ] Per-tenant allowed email domains for magic link auth
- [ ] Per-tenant fixed API token support (for automation)
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
- [ ] Users scoped to tenant
### Non-Functional Requirements
- [ ] Minimal code duplication - single codebase serves all tenants
- [ ] Easy to add new tenants via config, not code changes
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
- [ ] No performance impact on existing single-tenant flow
## Technical Approach
### Architecture Decision: Config-Driven Multi-Tenancy
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
2. **Frontend detects subdomain** and fetches tenant config from API
3. **Backend middleware** extracts tenant from request and applies config
4. **Firestore collections** get `tenant_id` field for isolation
This approach:
- Requires no database migrations for new tenants
- Configs can be managed via simple JSON files
- Easy to test locally with query param override (`?tenant=vocalstar`)
- Can evolve to admin UI later if needed
### Tenant Config Schema
```json
{
"id": "vocalstar",
"name": "Vocal Star",
"subdomain": "vocalstar.nomadkaraoke.com",
"branding": {
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
"primary_color": "#ffff00",
"secondary_color": "#006CF9",
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
"site_title": "Vocal Star Karaoke Generator"
},
"features": {
"audio_search": false,
"file_upload": true,
"youtube_upload": false,
"dropbox_upload": false,
"gdrive_upload": false,
"theme_selection": false
},
"defaults": {
"theme_id": "vocalstar",
"distribution_mode": "download_only"
},
"auth": {
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
"fixed_tokens": ["<will be in Secret Manager>"]
}
}
```
### Key Implementation Details
#### 1. Subdomain Detection (Frontend)
- Next.js middleware checks `request.headers.host`
- If subdomain matches tenant, store tenant ID in cookie/context
- Fetch `/api/tenant/config` to get branding/features
- Apply CSS variables dynamically for colors
- Swap logo component based on config
#### 2. Tenant Middleware (Backend)
- FastAPI middleware extracts tenant from:
- `X-Tenant-ID` header (set by frontend)
- Or subdomain from `Host` header
- Or query param `?tenant=` (dev only)
- Attaches `tenant_id` to request state
- Tenant config loaded from GCS (cached 5 min like themes)
#### 3. Auth Enhancement
- `AuthService.validate_token_full()` already returns `user_email`
- Add tenant validation: check if user's email domain is in tenant's allowed list
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
- Magic links include tenant context for proper redirect
#### 4. Data Isolation
- Add `tenant_id: Optional[str]` to User and Job models
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
- Query filters: `where('tenant_id', '==', tenant_id or None)`
- No schema migration needed - new field, optional
#### 5. Theme Integration
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
- Tenant config specifies `default_theme_id: "vocalstar"`
- When `theme_selection: false`, UI hides theme picker and uses default
## Implementation Steps
### Phase 1: Backend Tenant Foundation (Day 1-2)
1. [ ] **Create tenant model and config loader**
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
- Create `backend/services/tenant_service.py` with GCS config loading and caching
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
2. [ ] **Add tenant middleware**
- Create `backend/api/middleware/tenant.py`
- Extract tenant from subdomain or header
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
- Add `get_tenant()` dependency for routes
3. [ ] **Update data models**
- Add `tenant_id: Optional[str] = None` to `User` model
- Add `tenant_id: Optional[str] = None` to `Job` model
- Update Firestore indexes in Pulumi for tenant-scoped queries
4. [ ] **Update auth for tenant awareness**
- Modify `AuthService` to validate email domain against tenant config
- Add tenant context to magic link generation
- Support per-tenant fixed tokens in `auth_tokens` collection
### Phase 2: Upload Vocal Star Theme (Day 2)
5. [ ] **Prepare Vocal Star theme for GCS**
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
- Upload background images to `themes/vocalstar/assets/`
- Upload Oswald font to `themes/vocalstar/assets/`
- Upload `style_params.json` to `themes/vocalstar/`
- Create preview image and upload to `themes/vocalstar/preview.png`
- Add Vocal Star to `themes/_metadata.json`
6. [ ] **Create Vocal Star tenant config**
- Create `tenants/vocalstar/config.json` in GCS
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
- Upload favicon to `tenants/vocalstar/favicon.ico`
### Phase 3: Frontend Tenant Support (Day 3-4)
7. [ ] **Add tenant context provider**
- Create `frontend/lib/tenant.ts` with tenant config types
- Create `frontend/hooks/useTenant.ts` for tenant state
- Create `TenantProvider` component that fetches config on mount
- Detect subdomain in `_app.tsx` or middleware
8. [ ] **Dynamic branding**
- Create `TenantLogo` component that renders tenant or default logo
- Update `globals.css` to support CSS variable overrides
- Add `applyTenantColors()` function to set CSS variables from config
- Update page metadata (title, favicon) from tenant config
9. [ ] **Feature-flagged UI**
- Wrap audio search in `{tenant.features.audio_search && ...}`
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
- Wrap distribution options based on tenant features
- Show simplified form for Vocal Star (just file upload fields)
10. [ ] **Update magic link flow**
- Pass tenant context when requesting magic link
- Magic link redirect goes to tenant subdomain
- Verify endpoint extracts tenant from link context
### Phase 4: Backend Feature Enforcement (Day 4-5)
11. [ ] **Enforce tenant features in API**
- Audio search endpoint: return 403 if `audio_search: false`
- File upload: check `file_upload: true`
- Distribution settings: ignore/error if disabled features requested
- Theme selection: use default if `theme_selection: false`
12. [ ] **Job creation with tenant context**
- `JobCreate` handler sets `tenant_id` from request state
- Distribution config respects tenant defaults
- Theme defaults to tenant's default theme
13. [ ] **Query scoping**
- Update `JobManager.get_jobs()` to filter by tenant_id
- Update `UserService` queries to include tenant_id
- Admin endpoints bypass tenant filter (see all tenants)
### Phase 5: Infrastructure & Deployment (Day 5-6)
14. [ ] **DNS setup**
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
- Configure Cloudflare Pages to accept wildcard subdomain
- Add SSL certificate for subdomain
15. [ ] **Backend CORS update**
- Add `vocalstar.nomadkaraoke.com` to allowed origins
- Or use pattern matching for `*.nomadkaraoke.com`
16. [ ] **Pulumi updates**
- Add Firestore indexes for tenant_id queries
- Add any new secrets (tenant fixed tokens)
### Phase 6: Testing & Documentation (Day 6-7)
17. [ ] **Testing**
- Unit tests for TenantService config loading
- Unit tests for tenant middleware
- E2E test for Vocal Star flow (upload audio, generate, download)
- Test magic link works for @vocal-star.com email
18. [ ] **Documentation**
- Update CLAUDE.md with tenant info
- Create `docs/WHITE-LABEL.md` with tenant setup guide
- Document how to add new tenants
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/tenant.py` | Create | Tenant config models |
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
| `backend/models/user.py` | Modify | Add `tenant_id` field |
| `backend/models/job.py` | Modify | Add `tenant_id` field |
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
| `backend/main.py` | Modify | Add tenant middleware |
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
## Testing Strategy
### Unit Tests
- `TenantService`: Config loading, caching, validation
- `TenantMiddleware`: Subdomain extraction, header extraction
- `AuthService`: Tenant email domain validation
- `JobManager`: Tenant-scoped queries
### Integration Tests (Emulator)
- Create job with tenant context
- Query jobs filtered by tenant
- Auth with tenant-allowed email domain
- Auth rejection for wrong email domain
### E2E Tests
- Vocal Star happy path: Upload audio files, review lyrics, download video
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
- Feature hiding: Verify audio search not shown on Vocal Star portal
## Open Questions
- [x] **Q: Should tenant configs be in GCS or Firestore?**
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
- [x] **Q: How to handle localhost development for specific tenant?**
A: Query param override `?tenant=vocalstar` for local testing
- [x] **Q: Should admin portal be tenant-aware?**
A: No - admins see all tenants. Admin routes bypass tenant filter.
- [x] **Q: Do we need tenant-specific email sender identity?**
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
## Rollback Plan
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
1. Revert frontend to not check tenant
2. Revert backend middleware to not extract tenant
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
4. No schema migration to reverse
For DNS: Simply remove CNAME record if needed.
## Success Criteria
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
- [ ] Portal shows Vocal Star branding (logo, colors)
- [ ] Audio search is hidden; only file upload available
- [ ] Theme is locked to Vocal Star style
- [ ] Distribution options hidden; download only
- [ ] Magic link works for @vocal-star.com emails
- [ ] Fixed API token works for automation
- [ ] Generated videos use Vocal Star theme
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
-58
View File
@@ -1,58 +0,0 @@
#!/usr/bin/env bash
# Nomad Karaoke pre-commit hook (karaoke-gen variant)
# 1. Gitleaks secret scan (fail fast)
# 2. i18n auto-translation when en.json changes
# Enable: git config core.hooksPath .githooks
set -eu
# ---------------------------------------------------------------------------
# 1. Gitleaks secret scan
# ---------------------------------------------------------------------------
if command -v gitleaks >/dev/null 2>&1; then
if ! gitleaks protect --staged --redact --no-banner; then
printf '\n\033[31m✗ gitleaks detected a potential secret in staged changes.\033[0m\n' >&2
printf ' • If real: remove it, rotate the credential, then re-commit.\n' >&2
printf ' • If false positive: add an allowlist entry to .gitleaks.toml.\n' >&2
printf ' • Emergency override (not recommended): git commit --no-verify\n' >&2
exit 1
fi
else
printf '\033[33mwarn:\033[0m gitleaks not installed — skipping secret scan\n' >&2
printf ' install: brew install gitleaks\n' >&2
fi
# ---------------------------------------------------------------------------
# 2. i18n auto-translation
# ---------------------------------------------------------------------------
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
if [ -z "$STAGED_EN" ]; then
exit 0
fi
echo "i18n: English translations changed — running translation pipeline..."
for en_file in $STAGED_EN; do
dir=$(dirname "$en_file")
if [[ "$dir" == frontend/messages ]]; then
script="frontend/scripts/translate.py"
elif [[ "$dir" == backend/translations ]]; then
script="frontend/scripts/translate.py"
else
echo " Warning: Unknown messages dir $dir, skipping"
continue
fi
echo " Translating $dir..."
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
git add "$dir"/*.json
echo " Done: $dir"
else
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
fi
done
echo "i18n: Translation complete."
@@ -1,115 +0,0 @@
# Build Encoding Worker GCE Image
#
# Rebuilds the custom Packer image when provisioning changes.
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
#
# Triggers:
# - Push to main with changes to infrastructure/packer/**
# - Manual dispatch (workflow_dispatch)
#
# After image build, the encoding-worker VM will use the new image
# on next recreation (e.g., via pulumi up or manual restart).
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
inputs:
python_version:
description: 'Python version to install'
required: false
default: '3.13.1'
concurrency:
group: build-encoding-worker-image
cancel-in-progress: false # Don't cancel in-progress builds
jobs:
build-image:
name: Build Packer Image
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # Required for Workload Identity Federation
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Packer
run: |
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y packer
- name: Initialize Packer
working-directory: infrastructure/packer
run: packer init encoding-worker.pkr.hcl
- name: Validate Packer Template
working-directory: infrastructure/packer
run: packer validate encoding-worker.pkr.hcl
- name: Build Image
working-directory: infrastructure/packer
env:
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
run: |
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
packer build \
-var "python_version=${PYTHON_VERSION}" \
encoding-worker.pkr.hcl
- name: Get Latest Image Info
id: image-info
run: |
IMAGE_NAME=$(gcloud compute images list \
--project=nomadkaraoke \
--filter="family:encoding-worker" \
--sort-by="~creationTimestamp" \
--limit=1 \
--format="value(name)")
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
echo "Built image: ${IMAGE_NAME}"
- name: Summary
run: |
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
notify-on-failure:
name: Notify on Failure
runs-on: ubuntu-latest
needs: build-image
if: failure()
steps:
- name: Log Failure
run: |
echo "::error::Encoding worker image build failed. Check the build-image job logs."
-298
View File
@@ -1,298 +0,0 @@
name: Build GHA Runner Images
# Produces the custom GCE images used by the ephemeral GitHub Actions runner
# dispatcher: gha-runner-general, gha-runner-build, gha-runner-gpu,
# gha-runner-gpu-windows.
#
# Flow per variant:
# 1. Create a temporary build VM from the variant's base image with the
# variant's provisioning script as its startup script:
# - Linux variants: debian-12 + runner-image-provision.sh (startup-script)
# - gpu-windows: windows-2022 + runner-image-provision.ps1
# (windows-startup-script-ps1)
# 2. Poll the serial console for the READY/FAILED marker (survives reboots).
# 3. Stop the VM, snapshot its boot disk to a new image in the `gha-runner-<variant>`
# image family with name gha-runner-<variant>-<timestamp>.
# 4. Delete the build VM (boot disk auto-deletes).
# 5. Deprecate older images in the family, keeping the newest 3 ACTIVE.
on:
workflow_dispatch:
inputs:
variants:
description: 'Comma-separated variants to build (general,build,gpu,gpu-windows) or "all"'
required: false
default: 'all'
skip_prune:
description: 'Skip deprecation of old images (debugging)'
type: boolean
required: false
default: false
schedule:
# Monthly on the 1st at 02:00 UTC
- cron: '0 2 1 * *'
permissions:
contents: read
id-token: write
env:
GCP_PROJECT: nomadkaraoke
BUILD_REGION: us-central1
BUILD_ZONE: us-central1-a
BASE_IMAGE: projects/debian-cloud/global/images/family/debian-12
KEEP_IMAGES_PER_FAMILY: 3
jobs:
resolve-matrix:
name: Resolve variant matrix
runs-on: ubuntu-latest
outputs:
variants: ${{ steps.resolve.outputs.variants }}
steps:
- id: resolve
run: |
requested='${{ github.event.inputs.variants || 'all' }}'
if [[ "$requested" == "all" || -z "$requested" ]]; then
echo 'variants=["general","build","gpu","gpu-windows"]' >> "$GITHUB_OUTPUT"
else
json="$(echo "$requested" | tr ',' '\n' | jq -R . | jq -sc .)"
echo "variants=$json" >> "$GITHUB_OUTPUT"
fi
build:
name: Build ${{ matrix.variant }} image
needs: resolve-matrix
runs-on: ubuntu-latest
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
variant: ${{ fromJSON(needs.resolve-matrix.outputs.variants) }}
steps:
- uses: actions/checkout@v4
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
project_id: ${{ env.GCP_PROJECT }}
- name: Resolve build parameters
id: params
run: |
variant='${{ matrix.variant }}'
ts="$(date -u +%Y%m%d-%H%M%S)"
name="gha-runner-${variant}-${ts}"
family="gha-runner-${variant}"
vm="gha-image-build-${variant}-${ts}"
# Per-variant base image + startup-script metadata key: Windows
# VMs only execute scripts under windows-startup-script-ps1.
image_flag="--image=${{ env.BASE_IMAGE }}"
metadata_file="startup-script=infrastructure/scripts/runner-image-provision.sh"
case "$variant" in
general|build)
machine="n2-standard-8"
disk_size="50"
accel=""
;;
gpu)
# Build with a T4 attached so DKMS module is built and validated
# against the kernel that ships in the image.
machine="n1-standard-4"
disk_size="200"
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
;;
gpu-windows)
# T4 attached so the GRID driver install can validate WDDM mode
# (required for DirectML). Keep disk in sync with the
# gpu-windows FamilySpec in runner_manager/ephemeral.py.
machine="n1-standard-4"
disk_size="100"
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
image_flag="--image-family=windows-2022 --image-project=windows-cloud"
metadata_file="windows-startup-script-ps1=infrastructure/scripts/runner-image-provision.ps1"
;;
*)
echo "Unknown variant: $variant" >&2
exit 1
;;
esac
{
echo "name=$name"
echo "family=$family"
echo "vm=$vm"
echo "machine=$machine"
echo "disk_size=$disk_size"
echo "accel=$accel"
echo "image_flag=$image_flag"
echo "metadata_file=$metadata_file"
} >> "$GITHUB_OUTPUT"
- name: Create build VM
run: |
set -euo pipefail
gcloud compute instances create '${{ steps.params.outputs.vm }}' \
--project='${{ env.GCP_PROJECT }}' \
--zone='${{ env.BUILD_ZONE }}' \
--machine-type='${{ steps.params.outputs.machine }}' \
${{ steps.params.outputs.image_flag }} \
--boot-disk-size='${{ steps.params.outputs.disk_size }}GB' \
--boot-disk-type=pd-balanced \
--boot-disk-auto-delete \
--service-account="${{ secrets.GCP_SERVICE_ACCOUNT }}" \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--metadata=enable-oslogin=TRUE,image-variant='${{ matrix.variant }}' \
--metadata-from-file='${{ steps.params.outputs.metadata_file }}' \
--labels=purpose=gha-runner-image-builder,variant='${{ matrix.variant }}' \
${{ steps.params.outputs.accel }}
- name: Wait for provisioning to complete
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
set -uo pipefail
# Read the serial console — NOT SSH — to detect completion.
#
# The provisioner script tees its full output to /dev/ttyS0 (the GCE
# serial console). `gcloud compute instances get-serial-port-output`
# returns the full console buffer with zero SSH/IAP dependency. This
# makes the wait robust against the IAP tunnel issues that previously
# caused both false-positive readiness and false-negative timeouts.
#
# The script writes one of two distinctive marker lines:
# "### runner-image: READY: ..." — successful completion
# "### runner-image-provision FAILED ..." — fatal error (with line+rc)
#
# We also stream live progress to the GHA log so the operator can
# see what's happening without leaving the run page.
deadline=$(( $(date +%s) + 75 * 60 ))
last_size=0
attempt=0
while (( $(date +%s) < deadline )); do
attempt=$((attempt+1))
console=$(gcloud compute instances get-serial-port-output "$VM" \
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null || true)
cur_size=${#console}
# Stream new provisioner lines (filter syslog noise).
if (( cur_size > last_size )); then
echo "$console" | tail -c +$((last_size+1)) \
| grep -E "runner-image|### |Attempt " || true
last_size=$cur_size
fi
# Success marker
if echo "$console" | grep -qE 'runner-image: READY:'; then
echo
echo "### Image ready (attempt $attempt)"
echo "$console" | grep -E 'runner-image: READY:' | tail -1
exit 0
fi
# Failure marker
if echo "$console" | grep -qE 'runner-image-provision FAILED'; then
echo
echo "### Provisioner reported FAILURE — failure block + 40 lines of preceding context:" >&2
# -B 40: include 40 lines BEFORE the marker (where the actual
# error output of the failing command lives — python tracebacks,
# apt errors, curl messages, etc).
# -A 5: include the variant/line/cmd metadata lines after the marker.
echo "$console" | grep -B 40 -A 5 'runner-image-provision FAILED' \
| tail -80 >&2
exit 1
fi
sleep 30
done
echo
echo "### Timed out waiting for READY marker after 75 minutes" >&2
echo "### Last serial console output (tail):" >&2
gcloud compute instances get-serial-port-output "$VM" \
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null \
| grep -E "runner-image|### " | tail -50 >&2 || true
exit 1
- name: Stop build VM
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
set -euo pipefail
gcloud compute instances stop "$VM" --zone="$ZONE" --project="$PROJECT" --quiet
# Belt-and-suspenders wait for TERMINATED.
for _ in $(seq 1 60); do
status=$(gcloud compute instances describe "$VM" --zone="$ZONE" --project="$PROJECT" --format='value(status)')
[[ "$status" == "TERMINATED" ]] && break
sleep 5
done
- name: Create image
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
IMAGE: ${{ steps.params.outputs.name }}
FAMILY: ${{ steps.params.outputs.family }}
run: |
set -euo pipefail
# The "Create build VM" step doesn't pass --device-name or
# --boot-disk-name, so GCE auto-names the boot disk after the
# instance. We rely on that here instead of extracting the name
# via a format expression — the previous
# `disks[0].source.scope(disks).segment(1)` selector silently
# returned empty and gcloud rejected the call with
# "You cannot specify [--source-disk-zone] unless you are
# specifying [--source-disk]".
DISK="$VM"
echo "Creating image $IMAGE in family $FAMILY from disk $DISK in $ZONE..."
gcloud compute images create "$IMAGE" \
--project="$PROJECT" \
--source-disk="$DISK" \
--source-disk-zone="$ZONE" \
--family="$FAMILY" \
--description="GHA ephemeral runner image (${{ matrix.variant }}) built ${{ github.run_id }}" \
--labels=purpose=gha-ephemeral-runner,variant=${{ matrix.variant }},built-by=gha
echo "Created $IMAGE in family $FAMILY"
- name: Delete build VM
if: always()
env:
VM: ${{ steps.params.outputs.vm }}
ZONE: ${{ env.BUILD_ZONE }}
PROJECT: ${{ env.GCP_PROJECT }}
run: |
gcloud compute instances delete "$VM" --zone="$ZONE" --project="$PROJECT" --quiet || true
- name: Deprecate old images (keep newest ${{ env.KEEP_IMAGES_PER_FAMILY }})
if: ${{ github.event.inputs.skip_prune != 'true' }}
env:
PROJECT: ${{ env.GCP_PROJECT }}
FAMILY: ${{ steps.params.outputs.family }}
KEEP: ${{ env.KEEP_IMAGES_PER_FAMILY }}
run: |
set -euo pipefail
# List ACTIVE images in this family, newest first, skip the top $KEEP.
to_deprecate=$(gcloud compute images list \
--project="$PROJECT" \
--filter="family=$FAMILY AND status=READY AND -deprecated.state:*" \
--sort-by='~creationTimestamp' \
--format='value(name)' | tail -n +$((KEEP+1)))
if [[ -z "$to_deprecate" ]]; then
echo "Nothing to deprecate."
exit 0
fi
for img in $to_deprecate; do
echo "Deprecating $img"
gcloud compute images deprecate "$img" \
--project="$PROJECT" \
--state=DEPRECATED \
--replacement="projects/$PROJECT/global/images/family/$FAMILY"
done
+125 -1009
View File
File diff suppressed because it is too large. Load diff
-155
View File
@@ -1,155 +0,0 @@
name: DR backup freshness monitor
# Off-platform monitor: runs in GitHub (not GCP) so it survives a full GCP
# compromise. If the nightly backup pipeline dies silently, this catches it.
#
# Required secrets:
# AWS_BACKUP_READONLY_ACCESS_KEY_ID
# AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
# DR_MONITOR_DISCORD_WEBHOOK
#
# Setup: see docs/DISASTER-RECOVERY.md § "Backup freshness monitor".
on:
schedule:
# Daily at 14:00 UTC (~9h after the 1am ET backup window).
- cron: "0 14 * * *"
workflow_dispatch: {}
permissions:
contents: read
jobs:
check-freshness:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BACKUP_READONLY_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BACKUP_READONLY_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
BUCKET: nomadkaraoke-backup
DISCORD_WEBHOOK: ${{ secrets.DR_MONITOR_DISCORD_WEBHOOK }}
steps:
- name: Verify required secrets are set
run: |
missing=0
for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY DISCORD_WEBHOOK; do
if [ -z "${!var}" ]; then
echo "::error::Required secret $var is empty"
missing=1
fi
done
[ "$missing" = "0" ] || exit 1
- name: Install boto3
run: pip install --quiet boto3
- name: Check freshness of each tracked prefix
id: check
shell: bash
run: |
python3 - <<'PY'
import os, sys, datetime, boto3
BUCKET = os.environ["BUCKET"]
# (label, prefix, max_age_hours)
# NOTE: Firestore and BigQuery export to GCS staging nightly but are
# only uploaded to S3 *weekly* (Sundays) to cut cross-cloud egress —
# see backup_to_aws/main.py (firestore_to_s3_today). This monitor only
# sees the S3 (off-site) copy, so their limits must allow a full week
# plus slack (192h ≈ 8 days), NOT the 36h nightly cadence. GCS job
# files and secrets DO upload to S3 nightly, so they stay at 36h.
TARGETS = [
("Firestore", "firestore/", 192),
("GCS job files", "gcs/job-files/", 36),
("Encrypted secrets", "secrets/", 36),
("BigQuery weekly", "bigquery/daily-refresh/", 192),
]
s3 = boto3.client("s3")
now = datetime.datetime.now(datetime.timezone.utc)
stale, ok = [], []
# Paginated scan — JMESPath --query on `aws s3api list-objects-v2`
# operates per-page, which silently breaks "newest across all pages"
# queries on prefixes with >1000 objects (e.g. firestore/ has ~22k).
paginator = s3.get_paginator("list_objects_v2")
for label, prefix, max_hours in TARGETS:
latest = None
for page in paginator.paginate(Bucket=BUCKET, Prefix=prefix):
for obj in page.get("Contents", []):
if latest is None or obj["LastModified"] > latest:
latest = obj["LastModified"]
if latest is None:
stale.append(f"{label}: NO OBJECTS in s3://{BUCKET}/{prefix}")
continue
age = (now - latest).total_seconds()
age_h = int(age // 3600)
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
(stale if age > max_hours * 3600 else ok).append(line)
summary = ["## DR backup freshness", ""]
if stale:
summary.append("### Stale")
summary.extend(f"- {s}" for s in stale)
summary.append("")
summary.append("### Fresh")
summary.extend(f"- {o}" for o in ok)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write("\n".join(summary) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"ok_count={len(ok)}\n")
f.write(f"stale_count={len(stale)}\n")
if stale:
f.write("STALE_LIST<<EOF\n")
f.write("\n".join(stale) + "\n")
f.write("EOF\n")
for o in ok:
print("OK:", o)
if stale:
for s in stale:
print("STALE:", s, file=sys.stderr)
sys.exit(1)
PY
- name: Alert Discord on stale backup
if: failure() && steps.check.outputs.stale_count != ''
env:
STALE: ${{ steps.check.outputs.STALE_LIST }}
run: |
payload=$(jq -n \
--arg title "🚨 DR backup is stale" \
--arg desc "$STALE" \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15158332,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
- name: Alert Discord on workflow failure (catch-all)
if: failure() && steps.check.outputs.stale_count == ''
run: |
payload=$(jq -n \
--arg title "⚠️ DR freshness monitor failed to run" \
--arg desc "Could not list S3 backup bucket — credentials, network, or bucket policy may be broken." \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15844367,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
-345
View File
@@ -1,345 +0,0 @@
name: E2E Daily Test (Payment + Happy Path)
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
on:
schedule:
- cron: '0 6 * * *'
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
skip_payment:
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
required: false
default: 'false'
type: boolean
use_test_token:
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
# =========================================================================
# Stage 1: Purchase credits via real Stripe Checkout
# =========================================================================
e2e-credit-purchase:
name: "Stage 1: Credit Purchase"
runs-on: ubuntu-latest
timeout-minutes: 15
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
outputs:
session_token: ${{ steps.extract_token.outputs.token }}
result: ${{ steps.test_result.outputs.result }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run Credit Purchase Test
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
echo "Time: $(date)"
npx playwright test credit-purchase-real.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=600000 \
2>&1 | tee test-output-payment.log
# Capture Playwright's real exit code, not tee's (0) — the
# authoritative pass/fail signal. See Stage 2 for rationale.
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Check test result
id: test_result
run: |
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
echo "result=success" >> "$GITHUB_OUTPUT"
else
echo "result=failure" >> "$GITHUB_OUTPUT"
fi
- name: Extract session token
id: extract_token
if: steps.test_result.outputs.result == 'success'
working-directory: frontend
run: |
if [ -f test-results/e2e-session-token.txt ]; then
TOKEN=$(cat test-results/e2e-session-token.txt)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
echo "Session token extracted for Stage 2"
else
echo "No session token file found"
echo "token=" >> $GITHUB_OUTPUT
fi
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-payment-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/test-output-payment.log
retention-days: 30
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
# =========================================================================
e2e-happy-path:
name: "Stage 2: Happy Path"
runs-on: ubuntu-latest
timeout-minutes: 90
needs: e2e-credit-purchase
if: |
always() && (
needs.e2e-credit-purchase.result == 'success' ||
github.event.inputs.skip_payment == 'true' ||
github.event.inputs.use_test_token == 'true'
)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Refresh E2E session token (when use_test_token enabled)
id: refresh_token
if: github.event.inputs.use_test_token == 'true'
run: |
RESPONSE=$(curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run Happy Path Test
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 2: Happy Path ==="
echo "Time: $(date)"
TOKEN_SOURCE="none"
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
TOKEN_SOURCE="use_test_token (impersonation)"
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
TOKEN_SOURCE="Stage 1 (credit purchase)"
fi
echo "Token source: $TOKEN_SOURCE"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output-happy-path.log
# Capture Playwright's real exit code, not tee's (0). This is the
# authoritative pass/fail signal — grepping the log for "failed" is
# fragile because benign retry warnings (e.g. "goto attempt 1/3
# failed") contain that substring and produce false alarms.
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Check test result
id: test_result
run: |
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
echo "result=success" >> "$GITHUB_OUTPUT"
else
echo "result=failure" >> "$GITHUB_OUTPUT"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-happy-path.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Final: Send notifications
# =========================================================================
notify:
name: "Notify"
runs-on: ubuntu-latest
needs: [e2e-credit-purchase, e2e-happy-path]
if: always()
steps:
- name: Discord notification (success)
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Send email notification
if: github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.postmarkapp.com
server_port: 587
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Daily Test Results
======================
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+173
View File
@@ -0,0 +1,173 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
-132
View File
@@ -1,132 +0,0 @@
name: E2E Daily Test (Tenant Portals)
# Full production E2E per white-label tenant: submit a job with the tenant's own
# mixed + instrumental audio, approve the lyrics review, wait for the render, and assert
# the tenant distribution (locked theme, no YouTube, no Google Drive, Dropbox folder +
# downloadable outputs). Mirrors the consumer e2e-daily.yml. Each run cleans up its job.
on:
schedule:
# Daily at 00:07 UTC = ~8pm US Eastern (8:07pm EDT / 7:07pm EST). Cron is fixed
# UTC so the local time shifts 1h across DST. Off the :00 mark to avoid the
# top-of-hour scheduler herd.
- cron: '7 0 * * *'
workflow_dispatch: {}
permissions:
contents: read
id-token: write # for Workload Identity Federation
jobs:
e2e-tenant:
name: "Tenant E2E: ${{ matrix.tenant }}"
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
# Run tenants sequentially: the GCE encoder is a single shared worker, and two
# concurrent full renders overwhelm it ("lost contact with worker"). One at a time.
max-parallel: 1
matrix:
tenant: [vocalstar, singa]
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Authenticate to GCP
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps chromium
- name: Fetch test audio (private GCS)
run: bash scripts/e2e/fetch_test_audio.sh frontend/e2e/fixtures
- name: Ensure testmail.app domain is allowlisted (rate limit)
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run tenant E2E (frontend journey)
id: run_test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
TENANT_ID: ${{ matrix.tenant }}
TENANT_PORTAL_URL: https://${{ matrix.tenant }}.nomadkaraoke.com
run: |
echo "=== Tenant E2E (frontend): ${{ matrix.tenant }} ==="
npx playwright test tenant-happy-path.spec.ts \
--config=playwright.production.config.ts --project=chromium \
--reporter=list --timeout=2400000 2>&1 | tee e2e-output.log
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
continue-on-error: true
- name: Upload logs
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-tenant-${{ matrix.tenant }}-${{ github.run_number }}
path: |
frontend/e2e-output.log
frontend/test-results/
frontend/playwright-report/
retention-days: 14
- name: Discord notification (failure)
if: ${{ steps.run_test.outputs.exit_code != '0' }}
env:
DISCORD_WEBHOOK: ${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}
run: |
if [ -n "$DISCORD_WEBHOOK" ]; then
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"❌ Tenant E2E FAILED — ${{ matrix.tenant }}\", \"description\": \"Run: #${{ github.run_number }}\", \"color\": 15158332}]}" \
"$DISCORD_WEBHOOK" || true
fi
- name: Fail the job if the test failed
if: ${{ steps.run_test.outputs.exit_code != '0' }}
run: exit 1
notify-email:
name: "Email summary"
runs-on: ubuntu-latest
needs: e2e-tenant
if: ${{ always() && github.event_name == 'schedule' }}
steps:
- name: Send email notification
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.postmarkapp.com
server_port: 587
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
subject: "Tenant E2E Daily ${{ needs.e2e-tenant.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
Tenant E2E Daily Test (Vocal Star + Singa)
Result: ${{ needs.e2e-tenant.result }}
Run: #${{ github.run_number }}
Details: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
-10
View File
@@ -1,10 +0,0 @@
name: Security
on:
pull_request:
push:
branches: [main]
jobs:
gitleaks:
uses: nomadkaraoke/.github/.github/workflows/gitleaks.yml@main
+2 -6
View File
@@ -3,7 +3,6 @@
.vscode
*.csv
*.mp3
ABBA*
# Andrew functional adds
/audio/
@@ -13,9 +12,6 @@ tests/outputs
/input
/output/
# Local karaoke-gen CLI test output dirs (created during manual validation)
/test-output-*/
# Cloud Infra
.emulator-logs
@@ -62,7 +58,8 @@ frontend/.eslintcache
# Storybook
frontend/storybook-static/
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
# PWA files
frontend/**/public/sw.js
frontend/**/public/workbox-*.js
frontend/**/public/swe-worker-*.js
@@ -235,4 +232,3 @@ flacfetch
# Test output directories
Rick Astley*/
.superpowers/
-60
View File
@@ -1,60 +0,0 @@
# Nomad Karaoke shared gitleaks config.
# Extends the default ruleset with one custom rule for Discord webhooks
# and an allowlist covering known false-positive paths.
title = "Nomad Karaoke"
[extend]
useDefault = true
# ---------------------------------------------------------------------------
# Custom rules
# ---------------------------------------------------------------------------
[[rules]]
id = "discord-webhook-url"
description = "Discord webhook URL (token embedded in URL)"
regex = '''https://(?:discord|discordapp)\.com/api/webhooks/\d{17,20}/[A-Za-z0-9_-]{60,}'''
tags = ["discord", "webhook", "secret"]
# ---------------------------------------------------------------------------
# Global allowlist
# ---------------------------------------------------------------------------
[allowlist]
description = "Paths and patterns that are known false positives"
paths = [
'''(?i)\.(jpg|jpeg|png|gif|bmp|svg|webp|pdf|ico|mp3|mp4|wav|flac|m4a)$''',
'''(^|/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|poetry\.lock|Podfile\.lock|Cargo\.lock|go\.sum|composer\.lock|Gemfile\.lock|uv\.lock)$''',
'''(^|/)node_modules/''',
'''(^|/)vendor/''',
'''(^|/)wp-content/''',
'''(^|/)\.dart_tool/''',
'''(^|/)\.venv/''',
'''(^|/)\.next/''',
'''(^|/)build/''',
'''(^|/)dist/''',
'''(^|/)\.flutter-plugins''',
'''(^|/)test[-_]?fixtures?/''',
'''(^|/)tests?/.*conftest\.py$''',
'''(^|/)tests?/.*test_[^/]+\.py$''',
'''(^|/)tests?/.*_test\.py$''',
'''(^|/).*\.(test|spec)\.(js|mjs|cjs|ts|tsx|jsx)$''',
'''(^|/).*\.test\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
'''(^|/).*\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
]
regexes = [
# Explicit placeholder patterns
'''YOUR_[A-Z_]{2,}''',
'''EXAMPLE_[A-Z_]{2,}''',
'''PLACEHOLDER_[A-Z_]{2,}''',
'''CHANGE[_-]?ME''',
'''<[A-Z_]{2,}>''',
'''\$\{[A-Z_]{2,}\}''',
# Firebase / GCP client API keys — public by design (identify the project,
# access is gated by Firebase Security Rules).
'''AIzaSy[A-Za-z0-9_-]{33}''',
]
-1
View File
@@ -1 +0,0 @@
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
-1
View File
@@ -1 +0,0 @@
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+14 -86
View File
@@ -16,19 +16,14 @@
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
## Tech Stack
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
@@ -39,83 +34,23 @@
- Create PR with summary, changes, testing info
- Merge only after CI passes
### Testing (Required)
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
- Which test types are needed (unit, integration, E2E)
- Where tests should be placed
- What mocking approach to use
- Coverage expectations
- **Production E2E tests for user-facing features** - see below
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
**Before committing**, run all tests:
### Testing (Required Before Commit)
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
### Testing in Production (for Agents)
When asked to "test it yourself in prod" or verify a fix in production:
```bash
# Get admin token
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Option 1: Use the debug template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Option 2: Copy template for customization (gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script, then run it
node test-my-issue.local.mjs
# Option 3: Run existing production E2E tests
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
```
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
### Infrastructure
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
- **GCS ops on macOS: never use `gsutil -m`** — its fork-based multiprocessing segfaults
on macOS (crash-report popups + the parent process hangs forever). Use `gcloud storage`
instead (`gcloud storage cp/rm/rsync …`) — no fork, and faster. Plain `gsutil` (no `-m`)
is safe but slower. `~/.boto` pins `parallel_process_count=1` as a safety net; don't remove it.
### Internationalization (i18n)
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
- Components use `useTranslations('namespace')` from next-intl
- Pages are under `frontend/app/[locale]/` — locale-aware routing (customer-facing: `/`, `/app/*`, `/order/*`, `/auth/*`, `/payment/*`, `/r/*`)
- Non-locale routes at those same paths (e.g. `frontend/app/app/jobs/[[...slug]]/page.tsx`) are `<LocaleRedirect />` shims that redirect into `[locale]` — they are NOT English-only surfaces
- The `/app/jobs/#/{id}/review`, `/instrumental`, and `/audio-edit` review UIs are fully multilingual (components under `components/lyrics-review/`, `components/instrumental-review/`, `components/audio-editor/` use `useTranslations` heavily)
- Only `/admin/*` is English-only — it has no `[locale]/admin` counterpart and inherits `DefaultIntlProvider` from the root layout
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
## API Authentication (for Agents)
@@ -149,22 +84,15 @@ EOF
```
### GCE Encoding Worker
The worker is a c4d blue-green primary pair (`encoding-worker-a`/`-b`, us-central1-c)
plus 8 stopped fallback VMs across 6 machine families for stockout resilience
(`encoding-worker-fallback-{a,b,n2c,n2f,c4a,n4db,c2df,n2da}`). The currently-serving
VM is `active_override_vm` in `config/encoding-worker` if set, else `primary_vm`.
Use that VM name + its zone below (there is no VM literally named `encoding-worker`):
```bash
# Check health (includes wheel_version). Example uses the primary pair's VM.
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart the service (pick up new worker-side code after a deploy)
gcloud compute ssh encoding-worker-a --zone=us-central1-c --project=nomadkaraoke \
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
+15 -77
View File
@@ -1,77 +1,34 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop eval-custom-lyrics
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend unit tests (Jest)"
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make test - Run all tests (unit + backend + emulator-based)"
@echo " make test-unit - Run unit tests only (karaoke_gen)"
@echo " make test-backend - Run backend unit tests only"
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
@echo " make test-all - Alias for 'make test'"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@echo ""
@echo "Before committing, run: make test"
# Install dependencies (only if needed)
install-backend:
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
echo "=== Installing backend dependencies ==="; \
poetry install; \
fi
install-frontend:
@if [ ! -d "frontend/node_modules" ]; then \
echo "=== Installing frontend dependencies ==="; \
cd frontend && npm install; \
fi
install: install-backend install-frontend
# Build frontend and copy to Python package
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
# Run unit tests for karaoke_gen package
test-unit: install-backend
test-unit:
@echo "=== Running karaoke_gen unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
# Run backend unit tests
test-backend:
@echo "=== Running backend unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
test-e2e:
@echo "=== Running E2E integration tests with emulators ==="
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
@@ -80,28 +37,13 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
# Run all backend tests (unit + emulator)
test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend unit tests (Jest only - E2E requires manual setup)
test-frontend: install-frontend
@echo "=== Running frontend unit tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
# Run frontend E2E tests (requires dev server - run manually, not in make test)
test-frontend-e2e: install-frontend
@echo "=== Running frontend E2E tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
# Run all tests (use this before committing!)
test: test-unit test-backend test-e2e
@echo ""
@echo "✅ All tests passed! Ready to commit."
@@ -133,7 +75,3 @@ clean:
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
@echo "Cleaned up temporary files"
.PHONY: eval-custom-lyrics
eval-custom-lyrics:
cd backend && poetry run python -m backend.eval.custom_lyrics.run
+20 -5
View File
@@ -159,9 +159,6 @@ karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"
# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
@@ -406,8 +403,26 @@ gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
### Deploy Cloud Run
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
See `.github/workflows/ci.yml` for the full deployment workflow.
```bash
# Build and deploy
gcloud builds submit --config=cloudbuild.yaml
# Get outputs from Pulumi
SA_EMAIL=$(pulumi stack output service_account_email)
BUCKET_NAME=$(pulumi stack output bucket_name)
# Deploy Cloud Run service
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--service-account $SA_EMAIL \
--memory 2Gi \
--cpu 2 \
--timeout 600 \
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
```
### Point CLI to Your Backend
-2
View File
@@ -29,8 +29,6 @@ tmp/
# Google Cloud
.gcloud/
*.json
# Allow i18n translation files
!translations/*.json
# Testing
.pytest_cache/
+4 -11
View File
@@ -22,19 +22,12 @@ WORKDIR /app
COPY backend /app/backend
# Copy and reinstall karaoke_gen from local source
# This ensures new modules are included without rebuilding base image
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
# This ensures new modules (like style_loader.py) are included without rebuilding base image
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY pyproject.toml README.md LICENSE /app/
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
# CPU images: normal install resolves all deps from pyproject.toml.
RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir --no-deps -e .; \
else \
pip install --no-cache-dir -e .; \
fi
RUN pip install --no-cache-dir -e .
# Set environment variables
ENV PYTHONUNBUFFERED=1
+7 -27
View File
@@ -1,15 +1,8 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock or this Dockerfile changes
#
# This image is rebuilt only when poetry.lock changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. pyproject.toml + package stubs (changes with dep updates)
# 3. pip install (the slow part - cached unless pyproject.toml changes)
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
# 5. spaCy model download (cached unless pip install layer changes)
FROM python:3.11-slim
@@ -23,7 +16,6 @@ RUN apt-get update && apt-get install -y \
build-essential \
curl \
xz-utils \
chromium \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
@@ -58,26 +50,14 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
# Copy dependency files and source code needed for pip install
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds, so partial
# dependency updates only download changed packages instead of everything
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
-136
View File
@@ -1,136 +0,0 @@
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
+114 -14
View File
@@ -113,19 +113,6 @@ async def require_auth(
detail=f"Authentication failed: {auth_result.message}"
)
# If token has a tenant_id, override the hostname-based tenant detection.
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
# We must also load tenant_config so downstream handlers (e.g. job creation)
# that rely on tenant_config for setting tenant_id on jobs work correctly.
if auth_result.tenant_id:
request.state.tenant_id = auth_result.tenant_id
if not getattr(request.state, "tenant_config", None):
from backend.services.tenant_service import get_tenant_service
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
if tenant_config and tenant_config.is_active:
request.state.tenant_config = tenant_config
# Log successful auth with request_id for correlation with request audit
logger.info(
"auth_success",
@@ -135,7 +122,6 @@ async def require_auth(
"user_type": auth_result.user_type.value if auth_result.user_type else None,
"is_admin": auth_result.is_admin,
"remaining_uses": auth_result.remaining_uses,
"tenant_id": auth_result.tenant_id,
"audit_type": "auth_event",
}
)
@@ -355,3 +341,117 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
require_review_auth = require_review_auth_factory()
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create an instrumental review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token) - also validates job ownership
2. Job-specific instrumental token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates instrumental review access
"""
async def require_instrumental_auth(
request: Request,
job_id: str = Path(...),
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate instrumental review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
Raises:
HTTPException: 401 if authentication fails
HTTPException: 403 if user doesn't own the job (for full auth)
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
job_manager = JobManager()
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
auth_result = auth_service.validate_token_full(full_token)
if auth_result.is_valid:
# For full auth, also verify job ownership
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Check ownership: admin can access all, users only their own jobs
if not auth_result.is_admin:
if auth_result.user_email and job.user_email:
if auth_result.user_email.lower() != job.user_email.lower():
logger.warning(
f"Instrumental access denied: user {auth_result.user_email} "
f"tried to access job {job_id} owned by {job.user_email}"
)
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
elif job.user_email:
# Token auth without email trying to access a job with owner
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
return job_id, "full"
# Try instrumental token
if instrumental_token:
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate instrumental token matches
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
# Check expiry if set
if job.instrumental_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.instrumental_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Instrumental token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Instrumental review token has expired. Please request a new review link."
)
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
return job_id, "instrumental_token"
else:
logger.warning(f"Invalid instrumental token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
)
return require_instrumental_auth
# Default instance for most instrumental review endpoints
require_instrumental_auth = require_instrumental_auth_factory()
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+35 -41
View File
@@ -9,7 +9,7 @@ Provides endpoints for:
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel
from backend.services.credential_manager import (
@@ -18,7 +18,6 @@ from backend.services.credential_manager import (
CredentialCheckResult,
)
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@@ -126,16 +125,15 @@ async def get_credentials_status():
@router.get("/status/{service}", response_model=CredentialStatusResponse)
async def get_service_credential_status(service: str, request: Request):
async def get_service_credential_status(service: str):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
locale = get_locale_from_request(request)
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
@@ -143,7 +141,7 @@ async def get_service_credential_status(service: str, request: Request):
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
return CredentialStatusResponse(
service=result.service,
@@ -155,43 +153,42 @@ async def get_service_credential_status(service: str, request: Request):
@router.post("/validate", response_model=CredentialValidationResponse)
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
async def validate_credentials(request: CredentialValidationRequest):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
locale = get_locale_from_request(http_request)
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message=t(locale, "auth.validCredentials")
message="All requested credentials are valid"
)
@@ -200,41 +197,40 @@ async def validate_credentials(request: CredentialValidationRequest, http_reques
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -263,36 +259,35 @@ async def poll_youtube_device_auth(device_code: str):
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -316,20 +311,19 @@ async def poll_gdrive_device_auth(device_code: str):
# ===========================================================================
@router.post("/test-alert")
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
async def test_credential_alert(background_tasks: BackgroundTasks):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
locale = get_locale_from_request(request)
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail=t(locale, "auth.discordAlertNotConfigured")
detail="Discord alert webhook not configured"
)
manager = get_credential_manager()
-476
View File
@@ -1,476 +0,0 @@
"""Bulk Mode API routes.
Lets a user submit up to 100 karaoke jobs at once, by text (artist/title rows)
or by album (MusicBrainz release lookup). Album tracklists and text rows are
enriched with KaraokeNerds community-version availability so the user can skip
tracks that already exist.
Submission (POST /bulk/submit) and progress (GET /bulk/{batch_id}) live here too
(added in later phases).
"""
import logging
import uuid
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/bulk", tags=["bulk"])
MAX_BULK_SONGS = 100
# --- Models -----------------------------------------------------------------
class BulkArtist(BaseModel):
mbid: Optional[str] = None
name: str
disambiguation: str = ""
type: Optional[str] = None
country: Optional[str] = None
class BulkAlbum(BaseModel):
release_group_mbid: str
title: str
primary_type: Optional[str] = None
secondary_types: list[str] = []
first_release_date: str = ""
is_studio: bool = True
class BulkEdition(BaseModel):
release_mbid: str
title: Optional[str] = None
status: Optional[str] = None
date: str = ""
country: Optional[str] = None
track_count: int = 0
class BulkEditionVariant(BaseModel):
"""A distinct tracklist variant (country pressings collapsed by track count)."""
representative_release_mbid: str
label: Literal["Original", "Reissue"] = "Original"
track_count: int = 0
year: str = ""
pressing_count: int = 0
delta_vs_original: int = 0
class CommunityVersion(BaseModel):
"""An existing community karaoke version the user can preview on YouTube."""
brand: str
url: str
class BulkTrack(BaseModel):
position: Optional[int] = None
title: str
recording_mbid: Optional[str] = None
length_ms: Optional[int] = None
is_extra: bool = False
extra_reason: str = ""
# Availability (filled from KaraokeNerds); available=True => already exists
available: bool = False
brands: list[str] = []
versions: list[CommunityVersion] = []
class BulkTracklistResponse(BaseModel):
release_mbid: Optional[str] = None
canonical_release_mbid: Optional[str] = None
selected_variant_mbid: Optional[str] = None
title: Optional[str] = None
date: str = ""
tracks: list[BulkTrack] = []
editions: list[BulkEdition] = []
variants: list[BulkEditionVariant] = []
class AvailabilityItem(BaseModel):
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class AvailabilityRequest(BaseModel):
tracks: list[AvailabilityItem]
class AvailabilityResult(BaseModel):
artist: str
title: str
available: bool
brands: list[str] = []
brand_count: int = 0
versions: list[CommunityVersion] = []
class AvailabilityResponse(BaseModel):
results: list[AvailabilityResult]
# --- Album lookup routes ----------------------------------------------------
@router.get("/album/artists", response_model=list[BulkArtist])
async def album_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search"),
limit: int = Query(8, ge=1, le=25),
auth_result: AuthResult = Depends(require_auth),
):
"""Search MusicBrainz for artists (album-mode step 1)."""
from backend.services.musicbrainz_service import get_musicbrainz_service
try:
results = await get_musicbrainz_service().search_artists(q, limit=limit)
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz artist search failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Artist lookup is temporarily unavailable")
return results
@router.get("/album/albums", response_model=list[BulkAlbum])
async def album_albums(
request: Request,
artist_mbid: str = Query(..., min_length=1),
auth_result: AuthResult = Depends(require_auth),
):
"""List an artist's albums (release-groups) for the album picker."""
from backend.services.musicbrainz_service import get_musicbrainz_service
try:
return await get_musicbrainz_service().get_albums(artist_mbid)
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz album list failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Album lookup is temporarily unavailable")
@router.get("/album/tracklist", response_model=BulkTracklistResponse)
async def album_tracklist(
request: Request,
artist: str = Query(..., min_length=1, description="Artist name for availability checks"),
release_group_mbid: Optional[str] = Query(None),
release_mbid: Optional[str] = Query(None, description="Specific edition to load"),
locale: Optional[str] = Query(None, description="UI locale → biases representative-pressing choice"),
auth_result: AuthResult = Depends(require_auth),
):
"""Resolve an album's canonical tracklist (or a specific edition) and enrich
each track with KaraokeNerds community-version availability.
Response shape note: ``variants`` is only populated on the ``release_group_mbid``
path (the initial album load). When loading a specific edition via
``release_mbid`` it is intentionally ``[]`` — the client already holds the
variant list from the initial load and only needs the chosen edition's tracks.
"""
from backend.services.musicbrainz_service import (
get_musicbrainz_service,
country_prefs_for_locale,
)
from backend.services.karaokenerds_service import check_community_versions_batch
if not release_group_mbid and not release_mbid:
raise HTTPException(status_code=422, detail="release_group_mbid or release_mbid required")
mb = get_musicbrainz_service()
try:
if release_mbid:
# Loading a specific variant — variants already known to the client.
tracklist = await mb.get_release_tracklist(release_mbid)
tracklist.setdefault("editions", [])
tracklist.setdefault("canonical_release_mbid", release_mbid)
tracklist.setdefault("variants", [])
tracklist.setdefault("selected_variant_mbid", release_mbid)
else:
tracklist = await mb.get_album_tracklist(
release_group_mbid, preferred_countries=country_prefs_for_locale(locale)
)
except Exception as e: # noqa: BLE001
logger.warning("MusicBrainz tracklist failed: %r", e, exc_info=True)
raise HTTPException(status_code=502, detail="Tracklist lookup is temporarily unavailable")
tracks = tracklist.get("tracks", [])
# Availability enrichment — never fatal.
try:
avail = await check_community_versions_batch(
[{"artist": artist, "title": t["title"]} for t in tracks]
)
for t, a in zip(tracks, avail):
t["available"] = a["available"]
t["brands"] = a["brands"]
t["versions"] = a.get("versions", [])
except Exception as e: # noqa: BLE001
logger.warning("Availability enrichment failed (non-fatal): %s", e)
return BulkTracklistResponse(
release_mbid=tracklist.get("release_mbid"),
canonical_release_mbid=tracklist.get("canonical_release_mbid"),
selected_variant_mbid=tracklist.get("selected_variant_mbid"),
title=tracklist.get("title"),
date=tracklist.get("date", ""),
tracks=[BulkTrack(**t) for t in tracks],
editions=[BulkEdition(**e) for e in tracklist.get("editions", [])],
variants=[BulkEditionVariant(**v) for v in tracklist.get("variants", [])],
)
@router.post("/availability", response_model=AvailabilityResponse)
async def bulk_availability(
request: Request,
body: AvailabilityRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Batch KaraokeNerds availability check for text-mode rows."""
from backend.services.karaokenerds_service import check_community_versions_batch
if len(body.tracks) > MAX_BULK_SONGS:
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} tracks per request")
results = await check_community_versions_batch(
[{"artist": t.artist, "title": t.title} for t in body.tracks]
)
return AvailabilityResponse(results=[AvailabilityResult(**r) for r in results])
# --- Submission -------------------------------------------------------------
class BulkSong(BaseModel):
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
display_artist: Optional[str] = None
display_title: Optional[str] = None
class BulkSettings(BaseModel):
auto_select_if_lossless: bool = True
is_private: bool = False
skip_audio_edit: bool = True
skip_customization: bool = True
class BulkSubmitRequest(BaseModel):
songs: list[BulkSong]
settings: BulkSettings = Field(default_factory=BulkSettings)
class BulkSubmitResponse(BaseModel):
batch_id: str
job_ids: list[str]
total: int
@router.post("/submit", response_model=BulkSubmitResponse)
async def bulk_submit(
request: Request,
body: BulkSubmitRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""Create one job per song, gate on credits upfront, and dispatch the batch
search worker. Jobs start parked (auto_download=False) and the worker either
auto-selects a confident lossless match or leaves them awaiting selection."""
from backend.config import get_settings
from backend.exceptions import InsufficientCreditsError
from backend.models.job import JobCreate
from backend.services.job_manager import JobManager
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
from backend.services.theme_service import get_theme_service
from backend.services.worker_service import get_worker_service
n = len(body.songs)
if n == 0:
raise HTTPException(status_code=422, detail="At least one song is required")
if n > MAX_BULK_SONGS:
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} songs per batch")
user_email = auth_result.user_email
is_admin = bool(auth_result.is_admin)
# Upfront credit gate (1 credit/song minimum). Admins bypass.
if not is_admin and user_email:
from backend.services.user_service import get_user_service
available = get_user_service().check_credits(user_email)
if available < n:
raise HTTPException(
status_code=402,
detail={
"error": "insufficient_credits",
"message": "You don't have enough credits for this batch.",
"credits_available": available,
"credits_required": n,
},
)
settings = get_settings()
theme_service = get_theme_service()
theme_id = theme_service.get_default_theme_id()
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(theme_id, None, None)
batch_id = uuid.uuid4().hex[:12]
bulk_settings = body.settings.model_dump()
job_manager = JobManager()
job_ids: list[str] = []
for song in body.songs:
display_artist = (song.display_artist or song.artist).strip()
display_title = (song.display_title or song.title).strip()
request_metadata = {
"created_from": "bulk",
"batch_id": batch_id,
"user_email": user_email,
}
job_create = JobCreate(
artist=display_artist,
title=display_title,
theme_id=theme_id,
enable_cdg=resolved_cdg,
enable_txt=resolved_txt,
user_email=user_email,
audio_search_artist=song.artist.strip(),
audio_search_title=song.title.strip(),
auto_download=False,
is_private=body.settings.is_private,
brand_prefix=settings.default_brand_prefix,
enable_youtube_upload=settings.default_enable_youtube_upload,
youtube_description=settings.default_youtube_description,
discord_webhook_url=settings.default_discord_webhook_url,
dropbox_path=settings.default_dropbox_path,
gdrive_folder_id=settings.default_gdrive_folder_id,
request_metadata=request_metadata,
)
try:
job = job_manager.create_job(job_create, is_admin=is_admin)
except InsufficientCreditsError:
# Ran out mid-batch (e.g. concurrent spend after the gate). Stop here;
# already-created jobs are valid and will proceed.
logger.warning("[batch:%s] ran out of credits after %d/%d jobs", batch_id, len(job_ids), n)
break
job_manager.update_job(
job.job_id,
{
"state_data.batch_id": batch_id,
"state_data.bulk_settings": bulk_settings,
"state_data.bulk_pending_search": True,
},
)
job_ids.append(job.job_id)
if not job_ids:
raise HTTPException(status_code=402, detail="No jobs could be created")
# Dispatch the batch search worker. Retry once on transient failure. If it
# still fails the jobs exist (and are charged) but won't process, so surface
# an explicit error with the batch_id — and tell the user NOT to resubmit
# (that would create+charge a second batch); the batch can be re-dispatched.
worker_service = get_worker_service()
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
if not triggered:
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
if not triggered:
logger.critical(
"[batch:%s] created %d jobs but FAILED to dispatch bulk-search worker",
batch_id, len(job_ids),
)
raise HTTPException(
status_code=503,
detail={
"error": "dispatch_failed",
"message": "Your batch was created but processing couldn't start. "
"Please don't resubmit — contact support with this batch ID and we'll start it.",
"batch_id": batch_id,
"job_ids": job_ids,
},
)
logger.info("[batch:%s] created %d jobs, dispatched bulk-search worker", batch_id, len(job_ids))
return BulkSubmitResponse(batch_id=batch_id, job_ids=job_ids, total=len(job_ids))
# --- Progress ---------------------------------------------------------------
class BulkJobStatus(BaseModel):
job_id: str
artist: str
title: str
status: str
auto_selected: bool = False
class BulkBatchResponse(BaseModel):
batch_id: str
total: int
counts: dict
jobs: list[BulkJobStatus]
# Status -> progress bucket for the batch summary.
_AWAITING = "awaiting_audio_selection"
_TERMINAL_OK = "complete"
_TERMINAL_FAIL = "failed"
_SEARCHING = {"pending", "searching_audio"}
def _bucket(status: str, pending_search: bool) -> str:
if pending_search or status in _SEARCHING:
return "searching"
if status == _AWAITING:
return "awaiting_selection"
if status == _TERMINAL_OK:
return "complete"
if status == _TERMINAL_FAIL:
return "failed"
return "processing"
@router.get("/{batch_id}", response_model=BulkBatchResponse)
async def bulk_batch(
request: Request,
batch_id: str,
auth_result: AuthResult = Depends(require_auth),
):
"""Progress summary for a batch: counts by bucket + per-job status."""
from backend.services.job_manager import JobManager
job_manager = JobManager()
jobs = job_manager.list_jobs_by_batch_id(batch_id)
if not bool(auth_result.is_admin):
email = (auth_result.user_email or "").lower()
jobs = [j for j in jobs if (j.user_email or "").lower() == email]
if not jobs:
raise HTTPException(status_code=404, detail="Batch not found")
counts = {"searching": 0, "awaiting_selection": 0, "processing": 0, "complete": 0, "failed": 0}
job_statuses: list[BulkJobStatus] = []
for j in jobs:
status = j.status.value if hasattr(j.status, "value") else str(j.status)
sd = j.state_data or {}
bucket = _bucket(status, bool(sd.get("bulk_pending_search")))
counts[bucket] += 1
job_statuses.append(
BulkJobStatus(
job_id=j.job_id,
artist=j.audio_search_artist or j.artist or "",
title=j.audio_search_title or j.title or "",
status=status,
auto_selected=bool(sd.get("bulk_auto_selected")),
)
)
return BulkBatchResponse(
batch_id=batch_id, total=len(jobs), counts=counts, jobs=job_statuses
)
-223
View File
@@ -1,223 +0,0 @@
"""
Catalog API routes for song/artist lookup and community version detection.
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
and checks karaokenerds.com for existing community karaoke versions.
"""
import logging
import time
from typing import Literal, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/catalog", tags=["catalog"])
# Simple per-user rate limiting: 20 requests/minute
# { user_email: [timestamp, timestamp, ...] }
_rate_limit_windows: dict[str, list[float]] = {}
RATE_LIMIT_MAX_REQUESTS = 20
RATE_LIMIT_WINDOW_SECONDS = 60
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
"""Check per-user rate limit. Raises 429 if exceeded."""
now = time.monotonic()
window_start = now - RATE_LIMIT_WINDOW_SECONDS
# Get or create window for this user
if user_email not in _rate_limit_windows:
_rate_limit_windows[user_email] = []
# Remove expired entries
timestamps = _rate_limit_windows[user_email]
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
raise HTTPException(
status_code=429,
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
)
_rate_limit_windows[user_email].append(now)
# --- Response Models ---
class ArtistResult(BaseModel):
"""Artist search result from MusicBrainz/Spotify catalog."""
name: str
mbid: Optional[str] = None
disambiguation: Optional[str] = None
artist_type: Optional[str] = None
spotify_id: Optional[str] = None
popularity: Optional[int] = None
genres: Optional[list[str]] = None
tags: Optional[list[str]] = None
class TrackResult(BaseModel):
"""Track search result from Spotify catalog."""
track_name: str
artist_name: str
track_id: Optional[str] = None
artist_id: Optional[str] = None
popularity: Optional[int] = None
duration_ms: Optional[int] = None
explicit: Optional[bool] = None
class CommunityTrack(BaseModel):
"""A community-approved karaoke track from karaokenerds."""
brand_name: str
brand_code: str
youtube_url: str
is_community: bool = True
class CommunityCheckSong(BaseModel):
"""A song with community karaoke tracks."""
title: str
artist: str
community_tracks: list[CommunityTrack]
class CommunityCheckRequest(BaseModel):
"""Request body for community version check."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class CommunityCheckResponse(BaseModel):
"""Response for community version check."""
has_community: bool
songs: list[CommunityCheckSong]
best_youtube_url: Optional[str] = None
class MatchJudgeRequest(BaseModel):
"""Request body for the artist/title match judge."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
# The audio search's confidence tier (1=strong..3=weak), so the judge can
# tell whether weak results hint at a typo. Optional.
audio_confidence_tier: Optional[int] = Field(None, ge=1, le=3)
# "fast" = catalog-only pass (no AI, no tier needed) the client fires in
# parallel with the audio search; "full" = complete pipeline once the tier is
# known. Defaults to "full" so a single legacy call still works.
stage: Literal["fast", "full"] = "full"
class MatchJudgeAlternative(BaseModel):
artist: str
title: str
class MatchJudgeResponse(BaseModel):
"""Verdict on a typed artist/title — formatting + match acceptance."""
kind: str # "cosmetic" | "content" | "ambiguous" | "none"
confident: bool
canonical_artist: str
canonical_title: str
alternatives: list[MatchJudgeAlternative] = []
engine: str # "deterministic" | "catalog" | "ai"
reason: str = ""
# Set by a "fast" pass that couldn't decide — the client should follow up with
# a "full" call once the audio tier is known.
needs_ai: bool = False
# --- Routes ---
@router.get("/artists", response_model=list[ArtistResult])
async def search_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search query"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_artists
results = await search_artists(q, limit)
return results
@router.get("/tracks", response_model=list[TrackResult])
async def search_tracks(
request: Request,
q: str = Query(..., min_length=2, description="Track name search query"),
artist: Optional[str] = Query(None, description="Filter by artist name"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for tracks by name, optionally filtered by artist.
Returns canonical Spotify track data for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_tracks
results = await search_tracks(q, artist, limit)
return results
@router.post("/community-check", response_model=CommunityCheckResponse)
async def check_community_versions(
request: Request,
body: CommunityCheckRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Check if a song already has community-approved karaoke versions on YouTube
via karaokenerds.com. Non-blocking — just informs the user.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.karaokenerds_service import check_community_versions
result = await check_community_versions(body.artist, body.title)
return result
@router.post("/match-judge", response_model=MatchJudgeResponse)
async def match_judge(
request: Request,
body: MatchJudgeRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Judge a typed artist/title: decide official formatting + whether it matches a
real song. Deterministic + catalog layers run first; a light AI model fires
only when those aren't confident. Never blocks the submission flow — failures
degrade to a "none" verdict.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.match_judge.service import judge_match
verdict = await judge_match(
body.artist, body.title,
audio_tier=body.audio_confidence_tier,
stage=body.stage,
)
logger.info(
"match-judge stage=%s kind=%s engine=%s confident=%s needs_ai=%s",
body.stage, verdict.kind, verdict.engine, verdict.confident, verdict.needs_ai,
)
return verdict.to_dict()
-108
View File
@@ -1,108 +0,0 @@
"""POST /api/client-errors — ingest frontend crash reports.
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
via the existing error-monitor pipeline — see
``backend/services/error_monitor``.
"""
from __future__ import annotations
import logging
import time
from typing import Optional
from fastapi import APIRouter, Request, HTTPException
from pydantic import BaseModel, Field, field_validator
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
from backend.services.error_monitor.frontend_ingestion import (
FrontendErrorReport,
RateLimiter,
build_pattern_data,
is_bot_user_agent,
sanitize_url,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
_limiter = RateLimiter(max_per_minute=60)
_adapter_singleton: ErrorPatternsAdapter | None = None
def _get_adapter() -> ErrorPatternsAdapter:
global _adapter_singleton
if _adapter_singleton is None:
_adapter_singleton = ErrorPatternsAdapter()
return _adapter_singleton
class ClientErrorPayload(BaseModel):
message: str = Field(..., min_length=1, max_length=4000)
stack: Optional[str] = Field(None, max_length=1_000_000)
url: str = Field("", max_length=2048)
user_agent: str = Field("", max_length=1024)
release: str = Field("", max_length=64)
user_email: Optional[str] = Field(None, max_length=320)
viewport: Optional[dict] = None
locale: str = Field("en", max_length=10)
source: str = Field("unknown", max_length=64)
# optional free-form extra for debugging. Capped server-side.
extra: Optional[dict] = None
@field_validator("message", "url", "user_agent", "locale", "source")
@classmethod
def strip(cls, v: str) -> str:
return (v or "").strip()
class ClientErrorResponse(BaseModel):
pattern_id: str
is_new: bool
@router.post("", response_model=ClientErrorResponse, status_code=202)
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
client_ip = request.client.host if request.client else "unknown"
if not _limiter.allow(client_ip, time.monotonic()):
raise HTTPException(status_code=429, detail="too many reports")
# Crawler/bot crashes are pure alert noise (bots run headless Chrome and trip
# DOM errors no real user hits). Accept the report so the client doesn't retry,
# but don't persist/alert on it.
if is_bot_user_agent(payload.user_agent):
# Log the bot UA (not sensitive) + sanitized URL (query/token stripped)
# for visibility into what's being dropped, without persisting/alerting.
logger.info(
"frontend_crash_ignored_bot ua=%r url=%s source=%s",
payload.user_agent[:120],
sanitize_url(payload.url),
payload.source,
)
return ClientErrorResponse(pattern_id="bot-ignored", is_new=False)
report = FrontendErrorReport(
message=payload.message,
stack=payload.stack,
url=payload.url,
user_agent=payload.user_agent,
release=payload.release,
user_email=payload.user_email,
viewport=payload.viewport,
locale=payload.locale,
extra=payload.extra,
)
pattern_data = build_pattern_data(report)
try:
result = _get_adapter().upsert_pattern(pattern_data)
except Exception: # pragma: no cover - Firestore transient errors
logger.exception("Failed to upsert frontend error pattern")
raise HTTPException(status_code=503, detail="storage unavailable")
logger.info(
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
result.pattern_id,
result.is_new,
payload.source,
)
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
-96
View File
@@ -1,96 +0,0 @@
"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the lyrics-review frontend to ensure the
primary VM is running before the reviewer previews a render.
Auth: these are gated on ``require_review_auth`` (review access to a specific
job), NOT ``require_admin``. The callers are ordinary customers on the
lyrics-review page — gating on admin makes every non-admin caller 403, which
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
abuse posture identical to the rest of the review surface: you can only warm
the shared encoding VM if you already have review access to a real job.
"""
import logging
from typing import Tuple
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_review_auth
from backend.services.encoding_worker_manager import EncodingWorkerManager
from backend.services.encoding_errors import EncodingWorkerStartError
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup/{job_id}")
async def warmup_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped.
Called on lyrics-review page load so the VM is warm by the time the
reviewer previews a render. ``job_id`` scopes auth to the job under review.
"""
try:
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except EncodingWorkerStartError as e:
# The primary VM couldn't start (capacity exhaustion or a transient
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
# the encoding flow's own warmup (ensure_any_running) falls back to
# the alternate-zone workers, so the job still completes. Log at
# WARNING so it doesn't trip the production error monitor and page us
# for self-healing capacity events.
logger.warning(
f"Primary encoding worker warmup failed ({e}); "
f"encoding will fall back to an alternate-zone worker"
)
return {"started": False, "error": str(e)}
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat/{job_id}")
async def heartbeat_encoding_worker(
job_id: str,
_auth: Tuple[str, str] = Depends(require_review_auth),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown.
Called periodically while the reviewer works. ``job_id`` scopes auth to the
job under review.
"""
try:
manager.update_activity()
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}
File diff suppressed because it is too large. Load diff
+3 -370
View File
@@ -5,22 +5,14 @@ import os
import logging
import subprocess
import platform
from fastapi import APIRouter, Depends
from typing import Dict, Any, Optional, Tuple
from fastapi import APIRouter
from typing import Dict, Any, Optional
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
from backend.api.dependencies import optional_auth
from backend.services.auth_service import UserType
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -235,7 +227,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
"active_jobs": health.get("active_jobs", 0),
"queue_length": health.get("queue_length", 0),
"ffmpeg_version": health.get("ffmpeg_version"),
"wheel_version": health.get("wheel_version"),
}
except Exception as e:
return {
@@ -246,239 +237,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
}
@router.get("/health/encoding-worker")
async def encoding_worker_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check encoding worker status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_encoding_worker_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("wheel_version"),
"active_jobs": status.get("active_jobs", 0),
"queue_length": status.get("queue_length", 0),
}
@router.get("/health/flacfetch")
async def flacfetch_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check flacfetch service status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_flacfetch_service_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("version"),
}
def check_audio_separator_status() -> Dict[str, Any]:
"""Check audio separator availability without calling the remote GPU service."""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
@router.get("/health/audio-separator")
async def audio_separator_health() -> Dict[str, Any]:
"""
Return the installed audio-separator package version.
Does NOT call the remote GPU service — that would cold-start an expensive
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
installed package version, which matches what the audio worker uses.
No authentication required.
"""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
def _get_encoding_worker_manager():
"""Get the EncodingWorkerManager instance. Separated for testability."""
try:
encoding_service = get_encoding_service()
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
return encoding_service._worker_manager
except Exception:
pass
return None
@router.get("/health/system-status")
async def system_status(
auth: Optional[Tuple] = Depends(optional_auth),
) -> Dict[str, Any]:
"""
Aggregated system status for all services.
Returns basic info for everyone. Admin users get additional
blue-green deployment details for the encoder.
"""
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
# Fetch all health statuses concurrently
import asyncio
encoding_task = check_encoding_worker_status()
flacfetch_task = check_flacfetch_service_status()
encoding_status, flacfetch_status = await asyncio.gather(
encoding_task, flacfetch_task
)
separator_status = check_audio_separator_status()
# Build frontend service info (from env vars baked at build time — returned as-is)
frontend_svc = {
"status": "ok",
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
}
# Build backend service info
backend_svc = {
"status": "ok",
"version": VERSION,
"deployed_at": STARTUP_TIME,
}
if COMMIT_SHA:
backend_svc["commit_sha"] = COMMIT_SHA
if PR_NUMBER:
backend_svc["pr_number"] = PR_NUMBER
if PR_TITLE:
backend_svc["pr_title"] = PR_TITLE
# Build encoder service info
encoder_svc = {
"status": "ok" if encoding_status.get("available") else "offline",
"version": encoding_status.get("wheel_version"),
"active_jobs": encoding_status.get("active_jobs", 0),
}
# Build flacfetch service info
flacfetch_svc = {
"status": "ok" if flacfetch_status.get("available") else "offline",
"version": flacfetch_status.get("version"),
}
# Build separator service info
separator_svc = {
"status": "ok" if separator_status.get("available") else "offline",
"version": separator_status.get("version"),
}
# Admin-only: blue-green encoder details from Firestore
if is_admin:
manager = _get_encoding_worker_manager()
if manager:
try:
config = manager.get_config()
encoder_svc["admin_details"] = {
"primary_vm": config.primary_vm,
"primary_ip": config.primary_ip,
"primary_version": config.primary_version,
"primary_deployed_at": config.primary_deployed_at,
"secondary_vm": config.secondary_vm,
"secondary_ip": config.secondary_ip,
"secondary_version": config.secondary_version,
"secondary_deployed_at": config.secondary_deployed_at,
"last_swap_at": config.last_swap_at,
"deploy_in_progress": config.deploy_in_progress,
"active_jobs": encoding_status.get("active_jobs", 0),
"queue_length": encoding_status.get("queue_length", 0),
}
except Exception as e:
logger.warning(f"Failed to get blue-green config: {e}")
# Add error details for offline services
if not flacfetch_status.get("available"):
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
if not separator_status.get("available"):
separator_svc["admin_details"] = {"error": separator_status.get("error")}
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
return {
"services": {
"frontend": frontend_svc,
"backend": backend_svc,
"encoder": encoder_svc,
"flacfetch": flacfetch_svc,
"separator": separator_svc,
}
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -542,131 +300,6 @@ async def detailed_health_check() -> Dict[str, Any]:
}
@router.get("/health/preload-status")
async def preload_status() -> Dict[str, Any]:
"""
Check status of preloaded resources for performance optimization.
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
were successfully preloaded at container startup. If any show as
not preloaded, check Cloud Run startup logs for errors.
Expected state after successful deployment:
- spacy.preloaded: true
- nltk.preloaded: true
- langfuse.preloaded: true (if configured) or configured: false
"""
# SpaCy status
spacy_model = get_preloaded_model("en_core_web_sm")
spacy_status = {
"preloaded": is_model_preloaded("en_core_web_sm"),
"model": "en_core_web_sm",
}
if spacy_model:
spacy_status["vocab_size"] = len(spacy_model.vocab)
# NLTK status
cmudict = get_preloaded_cmudict()
nltk_status = {
"preloaded": is_cmudict_preloaded(),
"resource": "cmudict",
}
if cmudict:
nltk_status["entries"] = len(cmudict)
# Langfuse status
langfuse_handler = get_preloaded_langfuse_handler()
langfuse_status = {
"configured": is_langfuse_configured(),
"preloaded": is_langfuse_preloaded(),
}
if langfuse_handler:
langfuse_status["handler_type"] = type(langfuse_handler).__name__
# Overall status
all_preloaded = (
spacy_status["preloaded"]
and nltk_status["preloaded"]
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
)
return {
"status": "ok" if all_preloaded else "degraded",
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
"spacy": spacy_status,
"nltk": nltk_status,
"langfuse": langfuse_status,
"performance_impact": {
"spacy_preload": "Saves ~60s on first lyrics correction",
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
}
}
@router.get("/health/job-consistency")
async def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+3 -925
View File
@@ -20,13 +20,12 @@ from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
from pydantic import BaseModel
from backend.workers.audio_worker import process_audio_separation
from backend.workers.auto_correct_worker import process_proactive_auto_correct
from backend.workers.lyrics_worker import process_lyrics_transcription
from backend.workers.screens_worker import generate_screens
from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult, UserType
from backend.services.auth_service import UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
@@ -77,18 +76,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
message=f"Job {job_id} not found"
)
# Reject workers for jobs in terminal states (failed, cancelled, complete)
# This prevents Cloud Tasks retries from triggering workers on dead jobs
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
if job_status in terminal_statuses:
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
return WorkerResponse(
status="skipped",
job_id=job_id,
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
)
# Check worker-specific progress in state_data
progress_key = f"{worker_name}_progress"
worker_progress = job.state_data.get(progress_key, {})
@@ -115,31 +102,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
return None
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
async def sync_disposable_domains_endpoint(
request: Request,
auth_result: AuthResult = Depends(require_admin),
):
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import EmailValidationService
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
EmailValidationService._blocklist_cache = None
return WorkerResponse(
status="completed",
job_id="sync-disposable-domains",
message=f"Synced {result['external_count']} domains"
)
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
@@ -228,48 +190,6 @@ async def trigger_lyrics_worker(
)
@router.post("/workers/auto-correct", response_model=WorkerResponse)
async def trigger_auto_correct_worker(
request: WorkerRequest,
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""Proactively pre-generate + cache AI auto-correct suggestions for a job.
Called by the lyrics worker once transcription + references are ready, so
the suggestions are cached before the reviewer opens the lyrics UI. Runs
HERE (the API service) because the service has the working Anthropic key /
compare-models config; the lyrics job does not.
Best-effort and runs synchronously within the request (the caller awaits
with its own timeout): the work never fails the karaoke job, and the
response just reports what happened. Gated by AUTO_CORRECT_PROACTIVE_ENABLED.
"""
job_id = request.job_id
# Extract trace context from incoming request (propagated via the trigger).
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=auto-correct")
add_span_attribute("job_id", job_id)
add_span_attribute("worker", "auto-correct")
# process_proactive_auto_correct already swallows its own errors; the extra
# guard keeps the endpoint itself best-effort no matter what.
try:
result = await process_proactive_auto_correct(job_id)
add_span_event("auto_correct_proactive", {"result": result.get("status", "unknown")})
return WorkerResponse(
status=result.get("status", "unknown"),
job_id=job_id,
message=f"Proactive auto-correct: {result.get('status', 'unknown')}",
)
except Exception as e: # noqa: BLE001 — best-effort, never fail the caller
logger.warning(f"[job:{job_id}] auto-correct endpoint error (non-fatal): {e}")
add_span_event("auto_correct_proactive", {"result": "error"})
return WorkerResponse(status="error", job_id=job_id, message=f"error: {e}")
@router.post("/workers/screens", response_model=WorkerResponse)
async def trigger_screens_worker(
request: WorkerRequest,
@@ -368,7 +288,7 @@ async def trigger_render_video_worker(
Idempotency: If worker is already running or complete, returns early.
Output: with_vocals.mkv in GCS
Next state: INSTRUMENTAL_SELECTED (instrumental is now selected during review)
Next state: AWAITING_INSTRUMENTAL_SELECTION
"""
job_id = request.job_id
@@ -396,857 +316,15 @@ async def trigger_render_video_worker(
)
@router.post("/jobs/{job_id}/check-idle-reminder")
async def check_idle_reminder(
job_id: str,
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Check if a job needs an idle reminder email.
This endpoint is called by a Cloud Tasks scheduled task after a job enters
a blocking state (AWAITING_REVIEW, AWAITING_AUDIO_EDIT, AWAITING_DURATION_CONFIRM,
or the legacy AWAITING_INSTRUMENTAL_SELECTION).
If the job is still in the blocking state and no reminder has been sent yet,
sends a reminder email to the user.
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
"""
from backend.models.job import JobStatus
from backend.services.job_notification_service import get_job_notification_service
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
add_span_attribute("job_id", job_id)
add_span_attribute("operation", "idle_reminder_check")
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
add_span_event("job_not_found")
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
# Check if job is still in a blocking state
# Note: AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs only
blocking_states = [
JobStatus.AWAITING_REVIEW,
JobStatus.AWAITING_INSTRUMENTAL_SELECTION,
JobStatus.AWAITING_DURATION_CONFIRM,
]
if job.status not in [s.value for s in blocking_states]:
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
add_span_event("not_blocking", {"current_status": job.status})
return {
"status": "skipped",
"job_id": job_id,
"message": f"Job not in blocking state (current: {job.status})"
}
# Normalize state_data to prevent None errors
state_data = job.state_data or {}
# Check if reminder was already sent (idempotency)
if state_data.get('reminder_sent'):
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
add_span_event("already_sent")
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
if getattr(job, 'made_for_you', False):
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
add_span_event("made_for_you_skip")
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
# Check if user has an email
if not job.user_email:
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
add_span_event("no_email")
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
# Determine action type
action_type = state_data.get('blocking_action_type')
if not action_type:
if job.status == JobStatus.AWAITING_REVIEW.value:
action_type = "lyrics"
elif job.status == JobStatus.AWAITING_DURATION_CONFIRM.value:
action_type = "duration_confirm"
else:
action_type = "instrumental"
# Send the reminder email
try:
notification_service = get_job_notification_service()
if action_type == "duration_confirm":
# Duration-confirm has its own dedicated reminder method
success = await notification_service.send_duration_confirm_reminder(job)
else:
success = await notification_service.send_action_reminder_email(
job_id=job.job_id,
user_email=job.user_email,
action_type=action_type,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
audio_hash=job.audio_hash,
review_token=job.review_token,
instrumental_token=job.instrumental_token,
)
if success:
# Mark reminder as sent (prevents duplicate sends)
job_manager.firestore.update_job(job_id, {
'state_data': {**state_data, 'reminder_sent': True}
})
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
add_span_event("reminder_sent", {"action_type": action_type})
return {
"status": "sent",
"job_id": job_id,
"message": f"Sent {action_type} reminder to {job.user_email}"
}
else:
logger.error(f"[job:{job_id}] Failed to send reminder email")
add_span_event("send_failed")
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
except Exception as e:
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.post("/youtube-queue/process")
async def process_youtube_upload_queue(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process queued YouTube uploads.
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
due to YouTube API quota exhaustion. Can also be triggered manually
from the admin dashboard.
Processes queued uploads one at a time, stopping if quota is exhausted.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("YOUTUBE_QUEUE_PROCESS starting")
add_span_attribute("operation", "youtube_queue_process")
# Run in background so the HTTP response returns quickly
# (Cloud Scheduler has a 30min timeout but we don't want to block)
async def _process():
try:
result = await process_queue()
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("queue_processing_started")
return {
"status": "started",
"message": "YouTube upload queue processing started in background"
}
@router.post("/process-stale-reviews")
async def process_stale_reviews_endpoint(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process stale review jobs — send reminders and expire old ones.
Called by Cloud Scheduler (hourly). Can also be triggered manually
from the admin dashboard.
- Jobs in review for >= 24h get a reminder email
- Jobs in review for >= 48h are auto-cancelled with credit refund
"""
from backend.workers.stale_review_processor import process_stale_reviews
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("STALE_REVIEW_PROCESS starting")
add_span_attribute("operation", "stale_review_process")
async def _process():
try:
result = await process_stale_reviews()
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("stale_review_processing_started")
return {
"status": "started",
"message": "Stale review processing started in background"
}
@router.post("/trigger-gdrive-validation")
async def trigger_gdrive_validation_endpoint(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger GDrive validation via the Cloud Function.
Called by a Cloud Tasks delayed task after a job completes and uploads
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
finishes before the validator checks for sequence gaps.
"""
from backend.services.gdrive_validator_client import trigger_gdrive_validation
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
add_span_attribute("operation", "gdrive_validation_trigger")
try:
result = trigger_gdrive_validation()
if result is None:
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
status = result.get("status", "unknown")
add_span_event("validation_complete", {"result_status": status})
return {"status": status, "message": "GDrive validation completed", "result": result}
except Exception as e:
logger.exception(f"GDrive validation trigger failed: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "message": str(e)}
@router.post("/retry-pending-render-jobs")
async def retry_pending_render_jobs(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Auto-retry jobs parked in RENDER_PENDING_CAPACITY.
Called every 5 minutes by Cloud Scheduler. When the GCE encoding worker
can't be started because the zone is exhausted, the render worker parks
the job in this state instead of failing it. This endpoint re-attempts
the render — when GCE has capacity, the start succeeds and the job
completes normally.
Strategy:
- Process up to MAX_PER_TICK jobs (the GCE worker is single-tenant per
render, so spinning up many in parallel just collides on the same VM)
- Pick the oldest first (longest waiter gets fairness)
- Time out jobs that have been waiting longer than MAX_WAIT_SECONDS;
transition them to FAILED with a clear permanent-failure message
- For each remaining job: clear error state, reset to REVIEW_COMPLETE,
kick off the render worker again
"""
from datetime import datetime, UTC, timedelta
from google.cloud.firestore_v1 import FieldFilter
from backend.services.worker_service import get_worker_service
from backend.models.job import JobStatus
# Long enough to ride out a sustained capacity outage but short enough
# that an op gets paged about it before the user sees a "still pending"
# job >24h old.
MAX_WAIT_SECONDS = 24 * 60 * 60 # 24h
MAX_PER_TICK = 1 # one render at a time on the GCE worker
extract_trace_context(dict(http_request.headers))
add_span_attribute("operation", "retry_pending_render_jobs")
logger.info("RETRY_PENDING_RENDER_JOBS starting")
job_manager = JobManager()
worker_service = get_worker_service()
jobs_ref = job_manager.firestore.db.collection("jobs")
# Filter by status only — sorting by updated_at server-side would require
# a composite Firestore index that we don't have. The pending-capacity
# population is small (handful of jobs at most), so streaming and
# sorting in Python is cheap and avoids the index ceremony.
pending_query = (
jobs_ref
.where(filter=FieldFilter("status", "==", JobStatus.RENDER_PENDING_CAPACITY.value))
.limit(50) # cap defensively in case the queue ever grows
)
now = datetime.now(UTC)
timed_out = []
retried = []
skipped = 0
# Materialize and sort by first_seen_at (oldest waiter first). Falls back
# to updated_at, then doc id, when meta is missing.
pending_docs = list(pending_query.stream())
def _sort_key(doc):
data = doc.to_dict() or {}
meta = (data.get("state_data") or {}).get("render_pending_capacity") or {}
first_seen = meta.get("first_seen_at") or ""
updated = data.get("updated_at")
# Firestore returns a datetime; isoformat sorts correctly within the same TZ.
updated_str = updated.isoformat() if hasattr(updated, "isoformat") else str(updated or "")
return (first_seen, updated_str, doc.id)
pending_docs.sort(key=_sort_key)
for doc in pending_docs:
job_data = doc.to_dict()
job_id = job_data.get("job_id", doc.id)
sd = job_data.get("state_data") or {}
pending_meta = sd.get("render_pending_capacity") or {}
first_seen = pending_meta.get("first_seen_at")
# Permanent-failure timeout
if first_seen:
try:
first_seen_dt = datetime.fromisoformat(first_seen)
age = now - first_seen_dt
except (ValueError, TypeError):
age = timedelta(0)
if age.total_seconds() > MAX_WAIT_SECONDS:
permanent_msg = (
"Encoding capacity remained unavailable after "
f"{int(age.total_seconds() // 3600)}h of automatic retries. "
"Please retry manually or contact support."
)
logger.error(
f"[job:{job_id}] Capacity wait exceeded {MAX_WAIT_SECONDS}s — failing permanently"
)
job_manager.fail_job(job_id, permanent_msg, error_details={
"stage": "render_video",
"permanent_capacity_timeout": True,
"first_seen_at": first_seen,
"attempts": pending_meta.get("attempt_count", 0),
})
timed_out.append(job_id)
continue
if len(retried) >= MAX_PER_TICK:
skipped += 1
continue
# Clear error state and reset render progress for idempotency.
# render_pending_capacity meta is preserved so the next failure (if any)
# increments the existing attempt counter.
job_manager.update_job(job_id, {
"error_message": None,
"error_details": None,
})
job_manager.update_state_data(job_id, "render_progress", {"stage": "pending"})
if not job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.REVIEW_COMPLETE,
progress=70,
message="Auto-retrying video render — encoding capacity check",
raise_on_invalid=False,
):
logger.warning(f"[job:{job_id}] Could not transition to REVIEW_COMPLETE for retry")
continue
try:
await worker_service.trigger_render_video_worker(job_id)
retried.append(job_id)
logger.info(f"[job:{job_id}] Auto-retry triggered")
except Exception as e:
logger.error(f"[job:{job_id}] Failed to trigger render worker: {e}")
# Job remains in REVIEW_COMPLETE — next tick will pick it up if it
# falls back to RENDER_PENDING_CAPACITY again.
add_span_event("retry_complete", {
"retried": len(retried),
"timed_out": len(timed_out),
"skipped": skipped,
})
logger.info(
f"RETRY_PENDING_RENDER_JOBS complete: retried={len(retried)} "
f"timed_out={len(timed_out)} skipped={skipped}"
)
return {
"status": "success",
"retried_jobs": retried,
"retried_count": len(retried),
"timed_out_jobs": timed_out,
"timed_out_count": len(timed_out),
"skipped_for_next_tick": skipped,
}
@router.post("/recover-stuck-jobs")
async def recover_stuck_jobs(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Detect and recover jobs stuck in a processing status.
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
- DOWNLOADING_AUDIO stuck >10 min:
* torrent sources (RED/OPS) → park into DOWNLOAD_PENDING_RETRY and keep
auto-retrying for up to 24h (handles rare tracks with intermittent
seeders and transient tracker outages gracefully).
* other sources → fail so the user can retry.
- DOWNLOAD_PENDING_RETRY → re-trigger the download; permanently fail after 24h.
- RENDERING_VIDEO stuck with no progress (worker died mid-render) →
re-park into RENDER_PENDING_CAPACITY so the existing
retry-pending-render-jobs cron resumes it automatically. Closes the orphan
gap where a hard-killed render worker leaves the job frozen at
"rendering_video" forever with no error and no Retry button.
"""
from backend.services.job_health_service import check_job_consistency
from backend.services.worker_service import get_worker_service
from backend.models.job import JobStatus
extract_trace_context(dict(http_request.headers))
logger.info("RECOVER_STUCK_JOBS starting")
add_span_attribute("operation", "recover_stuck_jobs")
job_manager = JobManager()
worker_service = get_worker_service()
from google.cloud.firestore_v1 import FieldFilter
jobs_ref = job_manager.firestore.db.collection("jobs")
recovered = [] # download jobs failed for user retry
download_parked = [] # transient torrent downloads parked for auto-retry
download_retried = [] # parked downloads re-triggered this tick
download_timed_out = [] # parked downloads that exceeded the 24h ceiling
reparked = [] # render jobs re-parked for auto-retry
# --- Stuck audio downloads: park torrents for auto-retry, else fail ---
stuck_query = jobs_ref.where(
filter=FieldFilter("status", "==", JobStatus.DOWNLOADING_AUDIO.value)
).stream()
for doc in stuck_query:
job_id = (doc.to_dict() or {}).get("job_id", doc.id)
job = job_manager.get_job(job_id)
if not job:
continue
if not any("downloading_audio_stuck" in i for i in check_job_consistency(job)):
continue
if _is_retryable_torrent_source(job):
logger.warning(f"[job:{job_id}] Parking stuck torrent download for auto-retry")
if _park_download_for_retry(job_manager, job_id, "download stalled (>10 min)"):
download_parked.append(job_id)
else:
logger.warning(f"[job:{job_id}] Recovering stuck download (non-torrent)")
job_manager.fail_job(
job_id,
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
)
recovered.append(job_id)
# --- Parked downloads: re-trigger, or permanently fail after 24h ---
retried_this_tick = 0
pending_query = jobs_ref.where(
filter=FieldFilter("status", "==", JobStatus.DOWNLOAD_PENDING_RETRY.value)
).limit(100).stream()
for doc in pending_query:
job_id = (doc.to_dict() or {}).get("job_id", doc.id)
job = job_manager.get_job(job_id)
if not job:
continue
meta = (job.state_data or {}).get("download_retry", {}) or {}
first_seen = meta.get("first_seen_at")
if _download_retry_expired(first_seen):
hours = int(DOWNLOAD_RETRY_MAX_SECONDS // 3600)
logger.error(f"[job:{job_id}] Download retries exhausted after {hours}h — failing")
job_manager.fail_job(
job_id,
f"Couldn't find a working source for this track after {hours}h of "
"automatic retries. It may be too rare to source right now — please "
"try a different version or contact support.",
error_details={"stage": "audio_download", "permanent_download_timeout": True},
)
download_timed_out.append(job_id)
continue
if retried_this_tick >= DOWNLOAD_RETRIES_PER_TICK:
continue
# Count the dispatch ATTEMPT against the per-tick cap before awaiting, so
# a failing audio-worker (helper returns False after an external dispatch)
# can't fan out to every queried job in one tick.
retried_this_tick += 1
if await _retrigger_parked_download(job_manager, worker_service, job_id):
download_retried.append(job_id)
# --- Orphaned renders: re-park for automatic retry ---
render_query = jobs_ref.where(
filter=FieldFilter("status", "==", JobStatus.RENDERING_VIDEO.value)
).stream()
for doc in render_query:
job_id = (doc.to_dict() or {}).get("job_id", doc.id)
job = job_manager.get_job(job_id)
if not job:
continue
if any("rendering_video_stuck" in i for i in check_job_consistency(job)):
logger.warning(f"[job:{job_id}] Re-parking orphaned render for auto-retry")
if _repark_stalled_render(job_manager, job_id):
reparked.append(job_id)
logger.info(
f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} "
f"download_parked={len(download_parked)} download_retried={len(download_retried)} "
f"download_timed_out={len(download_timed_out)} reparked={len(reparked)}"
)
add_span_event("recovery_complete", {
"recovered_count": len(recovered),
"download_parked_count": len(download_parked),
"download_retried_count": len(download_retried),
"download_timed_out_count": len(download_timed_out),
"reparked_count": len(reparked),
})
return {
"status": "success",
"recovered_jobs": recovered,
"recovered_count": len(recovered),
"download_parked_jobs": download_parked,
"download_retried_jobs": download_retried,
"download_timed_out_jobs": download_timed_out,
"reparked_render_jobs": reparked,
"reparked_render_count": len(reparked),
}
# Torrent sources whose downloads are worth auto-retrying over a long window —
# they fail transiently when a rare release has few/intermittent seeders or the
# private tracker is briefly unreachable. Non-torrent sources (YouTube/Spotify/
# URL) fail deterministically, so they are not auto-retried here.
RETRYABLE_TORRENT_SOURCES = frozenset({"RED", "OPS"})
DOWNLOAD_RETRY_MAX_SECONDS = 24 * 60 * 60 # keep retrying a transient download for 24h
DOWNLOAD_RETRIES_PER_TICK = 10 # cap re-triggers per 5-min tick
def _is_retryable_torrent_source(job) -> bool:
"""True if the job's audio came from a torrent tracker (RED/OPS)."""
return (getattr(job, "source_name", "") or "").upper() in RETRYABLE_TORRENT_SOURCES
def _download_retry_expired(first_seen_at: Optional[str]) -> bool:
"""True if the first parked failure is older than the 24h retry ceiling."""
if not first_seen_at:
return False
from datetime import datetime, UTC
try:
age = datetime.now(UTC) - datetime.fromisoformat(first_seen_at)
except (ValueError, TypeError):
return False
return age.total_seconds() > DOWNLOAD_RETRY_MAX_SECONDS
def _park_download_for_retry(job_manager: JobManager, job_id: str, reason: str) -> bool:
"""Park a transient torrent-download failure into DOWNLOAD_PENDING_RETRY.
The recover-stuck cron re-triggers parked downloads on later ticks (up to
24h) instead of dead-ending the job at FAILED after ~1h.
"""
from datetime import datetime, UTC
from backend.models.job import JobStatus
now = datetime.now(UTC).isoformat()
job = job_manager.get_job(job_id)
existing = (job.state_data or {}).get("download_retry", {}) if job else {}
meta = {
"first_seen_at": existing.get("first_seen_at") or now,
"last_attempt_at": now,
"attempt_count": int(existing.get("attempt_count", 0)) + 1,
"last_reason": str(reason)[:300],
}
job_manager.update_state_data(job_id, "download_retry", meta)
# Clear any Cloud Run auto-retry marker so the UI reflects the parked state.
job_manager.update_state_data(job_id, "cloud_run_retry_pending", None)
return bool(job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOAD_PENDING_RETRY,
progress=12,
message=(
"Still finding a good source for this track — it may be rare, or a "
"source is temporarily offline. We'll keep trying automatically for up "
"to 24 hours; no action needed."
),
raise_on_invalid=False,
))
async def _retrigger_parked_download(job_manager: JobManager, worker_service, job_id: str) -> bool:
"""Re-trigger the audio download for a DOWNLOAD_PENDING_RETRY job."""
from backend.models.job import JobStatus
job_manager.update_job(job_id, {"error_message": None, "error_details": None})
if not job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.DOWNLOADING_AUDIO,
progress=12,
message="Retrying audio download — checking sources again",
raise_on_invalid=False,
):
logger.warning(f"[job:{job_id}] Could not transition parked download for retry")
return False
try:
await worker_service.trigger_audio_download_worker(job_id)
logger.info(f"[job:{job_id}] Parked download re-triggered")
return True
except Exception as e: # noqa: BLE001
logger.error(f"[job:{job_id}] Failed to re-trigger parked download: {e}")
return False
def _repark_stalled_render(job_manager: JobManager, job_id: str) -> bool:
"""Re-park a render orphaned at RENDERING_VIDEO into RENDER_PENDING_CAPACITY.
Mirrors render_video_worker._park_job_for_capacity_retry but for a mid-render
stall (no EncodingWorkerStartError to carry). The existing
retry-pending-render-jobs cron then resets it to REVIEW_COMPLETE and
re-triggers the render, with the same 24h permanent-failure ceiling.
"""
from datetime import datetime, UTC
from backend.models.job import JobStatus
now = datetime.now(UTC).isoformat()
job = job_manager.get_job(job_id)
existing = (job.state_data or {}).get("render_pending_capacity", {}) if job else {}
pending_meta = {
"first_seen_at": existing.get("first_seen_at") or now,
"last_attempt_at": now,
"attempt_count": int(existing.get("attempt_count", 0)) + 1,
"last_code": "render_stalled",
"last_zone": existing.get("last_zone", ""),
"last_vm": existing.get("last_vm", ""),
}
job_manager.update_state_data(job_id, "render_pending_capacity", pending_meta)
return bool(job_manager.transition_to_state(
job_id=job_id,
new_status=JobStatus.RENDER_PENDING_CAPACITY,
progress=70,
message=(
"The render was interrupted (a worker was recycled). Your job will "
"retry automatically — no action needed."
),
raise_on_invalid=False,
))
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Internal health check endpoint.
Used to verify the internal API is responsive.
Requires admin authentication.
"""
return {"status": "healthy", "service": "karaoke-backend-internal"}
# =============================================================================
# Test Webhook Endpoint (for E2E testing)
# =============================================================================
class TestWebhookRequest(BaseModel):
"""
Request to simulate a Stripe webhook event for E2E testing.
This allows E2E tests to trigger payment flow logic without requiring
actual Stripe checkout sessions or valid webhook signatures.
"""
event_type: str # e.g., "checkout.session.completed"
session_id: str # Must start with "e2e-test-" prefix
customer_email: str
metadata: dict # order_type, package_id, credits, artist, title, etc.
class TestWebhookResponse(BaseModel):
"""Response from test webhook processing."""
status: str # "processed", "already_processed", "error"
job_id: Optional[str] = None # For made-for-you orders
credits_added: Optional[int] = None # For credit purchases
new_balance: Optional[int] = None # For credit purchases
message: str
@router.post("/test-webhook", response_model=TestWebhookResponse)
async def test_webhook(
request: TestWebhookRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Test endpoint that simulates Stripe webhook events for E2E testing.
SECURITY:
- Protected by admin authentication (X-Admin-Token header)
- Session IDs must start with "e2e-test-" prefix to prevent collision
with real Stripe sessions
- Only for E2E testing - bypasses Stripe signature verification
This endpoint reuses the same handler logic as the real webhook endpoint,
ensuring E2E tests validate actual business logic.
Supported event types:
- checkout.session.completed: Handles credit purchases and made-for-you orders
For credit purchases, metadata must include:
- package_id: e.g., "1_credit"
- credits: e.g., "1"
- user_email: Email of user to credit
For made-for-you orders, metadata must include:
- order_type: "made_for_you"
- customer_email: Customer email for delivery
- artist: Song artist
- title: Song title
- source_type: "search" or "youtube"
- youtube_url: (optional) If source_type is "youtube"
- notes: (optional) Customer notes
"""
from backend.services.user_service import get_user_service
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.api.routes.users import _handle_made_for_you_order
# Validate session_id prefix for safety
if not request.session_id.startswith("e2e-test-"):
logger.warning(f"Test webhook rejected: session_id '{request.session_id}' missing required prefix")
raise HTTPException(
status_code=400,
detail="Session ID must start with 'e2e-test-' prefix for test webhooks"
)
logger.info(f"TEST_WEBHOOK event_type={request.event_type} session_id={request.session_id}")
add_span_attribute("event_type", request.event_type)
add_span_attribute("session_id", request.session_id)
add_span_attribute("is_test_webhook", True)
user_service = get_user_service()
email_service = get_email_service()
stripe_service = get_stripe_service()
if request.event_type == "checkout.session.completed":
session_id = request.session_id
metadata = request.metadata
# Idempotency check: Skip if this session was already processed
if user_service.is_stripe_session_processed(session_id):
logger.info(f"Test webhook: session {session_id} already processed")
return TestWebhookResponse(
status="already_processed",
message=f"Session {session_id} was already processed"
)
# Check if this is a made-for-you order
if metadata.get("order_type") == "made_for_you":
try:
# Call the same handler used by the real webhook
await _handle_made_for_you_order(
session_id=session_id,
metadata=metadata,
user_service=user_service,
email_service=email_service,
)
# Get the job ID from the most recent job for this customer
# The handler creates a job, so we need to find it
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
# Look for the job by session_id pattern in state_data or by customer_email
# Since the job was just created, query by customer_email and made_for_you flag
customer_email = metadata.get("customer_email", "")
jobs_query = db.collection("jobs").where(
filter=FieldFilter("customer_email", "==", customer_email)
).where(
filter=FieldFilter("made_for_you", "==", True)
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(1)
jobs = list(jobs_query.stream())
job_id = jobs[0].to_dict().get("job_id") if jobs else None
logger.info(f"Test webhook: made-for-you order processed, job_id={job_id}")
return TestWebhookResponse(
status="processed",
job_id=job_id,
message=f"Made-for-you order created successfully"
)
except Exception as e:
logger.exception(f"Test webhook: error processing made-for-you order: {e}")
return TestWebhookResponse(
status="error",
message=f"Error processing made-for-you order: {str(e)}"
)
else:
# Handle regular credit purchase
# Build a synthetic session object that matches Stripe's format
synthetic_session = {
"id": session_id,
"customer_email": request.customer_email,
"metadata": metadata,
}
success, user_email, credits, msg = stripe_service.handle_checkout_completed(
synthetic_session
)
if not success:
logger.warning(f"Test webhook: credit purchase validation failed: {msg}")
return TestWebhookResponse(
status="error",
message=msg
)
if user_email and credits > 0:
# Add credits to user account
ok, new_balance, credit_msg = user_service.add_credits(
email=user_email,
amount=credits,
reason="stripe_purchase",
stripe_session_id=session_id,
)
if ok:
# Send confirmation email (same as real webhook)
email_service.send_credits_added(user_email, credits, new_balance)
logger.info(f"Test webhook: added {credits} credits to {user_email}, new balance: {new_balance}")
return TestWebhookResponse(
status="processed",
credits_added=credits,
new_balance=new_balance,
message=f"Added {credits} credits to {user_email}"
)
else:
logger.error(f"Test webhook: failed to add credits: {credit_msg}")
return TestWebhookResponse(
status="error",
message=f"Failed to add credits: {credit_msg}"
)
return TestWebhookResponse(
status="error",
message="Invalid credit purchase data"
)
else:
# Unsupported event type
logger.warning(f"Test webhook: unsupported event type '{request.event_type}'")
return TestWebhookResponse(
status="error",
message=f"Unsupported event type: {request.event_type}"
)
+531 -1794
View File
File diff suppressed because it is too large. Load diff
-52
View File
@@ -1,52 +0,0 @@
"""POST /api/parse-karaoke-titles — batch karaoke-filename → artist/title.
Internal admin endpoint used by kjbox to canonicalise downloaded-file names.
Reuses the Vertex Gemini parser; never blocks the caller (kjbox degrades to a
deterministic guess when this is unavailable).
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.parse_titles import parse_titles
logger = logging.getLogger(__name__)
router = APIRouter(tags=["parse-titles"])
class ParseItem(BaseModel):
id: str
filename: str
channel: Optional[str] = None
source: Optional[str] = None
class ParseRequest(BaseModel):
items: list[ParseItem]
class ParseResult(BaseModel):
id: str
artist: str
title: str
confidence: float
class ParseResponse(BaseModel):
results: list[ParseResult]
@router.post("/parse-karaoke-titles", response_model=ParseResponse)
async def parse_karaoke_titles(
body: ParseRequest,
auth_result: AuthResult = Depends(require_admin),
):
items = [i.model_dump() for i in body.items]
results = await parse_titles(items)
logger.info("parse-karaoke-titles: %d items", len(items))
return {"results": results}
-263
View File
@@ -1,263 +0,0 @@
"""
Push Notification API routes.
Provides endpoints for managing Web Push notification subscriptions:
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
- POST /api/push/subscribe: Register a push subscription
- POST /api/push/unsubscribe: Remove a push subscription
- GET /api/push/subscriptions: List user's subscriptions
- POST /api/push/test: Send a test notification (admin only)
"""
import logging
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/push", tags=["push"])
# Request/Response Models
class VapidPublicKeyResponse(BaseModel):
"""Response containing VAPID public key."""
enabled: bool
vapid_public_key: Optional[str] = None
class SubscribeRequest(BaseModel):
"""Request to subscribe to push notifications."""
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
tenant_id: Optional[str] = None # Tenant scope for notification isolation
class SubscribeResponse(BaseModel):
"""Response after subscribing."""
status: str
message: str
class UnsubscribeRequest(BaseModel):
"""Request to unsubscribe from push notifications."""
endpoint: str
class UnsubscribeResponse(BaseModel):
"""Response after unsubscribing."""
status: str
message: str
class SubscriptionInfo(BaseModel):
"""Information about a push subscription."""
endpoint: str
device_name: Optional[str] = None
created_at: Optional[str] = None
last_used_at: Optional[str] = None
class SubscriptionsListResponse(BaseModel):
"""Response containing user's subscriptions."""
subscriptions: List[SubscriptionInfo]
count: int
class TestNotificationRequest(BaseModel):
"""Request to send a test notification."""
title: Optional[str] = "Test Notification"
body: Optional[str] = "This is a test push notification from Karaoke Generator"
class TestNotificationResponse(BaseModel):
"""Response after sending test notification."""
status: str
sent_count: int
message: str
# Routes
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
async def get_vapid_public_key():
"""
Get the VAPID public key for push subscription.
This endpoint is public - no authentication required.
Returns the public key needed for client-side PushManager.subscribe().
"""
settings = get_settings()
push_service = get_push_notification_service()
if not settings.enable_push_notifications:
return VapidPublicKeyResponse(enabled=False)
public_key = push_service.get_public_key()
if not public_key:
return VapidPublicKeyResponse(enabled=False)
return VapidPublicKeyResponse(
enabled=True,
vapid_public_key=public_key
)
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
subscribe_request: SubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
Register a push notification subscription for the current user.
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage. The tenant is resolved from:
1. Request body tenant_id field (explicit from frontend)
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
# Resolve tenant_id: prefer explicit from body, fall back to middleware
tenant_id = subscribe_request.tenant_id
if tenant_id is None:
tenant_id = getattr(http_request.state, "tenant_id", None)
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=subscribe_request.endpoint,
keys=subscribe_request.keys,
device_name=subscribe_request.device_name,
tenant_id=tenant_id or None,
)
if not success:
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
return SubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRegistered")
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
Remove a push notification subscription.
Requires authentication. Users can only remove their own subscriptions.
"""
locale = get_locale_from_request(http_request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
success = await push_service.remove_subscription(
user_email=auth_result.user_email,
endpoint=request.endpoint
)
if not success:
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionNotFound")
)
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRemoved")
)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
List all push notification subscriptions for the current user.
Requires authentication.
"""
locale = get_locale_from_request(request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
return SubscriptionsListResponse(
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
count=len(subscriptions)
)
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_admin)
):
"""
Send a test push notification to the current user's devices.
Admin only. Useful for testing push notification setup.
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
sent_count = await push_service.send_push(
user_email=auth_result.user_email,
title=request.title or "Test Notification",
body=request.body or "This is a test push notification",
url="/app/",
tag="test"
)
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
)
-346
View File
@@ -1,346 +0,0 @@
"""
Admin API routes for blocklist and YouTube queue management.
Handles:
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- YouTube upload queue management
"""
import logging
from datetime import datetime
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# =============================================================================
# Response Models
# =============================================================================
class YouTubeQueueEntry(BaseModel):
"""A queued YouTube upload entry."""
job_id: str
status: str
reason: Optional[str] = None
user_email: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
brand_code: Optional[str] = None
queued_at: Optional[datetime] = None
attempts: int = 0
max_attempts: int = 5
last_error: Optional[str] = None
youtube_url: Optional[str] = None
notification_sent: bool = False
class YouTubeQueueListResponse(BaseModel):
"""List of YouTube upload queue entries."""
entries: List[YouTubeQueueEntry]
stats: dict
class BlocklistsResponse(BaseModel):
"""All blocklist data with domain source separation."""
external_domains: List[str]
manual_domains: List[str]
allowlisted_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
last_sync_at: Optional[datetime] = None
last_sync_count: Optional[int] = None
updated_at: Optional[datetime] = None
updated_by: Optional[str] = None
class DomainRequest(BaseModel):
"""Request to add/remove a domain."""
domain: str
class EmailRequest(BaseModel):
"""Request to add/remove an email."""
email: str
class IPRequest(BaseModel):
"""Request to add/remove an IP."""
ip_address: str
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@router.get("/blocklists", response_model=BlocklistsResponse)
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all blocklist data with domain source separation."""
email_validation = get_email_validation_service()
raw_data = email_validation.get_blocklist_raw_data()
return BlocklistsResponse(**raw_data)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
async def add_disposable_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.domainAddedToBl", domain=domain)
)
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
async def remove_disposable_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.domainRemovedFromBl", domain=domain)
)
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
async def add_allowlisted_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the allowlist (overrides external blocklist)."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
async def remove_allowlisted_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the allowlist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
@router.post("/blocklists/sync", response_model=SuccessResponse)
async def trigger_sync(
auth_result: AuthResult = Depends(require_admin),
):
"""Manually trigger a sync of the external disposable domain blocklist."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
# Invalidate cache
EmailValidationService._blocklist_cache = None
return SuccessResponse(
success=True,
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
)
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
async def add_blocked_email(
request: EmailRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an email to the blocked emails list."""
email_validation = get_email_validation_service()
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.emailAddedToBl", email=email)
)
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
async def remove_blocked_email(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an email from the blocked emails list."""
email_validation = get_email_validation_service()
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.emailRemovedFromBl", email=email)
)
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
async def add_blocked_ip(
request: IPRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an IP address to the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
)
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
async def remove_blocked_ip(
ip_address: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an IP address from the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
)
# =============================================================================
# YouTube Upload Queue Management Endpoints
# =============================================================================
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
async def get_youtube_queue(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get YouTube upload queue entries and stats.
Returns all queue entries (any status) and aggregate statistics.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
entries = queue_service.get_all_queue_entries(limit=50)
stats = queue_service.get_queue_stats()
return YouTubeQueueListResponse(
entries=[YouTubeQueueEntry(**entry) for entry in entries],
stats=stats,
)
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
async def retry_youtube_upload(
job_id: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually retry a failed YouTube upload.
Resets the entry back to 'queued' status with 0 attempts.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
if not queue_service.retry_upload(job_id):
raise HTTPException(
status_code=404,
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
)
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
)
@router.post("/youtube-queue/process", response_model=SuccessResponse)
async def trigger_youtube_queue_processing(
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually trigger YouTube upload queue processing.
Same as the scheduled processor but triggered from the admin dashboard.
Runs in the background and returns immediately.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
import asyncio
async def _process():
try:
result = await process_youtube_upload_queue()
logger.info(f"Manual YouTube queue processing complete: {result}")
except Exception as e:
logger.exception(f"Manual YouTube queue processing failed: {e}")
# Fire and forget
try:
loop = asyncio.get_running_loop()
loop.create_task(_process())
except RuntimeError:
pass
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeQueueProcessingStarted")
)
-374
View File
@@ -1,374 +0,0 @@
"""
Referral API routes.
Handles:
- Public referral interstitial (click tracking)
- Authenticated referral dashboard
- Stripe Connect onboarding
- Admin vanity link creation and link management
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth, require_admin
from backend.models.referral import (
CreateReferralLinkRequest,
UpdateReferralLinkRequest,
ReferralInterstitialResponse,
)
from backend.services.flyer_service import get_flyer_service, FlyerError
from backend.services.referral_service import get_referral_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/referrals", tags=["referrals"])
class GenerateFlyerRequest(BaseModel):
theme: str = Field(..., pattern="^(light|dark)$")
qr_data_url: str = Field(..., max_length=500_000)
# ============================================================================
# Public endpoints
# ============================================================================
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
async def get_referral_interstitial(code: str, no_track: bool = False):
"""
Public endpoint: get interstitial data for a referral link.
Increments click count unless no_track=true. Returns valid=False if code not found.
"""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
return ReferralInterstitialResponse(valid=False)
# Track the click (skip when fetching for display purposes)
if not no_track:
try:
service.increment_clicks(code)
except Exception:
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
return ReferralInterstitialResponse(
valid=True,
referral_code=link.code,
referrer_display_name=link.display_name,
discount_percent=link.discount_percent,
discount_duration_days=link.discount_duration_days,
custom_message=link.custom_message,
)
# ============================================================================
# Authenticated user endpoints
# ============================================================================
@router.get("/me")
async def get_my_dashboard(auth=Depends(require_auth)):
"""Get referral dashboard data for the authenticated user."""
service = get_referral_service()
return service.get_dashboard_data(auth.user_email)
@router.put("/me")
async def update_my_link(
updates: UpdateReferralLinkRequest,
auth=Depends(require_auth),
):
"""Update display_name and custom_message on the user's own referral link."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
update_fields = {}
if updates.display_name is not None:
update_fields["display_name"] = updates.display_name
if updates.custom_message is not None:
update_fields["custom_message"] = updates.custom_message
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(link.code, **update_fields)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.post("/me/connect")
async def start_stripe_connect(auth=Depends(require_auth)):
"""Start Stripe Connect onboarding for the authenticated user."""
# Check if user already has a Connect account
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(auth.user_email)
if user and user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
service = get_referral_service()
account_id, onboarding_url = service.create_connect_account(auth.user_email)
if not account_id:
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
return {"account_id": account_id, "onboarding_url": onboarding_url}
@router.post("/me/connect/dashboard-link")
async def get_connect_dashboard_link(auth=Depends(require_auth)):
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_login_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
return {"url": url}
@router.post("/me/connect/update-link")
async def get_connect_update_link(auth=Depends(require_auth)):
"""Get a link to update Stripe Connect account info (re-onboarding)."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_update_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create update link")
return {"url": url}
class VanityRequest(BaseModel):
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
@router.post("/me/vanity-request")
async def request_vanity_url(
request: VanityRequest,
auth=Depends(require_auth),
):
"""Request a custom vanity referral URL. Sends email to admin for review."""
from backend.services.email_service import get_email_service
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
# Check if code is already taken
existing = service.get_link_by_code(request.desired_code)
if existing:
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
email_service = get_email_service()
email_service.provider.send_email(
to_email="andrew@nomadkaraoke.com",
subject=f"Vanity referral URL request: {request.desired_code}",
html_content=f"""
<h2>Vanity Referral URL Request</h2>
<p><strong>User:</strong> {auth.user_email}</p>
<p><strong>Current code:</strong> {link.code}</p>
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
""",
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
)
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
return {"ok": True, "message": "Request sent"}
@router.post("/me/flyer")
async def generate_flyer(
request: GenerateFlyerRequest,
auth=Depends(require_auth),
):
"""Generate a personalized referral flyer PDF."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
},
)
# ============================================================================
# Admin endpoints
# ============================================================================
@router.post("/admin/vanity")
async def create_vanity_link(
request: CreateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: create a vanity referral link."""
if not request.vanity_code:
raise HTTPException(status_code=400, detail="vanity_code is required")
service = get_referral_service()
success, link, message = service.create_vanity_link(
code=request.vanity_code,
owner_email=request.owner_email,
display_name=request.display_name,
custom_message=request.custom_message,
discount_percent=request.discount_percent,
kickback_percent=request.kickback_percent,
discount_duration_days=request.discount_duration_days,
earning_duration_days=request.earning_duration_days,
)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "code": link.code, "message": message}
@router.get("/admin/links")
async def list_links(
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
auth=Depends(require_admin),
):
"""Admin: list all referral links."""
service = get_referral_service()
links = service.list_links(limit=limit, offset=offset)
return {
"links": [link.model_dump(mode="json") for link in links],
"count": len(links),
}
@router.post("/admin/apply-discount")
async def admin_apply_discount(
request: Request,
auth=Depends(require_admin),
):
"""Admin: apply a referral discount to any user, even without a referral link."""
from backend.services.user_service import get_user_service
body = await request.json()
user_email = body.get("email", "").lower()
discount_percent = body.get("discount_percent", 10)
duration_days = body.get("duration_days", 30)
referral_code = body.get("referral_code", "admin-grant")
if not user_email:
raise HTTPException(status_code=400, detail="Email is required")
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
from datetime import datetime, timedelta
now = datetime.utcnow()
user_service.update_user(
user_email,
referred_by_code=referral_code,
referred_at=now,
referral_discount_expires_at=now + timedelta(days=duration_days),
)
# Create/ensure the Stripe coupon exists for this discount percent
service = get_referral_service()
service.get_or_create_stripe_coupon(discount_percent)
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
return {
"ok": True,
"message": f"Applied {discount_percent}% discount for {duration_days} days",
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
}
@router.put("/admin/links/{code}")
async def update_link(
code: str,
updates: UpdateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: update any referral link's settings."""
service = get_referral_service()
update_fields = {
k: v for k, v in updates.model_dump().items() if v is not None
}
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(code, **update_fields)
if not success:
raise HTTPException(status_code=404, detail=message)
return {"ok": True, "message": message}
@router.post("/admin/links/{code}/flyer")
async def admin_generate_flyer(
code: str,
request: GenerateFlyerRequest,
auth=Depends(require_admin),
):
"""Admin: generate a personalized referral flyer PDF for any code."""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
},
)
+187 -1963
View File
File diff suppressed because it is too large. Load diff
-121
View File
@@ -1,121 +0,0 @@
"""
Tenant API routes for white-label portal configuration.
These endpoints are public (no auth required) since the frontend needs
to fetch tenant branding before the user logs in.
"""
import logging
from typing import Optional
from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
@router.get("/config", response_model=TenantConfigResponse)
async def get_tenant_config(
request: Request,
tenant: Optional[str] = Query(
None,
description="Tenant ID override (for development). If not provided, detected from subdomain.",
),
):
"""
Get tenant configuration for the current portal.
This endpoint detects the tenant from:
1. Query parameter `tenant` (for development/testing)
2. X-Tenant-ID header (set by frontend)
3. Host header subdomain
Returns the public tenant configuration (branding, features, defaults)
or indicates this is the default Nomad Karaoke portal.
"""
tenant_service = get_tenant_service()
# Priority 1: Query parameter (for dev/testing)
tenant_id = tenant
# Priority 2: X-Tenant-ID header
if not tenant_id:
tenant_id = request.headers.get("X-Tenant-ID")
# Priority 3: Detect from Host header
if not tenant_id:
host = request.headers.get("Host", "")
# Check if this is a tenant subdomain
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
if host and "nomadkaraoke.com" in host.lower():
parts = host.lower().split(".")
# Skip known non-tenant subdomains
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
potential_tenant = parts[0]
if tenant_service.tenant_exists(potential_tenant):
tenant_id = potential_tenant
# If no tenant detected, return default config
if not tenant_id:
logger.debug("No tenant detected, returning default config")
return TenantConfigResponse(tenant=None, is_default=True)
# Load tenant config
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
logger.warning(f"Tenant not found: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
logger.info(f"Returning config for tenant: {tenant_id}")
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
async def get_tenant_config_by_id(tenant_id: str):
"""
Get tenant configuration by explicit tenant ID.
This is useful for admin tools or debugging.
"""
tenant_service = get_tenant_service()
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
return TenantConfigResponse(tenant=None, is_default=True)
# Check if tenant is active
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/asset/{tenant_id}/{asset_name}")
async def get_tenant_asset(tenant_id: str, asset_name: str):
"""
Get a signed URL for a tenant asset (logo, favicon, etc.).
This redirects to the signed GCS URL for the asset.
"""
from fastapi.responses import RedirectResponse
tenant_service = get_tenant_service()
url = tenant_service.get_asset_url(tenant_id, asset_name)
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
return RedirectResponse(url=url)
+13 -18
View File
@@ -9,11 +9,10 @@ before job creation.
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -21,7 +20,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes(request: Request) -> ThemesListResponse:
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
@@ -30,7 +29,6 @@ async def list_themes(request: Request) -> ThemesListResponse:
This endpoint is public and does not require authentication.
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
@@ -39,12 +37,12 @@ async def list_themes(request: Request) -> ThemesListResponse:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themesLoadError")
detail="Failed to load themes"
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
@@ -60,7 +58,6 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
@@ -68,7 +65,7 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
if theme is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
@@ -78,12 +75,12 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themeLoadError")
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str, request: Request) -> dict:
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
@@ -99,21 +96,20 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
Raises:
404: Theme not found or no preview available
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
@@ -123,12 +119,12 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.previewLoadError")
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
@@ -145,14 +141,13 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
@@ -163,5 +158,5 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.youtubeDescriptionLoadError")
detail="Failed to load YouTube description"
)
+369 -1356
View File
File diff suppressed because it is too large. Load diff
-35
View File
@@ -1,35 +0,0 @@
# Cloud Build config for GPU base image
#
# Builds the CUDA + PyTorch + models base image used by the audio worker.
# This image is large (~10GB) so builds take 15-30 minutes.
#
# Trigger manually:
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
#
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '-f'
- 'backend/Dockerfile.gpu-base'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
- '--build-arg'
- 'BUILD_DATE=$_BUILD_DATE'
- '--build-arg'
- 'BUILD_ID=$BUILD_ID'
- '.'
images:
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
timeout: '3600s'
options:
machineType: 'E2_HIGHCPU_32'
diskSizeGb: 100
+2 -144
View File
@@ -20,17 +20,6 @@ class Settings(BaseSettings):
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
# Nomad master fast-sync: push freshly-published Nomad 720p masters straight to the
# Divebar GCS mirror so kjbox (which mirrors that folder every 5 min) picks them up
# within minutes instead of waiting for the nightly Drive->GCS VM. Nomad-brand only.
divebar_files_bucket: str = os.getenv("DIVEBAR_FILES_BUCKET", "nomadkaraoke-divebar-files")
nomad_master_gcs_prefix: str = os.getenv("NOMAD_MASTER_GCS_PREFIX", "files/Nomad Karaoke/MP4-720p")
nomad_master_fast_sync_enabled: bool = os.getenv("NOMAD_MASTER_FAST_SYNC_ENABLED", "true").lower() == "true"
# Original-vocals guide: a padded isolated-vocals file (silence[intro] + vocals) pushed to
# a sibling prefix in the same Divebar bucket, which kjbox mirrors into NOMAD-vocals-padded/
# to power the "Original Vocals" sing-along slider. Same brand gating as the master push.
nomad_vocals_guide_gcs_prefix: str = os.getenv("NOMAD_VOCALS_GUIDE_GCS_PREFIX", "files/Nomad Karaoke/vocals-padded")
# Audio Separator API (for GPU processing)
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
@@ -43,10 +32,6 @@ class Settings(BaseSettings):
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
@@ -56,15 +41,7 @@ class Settings(BaseSettings):
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
# Lyrics Correction Settings
# Skip all auto-correction (both agentic AI and heuristic handlers).
# When true, raw transcription goes directly to human review without any automatic fixes.
# This is the default because auto-correction currently creates more work for reviewers.
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
# Agentic AI Correction (for lyrics correction via LLM)
# Only used when skip_correction=false
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
@@ -72,75 +49,6 @@ class Settings(BaseSettings):
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# Lyrics transcription outer (safety-net) timeout, scaled by audio duration.
# Long inputs (e.g. 30+ minute mashups) legitimately take longer to transcribe,
# so the timeout = clamp(floor, duration * per_audio_second, cap). The cap must
# stay under the Cloud Run Job task timeout so we surface a clean RuntimeError
# before the platform hard-kills the task. The lyrics-transcription-job sets the
# task timeout to 3000s and overrides this cap to 2700s via env; the 1700s
# default here is the safe value if that env is ever unset (under an 1800s task).
transcription_timeout_floor_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_FLOOR_SECONDS", "1200"))
transcription_timeout_cap_seconds: int = int(os.getenv("TRANSCRIPTION_TIMEOUT_CAP_SECONDS", "1700"))
transcription_timeout_per_audio_second: float = float(os.getenv("TRANSCRIPTION_TIMEOUT_PER_AUDIO_SECOND", "1.0"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
# Custom Lyrics (LLM-powered) — used by the Custom Lyrics mode in the
# Edit All Lyrics modal. Stateless service: takes operator-provided
# custom-lyrics text/file and returns transformed lyric lines.
custom_lyrics_model: str = os.getenv("CUSTOM_LYRICS_MODEL", "gemini-3.1-pro-preview")
custom_lyrics_max_file_mb: int = int(os.getenv("CUSTOM_LYRICS_MAX_FILE_MB", "5"))
custom_lyrics_max_input_lines: int = int(os.getenv("CUSTOM_LYRICS_MAX_INPUT_LINES", "500"))
custom_lyrics_max_iterations: int = int(os.getenv("CUSTOM_LYRICS_MAX_ITERATIONS", "4"))
custom_lyrics_default_strictness: str = os.getenv("CUSTOM_LYRICS_DEFAULT_STRICTNESS", "balanced")
custom_lyrics_max_output_lines_multiplier: float = float(os.getenv("CUSTOM_LYRICS_MAX_OUTPUT_LINES_MULTIPLIER", "2.0"))
# AI auto-correct suggestions — opt-in, user-triggered from the lyrics
# review UI. Stateless: one whole-song LLM call returning word-level
# suggestions the reviewer accepts/rejects individually.
auto_correct_model: str = os.getenv("AUTO_CORRECT_MODEL", "gemini-3.1-pro-preview")
# Models used by the multi-model "compare" mode (semicolon-separated —
# Cloud Run --set-env-vars is comma-delimited). Empty = single-model only.
auto_correct_compare_models: str = os.getenv("AUTO_CORRECT_COMPARE_MODELS", "")
# Proactively generate (and cache) auto-correct suggestions in the lyrics
# worker once transcription + references are ready, so the review UI has
# them instantly. Best-effort: failures never block or fail the job. Only
# meaningful where ANTHROPIC_API_KEY + AUTO_CORRECT_COMPARE_MODELS are set
# (the lyrics-transcription-job, via ci.yml). Default off.
auto_correct_proactive_enabled: bool = os.getenv(
"AUTO_CORRECT_PROACTIVE_ENABLED", "false"
).lower() in ("true", "1", "yes")
# Match judge — artist/title formatting + match-acceptance for the job
# submission flow. Deterministic + catalog layers always run; the light AI
# layer (Vertex Gemini Flash) fires only when those aren't confident.
match_judge_enabled: bool = os.getenv("MATCH_JUDGE_ENABLED", "true").lower() in (
"true", "1", "yes",
)
match_judge_model: str = os.getenv("MATCH_JUDGE_MODEL", "gemini-3.5-flash")
# Bounds the AI call so a hung model never strands the submission UI. Kept
# generous because Vertex returns 504 DEADLINE_EXCEEDED if generation can't
# finish within this deadline (3s was too tight for gemini-3.5-flash); the
# call still runs off the critical path and degrades to "none" on timeout.
match_judge_timeout_ms: int = int(os.getenv("MATCH_JUDGE_TIMEOUT_MS", "12000"))
# Batch karaoke-filename parser (kjbox download-naming). Reuses the Vertex
# Gemini stack. Larger timeout than match_judge because it batches ~200 items.
parse_titles_enabled: bool = os.getenv("PARSE_TITLES_ENABLED", "true").lower() in (
"1", "true", "yes",
)
parse_titles_model: str = os.getenv("PARSE_TITLES_MODEL", "gemini-3.5-flash")
parse_titles_timeout_ms: int = int(os.getenv("PARSE_TITLES_TIMEOUT_MS", "20000"))
parse_titles_max_items: int = int(os.getenv("PARSE_TITLES_MAX_ITEMS", "200"))
# Items per Gemini call: large client batches are split into concurrent
# chunks of this size so one call never outgrows parse_titles_timeout_ms
# (a 100-item single call did, blanking the whole batch).
parse_titles_chunk_size: int = int(os.getenv("PARSE_TITLES_CHUNK_SIZE", "10"))
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
@@ -160,23 +68,7 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# Optional capacity-fallback VMs in alternate zones. JSON list of objects:
# '[{"vm":"encoding-worker-fallback-a","zone":"us-central1-a","ip":"34.x.x.x"}]'
# Empty by default. After `pulumi up` provisions the fallback VMs (Phase 3
# of the capacity-resilience plan), populate this var with their public
# IPs to enable multi-zone failover when us-central1-c is exhausted.
encoding_worker_fallback_vms: Optional[str] = os.getenv("ENCODING_WORKER_FALLBACK_VMS")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide catalog API (for song/artist autocomplete)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -191,13 +83,7 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -209,41 +95,13 @@ class Settings(BaseSettings):
# Default values for web service jobs (YouTube/Dropbox distribution)
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Default CDG/TXT generation settings
# When True, CDG and TXT packages are generated by default (when a theme is set)
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
# Push Notifications Configuration
# When enabled, users can subscribe to push notifications for job status updates
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
# Maximum number of push subscriptions per user (oldest removed when exceeded)
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
# VAPID subject (email or URL for push service to contact)
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
# Secret Manager cache
_secret_cache: Dict[str, str] = {}
@@ -1,81 +0,0 @@
"""One-shot script to populate the year-5-stars fixture from a real production job.
Run once:
poetry run python -m backend.eval.custom_lyrics._bootstrap_year5
"""
from __future__ import annotations
import json
import shutil
from pathlib import Path
from google.cloud import firestore
JOB_ID = "2cb49a45"
FIXTURE_DIR = Path(__file__).parent / "fixtures" / "year-5-stars-shake-it-off"
CLIENT_INPUT_SRC = Path("/Users/andrew/Projects/nomadkaraoke/year-5-stars-client-custom-lyrics.txt")
def main() -> None:
FIXTURE_DIR.mkdir(parents=True, exist_ok=True)
fs = firestore.Client(project="nomadkaraoke")
job_doc = fs.collection("jobs").document(JOB_ID).get()
if not job_doc.exists:
raise SystemExit(f"Job {JOB_ID} not found")
job = job_doc.to_dict()
# Corrected lyrics are stored inline in state_data (not in GCS for this job)
corrected = (
job.get("state_data", {}).get("corrected_lyrics")
or job.get("corrected_lyrics")
or {}
)
if not corrected:
# Fallback: try GCS path if present
corrected_path = job.get("corrected_json_gcs_path") or job.get("lyrics", {}).get("corrected_path")
if corrected_path:
from google.cloud import storage # noqa: PLC0415
storage_client = storage.Client(project="nomadkaraoke")
bucket_name, *blob_parts = corrected_path.replace("gs://", "").split("/", 1)
blob = storage_client.bucket(bucket_name).blob(blob_parts[0])
corrected = json.loads(blob.download_as_bytes())
else:
raise SystemExit(f"Job {JOB_ID} has no corrected lyrics (checked state_data and GCS path fields)")
segments = corrected.get("corrected_segments") or corrected.get("segments") or []
if not segments:
raise SystemExit("No segments found in corrected.json")
(FIXTURE_DIR / "original_segments.json").write_text(json.dumps(segments, indent=2))
(FIXTURE_DIR / "original_lyrics.txt").write_text(
"\n".join(seg.get("text", "").strip() for seg in segments) + "\n"
)
if CLIENT_INPUT_SRC.exists():
shutil.copy(CLIENT_INPUT_SRC, FIXTURE_DIR / "client_input.txt")
else:
print(f"WARNING: {CLIENT_INPUT_SRC} not found; populate client_input.txt manually")
metadata = {
"id": "year-5-stars-shake-it-off",
"artist": job.get("artist") or "Taylor Swift",
"title": job.get("title") or "Shake It Off",
"source_job_id": JOB_ID,
"difficulty": "hard",
"input_style": "long substantive lines, name-heavy, mismatched syllable budget",
"settings_to_test": [
{"name": "default", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "balanced"},
{"name": "verbatim", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "verbatim"},
{"name": "strict", "allow_reword": True, "allow_omit": True, "fixed_line_count": True, "strictness": "strict"},
],
}
(FIXTURE_DIR / "metadata.json").write_text(json.dumps(metadata, indent=2))
print(f"Fixture populated at {FIXTURE_DIR}")
print(f" - {len(segments)} segments")
if __name__ == "__main__":
main()
-64
View File
@@ -1,64 +0,0 @@
"""Disk-backed cache for LLM responses, keyed by (system, user, model, settings)."""
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Any, Optional
class LlmCallCache:
def __init__(self, cache_dir: Path, replay_only: bool = False) -> None:
self.cache_dir = cache_dir
self.cache_dir.mkdir(parents=True, exist_ok=True)
self.replay_only = replay_only
def _key(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
) -> str:
h = hashlib.sha256()
h.update(system_prompt.encode())
h.update(b"\x00")
h.update(user_prompt.encode())
h.update(b"\x00")
h.update(model.encode())
h.update(b"\x00")
h.update(json.dumps(settings_dict, sort_keys=True).encode())
return h.hexdigest()
def _path(self, key: str) -> Path:
return self.cache_dir / f"{key}.json"
def get(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
) -> Optional[list[str]]:
path = self._path(self._key(system_prompt, user_prompt, model=model, settings_dict=settings_dict))
if not path.exists():
if self.replay_only:
raise RuntimeError(f"cache miss in replay-only mode: {path.name}")
return None
return json.loads(path.read_text())["lines"]
def set(
self,
system_prompt: str,
user_prompt: str,
*,
model: str,
settings_dict: dict[str, Any],
response_lines: list[str],
) -> None:
if self.replay_only:
return
path = self._path(self._key(system_prompt, user_prompt, model=model, settings_dict=settings_dict))
path.write_text(json.dumps({"lines": response_lines}, indent=2))
@@ -1,2 +0,0 @@
*
!.gitignore
@@ -1 +0,0 @@
We're singing this for Alex's birthday! Can you change the lyrics so it says Alex's name instead of "he"? Like "For she's a jolly good fellow" or however it makes sense with Alex. Just make it clear it's for Alex. It's a birthday celebration for our friend Alex.
@@ -1,11 +0,0 @@
{
"id": "jolly-good-fellow-name-replace",
"artist": "Traditional",
"title": "For He's a Jolly Good Fellow",
"difficulty": "easy",
"input_style": "explicit name replacement instruction",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,4 +0,0 @@
For he's a jolly good fellow
For he's a jolly good fellow
For he's a jolly good fellow
Which nobody can deny
@@ -1,30 +0,0 @@
[
{
"id": "seg-00",
"start_time": 0.0,
"end_time": 2.5,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-01",
"start_time": 2.5,
"end_time": 5.0,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-02",
"start_time": 5.0,
"end_time": 7.5,
"text": "For he's a jolly good fellow",
"words": []
},
{
"id": "seg-03",
"start_time": 7.5,
"end_time": 10.0,
"text": "Which nobody can deny",
"words": []
}
]
@@ -1 +0,0 @@
We need this song to be about Mike and Sarah's wedding. Mike is a software engineer who loves climbing mountains, hiking on Saturdays with his border collie Max, and brewing his own beer in the basement. Sarah is a pediatric nurse with three sisters — Jenny, Tara, and Rachel — and she runs marathons every fall in the city park near her apartment. They met five years ago at a friend's birthday party in Brooklyn and bought their first house together last summer in Queens. Make the song celebrate their journey together. Highlight Max the dog if you can, and mention the marathons and the brewing.
@@ -1,11 +0,0 @@
{
"id": "row-row-line-surplus",
"artist": "Traditional",
"title": "Row, Row, Row Your Boat",
"difficulty": "medium",
"input_style": "client input far exceeds song's structural capacity (line surplus)",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,4 +0,0 @@
Row, row, row your boat,
Gently down the stream,
Merrily, merrily, merrily, merrily,
Life is but a dream.
@@ -1,30 +0,0 @@
[
{
"text": "Row, row, row your boat,",
"start_time": 0.0,
"end_time": 1.5,
"id": "rr-0",
"words": []
},
{
"text": "Gently down the stream,",
"start_time": 1.5,
"end_time": 3.0,
"id": "rr-1",
"words": []
},
{
"text": "Merrily, merrily, merrily, merrily,",
"start_time": 3.0,
"end_time": 4.5,
"id": "rr-2",
"words": []
},
{
"text": "Life is but a dream.",
"start_time": 4.5,
"end_time": 6.0,
"id": "rr-3",
"words": []
}
]
@@ -1 +0,0 @@
Hi! Can you change the word "star" to "Sam" throughout the song? Every time "star" appears, I want it to say "Sam" instead. It's for my son Sam's birthday party. Thanks!
@@ -1,11 +0,0 @@
{
"id": "twinkle-name-swap",
"artist": "Traditional",
"title": "Twinkle Twinkle Little Star",
"difficulty": "easy",
"input_style": "simple word substitution, well-defined target",
"settings_to_test": [
{"name": "default", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "balanced"},
{"name": "strict", "allow_reword": true, "allow_omit": true, "fixed_line_count": true, "strictness": "strict"}
]
}
@@ -1,6 +0,0 @@
Twinkle, twinkle, little star
How I wonder what you are
Up above the world so high
Like a diamond in the sky
Twinkle, twinkle, little star
How I wonder what you are
@@ -1,44 +0,0 @@
[
{
"id": "seg-00",
"start_time": 0.0,
"end_time": 2.0,
"text": "Twinkle, twinkle, little star",
"words": []
},
{
"id": "seg-01",
"start_time": 2.0,
"end_time": 4.0,
"text": "How I wonder what you are",
"words": []
},
{
"id": "seg-02",
"start_time": 4.0,
"end_time": 6.0,
"text": "Up above the world so high",
"words": []
},
{
"id": "seg-03",
"start_time": 6.0,
"end_time": 8.0,
"text": "Like a diamond in the sky",
"words": []
},
{
"id": "seg-04",
"start_time": 8.0,
"end_time": 10.0,
"text": "Twinkle, twinkle, little star",
"words": []
},
{
"id": "seg-05",
"start_time": 10.0,
"end_time": 12.0,
"text": "How I wonder what you are",
"words": []
}
]
@@ -1,68 +0,0 @@
We walk in the room, yeah we own this place
Anna hits the floor, watch her lead the way
Grace brings the rhythm, dancing every day
Oona on guitar, getting in the groove
Savannah steps forward, watch her make her move
Eirini singing, Frankie right there
Voices together, filling the air
Step aside now, nobody stands a chance
Mr Moulton jokes, we try not to laugh
But we’re ready now, we’ve done the graft
We’re gonna win, win, win, hear the crowd go loud
Singing all together, yeah we’re making you proud
Clap your hands, feel the beat, come and take a glance
We’re Year 5 strong, nobody stands a chance
Oh oh, here we go
Oh oh, steal the show
Watch us now, take our stance
Nobody stands a chance
Mr Bunney listening, nodding to the beat
Mr Woodward watching, tapping his feet
We’ll impress him with every note we play
Best performance of the whole school day
Mrs Woodbridge smiling, watching from the side
Thinking “Year 6 will be full of pride”
We’re stepping up, this is our dance
Better step back, nobody stands a chance
If we feel nervous, we just smile
We’ve been ready for a while
We’re gonna win, win, win, hear the crowd go loud
Singing all together, yeah we’re making you proud
Clap your hands, feel the beat, come and take a glance
We’re Year 5 strong, nobody stands a chance
Oh oh, here we go
Oh oh, steal the show
Watch us now, take our stance
Nobody stands a chance
Savannah solo
I step to the front, now it’s my time
Hear my voice, watch me shine
All eyes on me, feel that vibe
This is my moment, I come alive
Strum it strong, feel the sound
Let the music spin around
Keep the rhythm, don’t slow down
Now we bring it back around
We’re gonna win, win, win, loud and clear
Best group performance of the year
Sing it out, give it all, take a chance
We’re Year 5 stars, nobody stands a chance
Oh oh, this is it
Oh oh, we won’t quit
Watch us now, take our stance
Nobody stands a chance
@@ -1,31 +0,0 @@
{
"id": "year-5-stars-shake-it-off",
"artist": "Taylor Swift",
"title": "Year 5 Stars",
"source_job_id": "2cb49a45",
"difficulty": "hard",
"input_style": "long substantive lines, name-heavy, mismatched syllable budget",
"settings_to_test": [
{
"name": "default",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "balanced"
},
{
"name": "verbatim",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "verbatim"
},
{
"name": "strict",
"allow_reword": true,
"allow_omit": true,
"fixed_line_count": true,
"strictness": "strict"
}
]
}
@@ -1,70 +0,0 @@
We walk in the room
Yeah we own this place
Anna hits the floor, mm- mm
Watch her lead the way, mm- mm
Grace brings the rhythm
Dancing every day
Oona on guitar, mm- mm
Getting in the groove, mm- mm
Eirini singing, Frankie right there
Voices together, filling the air
Nobody stands a chance
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake, fake
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Mr Moulton jokes
We try not to laugh
But we’re ready now, mm- mm
We’ve done the graft, mm- mm
We’re gonna win (hear the crowd go loud)
Singing all together (yeah we’re making you proud)
Clap your hands, feel the beat, mm- mm
Come and take a glance, mm- mm
We’re Year 5 strong
Here we go, steal the show
Watch us now, take our stance
Nobody stands a chance
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake, fake
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
Hey, hey, hey
Just think, while you've been gettin' down and out about the liars
And the dirty, dirty cheats of the world
You could've been gettin' down to this sick beat
My ex- man brought his new girlfriend
She's like, "Oh my God"
But I'm just gonna shake
And to the fella over there with the hella good hair
Won't you come on over, baby? We can shake, shake, shake
Yeah, oh
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Heartbreakers gonna break, break, break, break, break
And the fakers gonna fake, fake, fake, fake
I'm just gonna shake, shake, shake, shake
I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
Shake it off, I shake it off
I- I- I shake it off, I shake it off
(You got to)
I- I- I shake it off, I shake it off
I- I- I shake it off, I shake it off
-109
View File
@@ -1,109 +0,0 @@
"""Markdown report generation for eval runs."""
from __future__ import annotations
import json
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
from backend.eval.custom_lyrics.runner import FixtureRunResult
from backend.eval.custom_lyrics.scorer import aggregate_corpus
def write_run_reports(
results: list[FixtureRunResult],
out_dir: Path,
baseline: Optional[dict] = None,
) -> None:
out_dir.mkdir(parents=True, exist_ok=True)
(out_dir / "per_fixture").mkdir(exist_ok=True)
summary_lines = ["# Custom Lyrics Eval Run", ""]
summary_lines.append(f"_Generated: {datetime.now(timezone.utc).isoformat()}_")
summary_lines.append("")
summary_lines.append("## Per-fixture summary")
summary_lines.append("")
summary_lines.append("| Fixture | Settings | pass@±2 | Δ | mean_Δ | iters | stop |")
summary_lines.append("|---------|----------|---------|---|--------|-------|------|")
for r in results:
s = r.score
delta_str = ""
if baseline:
prev = baseline.get(r.fixture_id, {}).get(r.settings_name, {})
if "pct_pass_at_2" in prev:
d = s.pct_pass_at_2 - prev["pct_pass_at_2"]
delta_str = f"{d:+.2f}"
summary_lines.append(
f"| {r.fixture_id} | {r.settings_name} | {s.pct_pass_at_2:.2f} | "
f"{delta_str} | {s.mean_delta:.2f} | {s.iterations_used} | {s.stop_reason} |"
)
summary_lines.append("")
corpus = aggregate_corpus([r.score for r in results])
summary_lines.append("## Corpus aggregate")
summary_lines.append("")
summary_lines.append(f"- Macro pass@±2: {corpus.get('macro_pct_pass_at_2', 0):.2f}")
summary_lines.append(f"- Macro pass@±1: {corpus.get('macro_pct_pass_at_1', 0):.2f}")
summary_lines.append(f"- Macro pass@±0: {corpus.get('macro_pct_pass_at_0', 0):.2f}")
summary_lines.append(f"- Total Gemini calls: {corpus.get('total_gemini_calls', 0)}")
(out_dir / "summary.md").write_text("\n".join(summary_lines))
for r in results:
per_dir = out_dir / "per_fixture" / f"{r.fixture_id}__{r.settings_name}"
per_dir.mkdir(parents=True, exist_ok=True)
(per_dir / "output.json").write_text(json.dumps({
"fixture_id": r.fixture_id,
"settings_name": r.settings_name,
"score": r.score.__dict__,
"lines": r.output_lines,
"line_metadata": r.line_metadata,
}, indent=2))
_write_per_fixture_md(per_dir, r)
def _write_per_fixture_md(out_dir: Path, r: FixtureRunResult) -> None:
lines = [
f"# {r.fixture_id} — {r.settings_name}",
"",
f"Stop reason: **{r.score.stop_reason}** after {r.score.iterations_used} iterations.",
f"Pass@±2: **{r.score.pct_pass_at_2:.0%}** ({sum(1 for v in r.line_metadata if v['min_delta']<=2)}/{r.score.line_count})",
"",
"| # | Original target | Candidate | Δ | Severity |",
"|---|------------------|-----------|---|----------|",
]
for v in r.line_metadata:
lines.append(
f"| {v['line_index']+1} | {_escape(v['target_text'])} | "
f"{_escape(v['candidate_text'])} | {v['min_delta']} | {v['severity']} |"
)
lines += [
"",
"## Your rating",
"",
"Add your qualitative notes here:",
"- [ ] Singable end-to-end",
"- [ ] Names placed naturally",
"- [ ] Stress patterns reasonable",
"- Notes:",
"",
]
(out_dir / "output.md").write_text("\n".join(lines))
def _escape(s: str) -> str:
return s.replace("|", "\\|").replace("\n", " ")
def write_baseline(results: list[FixtureRunResult], path: Path) -> None:
out: dict = {}
for r in results:
bucket = out.setdefault(r.fixture_id, {})
bucket[r.settings_name] = {
"pct_pass_at_2": r.score.pct_pass_at_2,
"pct_pass_at_0": r.score.pct_pass_at_0,
"mean_delta": r.score.mean_delta,
"iterations_used": r.score.iterations_used,
"captured_at": datetime.now(timezone.utc).date().isoformat(),
}
path.write_text(json.dumps(out, indent=2))
@@ -1,2 +0,0 @@
*
!.gitignore
-86
View File
@@ -1,86 +0,0 @@
"""CLI: python -m backend.eval.custom_lyrics.run"""
from __future__ import annotations
import argparse
import json
import sys
from datetime import datetime, timezone
from pathlib import Path
from backend.eval.custom_lyrics.cache import LlmCallCache
from backend.eval.custom_lyrics.report import write_baseline, write_run_reports
from backend.eval.custom_lyrics.runner import run_fixture
from backend.services.custom_lyrics.service import CustomLyricsService
EVAL_ROOT = Path(__file__).parent
FIXTURES_DIR = EVAL_ROOT / "fixtures"
RESULTS_DIR = EVAL_ROOT / "results"
CACHE_DIR = EVAL_ROOT / "cache"
BASELINE_PATH = EVAL_ROOT / "baseline.json"
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--fixtures", default="all", help="fixture id, or comma-separated, or 'all'")
parser.add_argument("--settings", default=None, help="settings name from metadata.settings_to_test")
parser.add_argument("--no-cache", action="store_true")
parser.add_argument("--replay-only", action="store_true")
parser.add_argument("--save-as-baseline", default=None,
help="path to write baseline JSON (e.g., baseline-post.json)")
args = parser.parse_args()
cache = LlmCallCache(
CACHE_DIR,
replay_only=args.replay_only,
) if not args.no_cache else _NullCache()
fixture_dirs = _select_fixtures(args.fixtures)
service = CustomLyricsService()
results = []
for fdir in fixture_dirs:
meta = json.loads((fdir / "metadata.json").read_text())
candidate_settings = meta.get("settings_to_test", [{
"name": "default",
"allow_reword": True,
"allow_omit": True,
"fixed_line_count": True,
"strictness": "balanced",
}])
for s in candidate_settings:
if args.settings and s["name"] != args.settings:
continue
settings_dict = {k: v for k, v in s.items() if k != "name"}
print(f"Running {fdir.name} / {s['name']}...", file=sys.stderr)
results.append(run_fixture(
fdir, s["name"], settings_dict,
cache=cache, service=service,
))
ts = datetime.now(timezone.utc).strftime("%Y-%m-%d-%H%M")
out_dir = RESULTS_DIR / ts
baseline = json.loads(BASELINE_PATH.read_text()) if BASELINE_PATH.exists() else None
write_run_reports(results, out_dir, baseline=baseline)
if args.save_as_baseline:
write_baseline(results, EVAL_ROOT / args.save_as_baseline)
print(f"Baseline saved to {args.save_as_baseline}", file=sys.stderr)
print(f"\nReport: {out_dir / 'summary.md'}", file=sys.stderr)
return 0
def _select_fixtures(spec: str) -> list[Path]:
if spec == "all":
return sorted(p for p in FIXTURES_DIR.iterdir() if p.is_dir() and (p / "metadata.json").exists())
ids = [s.strip() for s in spec.split(",")]
return [FIXTURES_DIR / i for i in ids]
class _NullCache:
def get(self, *args, **kwargs): return None
def set(self, *args, **kwargs): pass
if __name__ == "__main__":
sys.exit(main())
-112
View File
@@ -1,112 +0,0 @@
"""Eval runner: load fixture → call service (cached) → score → return result."""
from __future__ import annotations
import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from unittest.mock import patch
from backend.eval.custom_lyrics.cache import LlmCallCache
from backend.eval.custom_lyrics.scorer import FixtureScore, aggregate_fixture
from backend.services.custom_lyrics.service import CustomLyricsService
from backend.services.custom_lyrics.settings import (
GenerationSettings,
settings_from_dict,
)
@dataclass
class FixtureRunResult:
fixture_id: str
settings_name: str
score: FixtureScore
output_lines: list[str]
line_metadata: list[dict[str, Any]]
fixture_dir: Path
def load_fixture(fixture_dir: Path) -> dict:
metadata = json.loads((fixture_dir / "metadata.json").read_text())
metadata["original_lyrics"] = (fixture_dir / "original_lyrics.txt").read_text().strip().splitlines()
metadata["original_segments"] = json.loads((fixture_dir / "original_segments.json").read_text())
metadata["client_input"] = (fixture_dir / "client_input.txt").read_text()
notes_path = fixture_dir / "notes.txt"
metadata["notes"] = notes_path.read_text() if notes_path.exists() else None
return metadata
def run_fixture(
fixture_dir: Path,
settings_name: str,
settings_dict: dict[str, Any],
*,
cache: LlmCallCache,
service: CustomLyricsService,
) -> FixtureRunResult:
fixture = load_fixture(fixture_dir)
settings = settings_from_dict(settings_dict)
gemini_call_count = 0
real_call = service._call_gemini
def cached_call(*, system_prompt: str, user_prompt: str, pdf_bytes, settings: GenerationSettings):
nonlocal gemini_call_count
cached = cache.get(
system_prompt, user_prompt,
model=service.settings.custom_lyrics_model,
settings_dict=settings.to_dict(),
)
if cached is not None:
return cached
gemini_call_count += 1
result = real_call(
system_prompt=system_prompt, user_prompt=user_prompt,
pdf_bytes=pdf_bytes, settings=settings,
)
cache.set(
system_prompt, user_prompt,
model=service.settings.custom_lyrics_model,
settings_dict=settings.to_dict(),
response_lines=result,
)
return result
with patch.object(service, "_call_gemini", side_effect=cached_call):
result = service.generate(
job_id=fixture["id"],
target_lines=fixture["original_lyrics"],
target_segments=fixture["original_segments"],
artist=fixture.get("artist"),
title=fixture.get("title"),
custom_text=fixture["client_input"],
file_bytes=None, file_mime=None, file_name=None,
notes=fixture.get("notes"),
settings=settings,
)
score = aggregate_fixture(
fixture_id=fixture["id"],
settings_name=settings_name,
metadata=result.line_metadata,
iterations_used=result.iterations_used,
stop_reason=result.stop_reason.value,
duration_ms=result.duration_ms,
gemini_calls=gemini_call_count,
line_count_match=not result.line_count_mismatch,
)
return FixtureRunResult(
fixture_id=fixture["id"],
settings_name=settings_name,
score=score,
output_lines=result.lines,
line_metadata=[{
"line_index": v.line_index,
"target_text": v.target_text,
"candidate_text": v.candidate_text,
"min_delta": v.min_delta,
"severity": v.severity.value,
} for v in result.line_metadata],
fixture_dir=fixture_dir,
)
-113
View File
@@ -1,113 +0,0 @@
"""Pure scoring functions for custom-lyrics eval."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Iterable
from backend.services.custom_lyrics.validator import LineValidation
@dataclass
class PerLineScore:
line_index: int
min_delta: int
pass_at_0: bool
pass_at_1: bool
pass_at_2: bool
pass_at_4: bool
severity: str
@dataclass
class FixtureScore:
fixture_id: str
settings_name: str
line_count: int
pct_pass_at_2: float
pct_pass_at_1: float
pct_pass_at_0: float
pct_pass_at_4: float
mean_delta: float
median_delta: float
max_delta: int
iterations_used: int
stop_reason: str
duration_ms: int
gemini_calls: int
line_count_match: bool
severity_breakdown: dict[str, int]
def score_per_line(val: LineValidation) -> PerLineScore:
return PerLineScore(
line_index=val.line_index,
min_delta=val.min_delta,
pass_at_0=val.min_delta <= 0,
pass_at_1=val.min_delta <= 1,
pass_at_2=val.min_delta <= 2,
pass_at_4=val.min_delta <= 4,
severity=val.severity.value,
)
def aggregate_fixture(
*,
fixture_id: str,
settings_name: str,
metadata: list[LineValidation],
iterations_used: int,
stop_reason: str,
duration_ms: int,
gemini_calls: int,
line_count_match: bool,
) -> FixtureScore:
n = len(metadata)
if n == 0:
return FixtureScore(
fixture_id=fixture_id, settings_name=settings_name,
line_count=0,
pct_pass_at_2=0, pct_pass_at_1=0, pct_pass_at_0=0, pct_pass_at_4=0,
mean_delta=0, median_delta=0, max_delta=0,
iterations_used=iterations_used, stop_reason=stop_reason,
duration_ms=duration_ms, gemini_calls=gemini_calls,
line_count_match=line_count_match,
severity_breakdown={"ok": 0, "minor": 0, "major": 0},
)
deltas = sorted(v.min_delta for v in metadata)
severity_breakdown = {"ok": 0, "minor": 0, "major": 0}
for v in metadata:
severity_breakdown[v.severity.value] += 1
return FixtureScore(
fixture_id=fixture_id,
settings_name=settings_name,
line_count=n,
pct_pass_at_2=sum(1 for d in deltas if d <= 2) / n,
pct_pass_at_1=sum(1 for d in deltas if d <= 1) / n,
pct_pass_at_0=sum(1 for d in deltas if d <= 0) / n,
pct_pass_at_4=sum(1 for d in deltas if d <= 4) / n,
mean_delta=sum(deltas) / n,
median_delta=deltas[n // 2],
max_delta=deltas[-1],
iterations_used=iterations_used,
stop_reason=stop_reason,
duration_ms=duration_ms,
gemini_calls=gemini_calls,
line_count_match=line_count_match,
severity_breakdown=severity_breakdown,
)
def aggregate_corpus(per_fixture: Iterable[FixtureScore]) -> dict:
"""Macro-averaged corpus aggregate."""
fixtures = list(per_fixture)
if not fixtures:
return {}
return {
"fixture_count": len(fixtures),
"macro_pct_pass_at_2": sum(f.pct_pass_at_2 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_1": sum(f.pct_pass_at_1 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_0": sum(f.pct_pass_at_0 for f in fixtures) / len(fixtures),
"macro_pct_pass_at_4": sum(f.pct_pass_at_4 for f in fixtures) / len(fixtures),
"macro_mean_delta": sum(f.mean_delta for f in fixtures) / len(fixtures),
"total_gemini_calls": sum(f.gemini_calls for f in fixtures),
}
@@ -1,38 +0,0 @@
"""Tests for the LLM-call disk cache."""
from __future__ import annotations
import tempfile
from pathlib import Path
from backend.eval.custom_lyrics.cache import LlmCallCache
def test_miss_returns_none() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
assert c.get("system", "user", model="m1", settings_dict={"x": 1}) is None
def test_set_then_get_hits() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
c.set("system", "user", model="m1", settings_dict={"x": 1}, response_lines=["a", "b"])
assert c.get("system", "user", model="m1", settings_dict={"x": 1}) == ["a", "b"]
def test_different_settings_different_keys() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp))
c.set("system", "user", model="m1", settings_dict={"x": 1}, response_lines=["a"])
assert c.get("system", "user", model="m1", settings_dict={"x": 2}) is None
def test_replay_only_raises_on_miss() -> None:
with tempfile.TemporaryDirectory() as tmp:
c = LlmCallCache(Path(tmp), replay_only=True)
try:
c.get("system", "user", model="m1", settings_dict={"x": 1})
except RuntimeError as e:
assert "cache miss" in str(e).lower()
else:
raise AssertionError("expected RuntimeError")
@@ -1,66 +0,0 @@
"""Tests for eval scorer."""
from __future__ import annotations
import pytest
from backend.eval.custom_lyrics.scorer import (
PerLineScore,
FixtureScore,
score_per_line,
aggregate_fixture,
)
from backend.services.custom_lyrics.validator import LineValidation, Severity
def _line_val(idx: int, target: list[int], cand: list[int], passes: bool, severity: Severity, delta: int) -> LineValidation:
return LineValidation(
line_index=idx,
target_text=f"t{idx}",
candidate_text=f"c{idx}",
target_syllables=target,
candidate_syllables=cand,
min_delta=delta,
passes=passes,
severity=severity,
time_budget_seconds=1.0,
)
def test_score_per_line_pass_fail_thresholds() -> None:
val = _line_val(0, [5, 5, 5, 5], [6, 6, 6, 6], passes=False, severity=Severity.MINOR, delta=1)
s = score_per_line(val)
assert s.pass_at_0 is False
assert s.pass_at_1 is True
assert s.pass_at_2 is True
assert s.pass_at_4 is True
assert s.min_delta == 1
def test_score_per_line_major_violation() -> None:
val = _line_val(0, [5, 5, 5, 5], [12, 12, 12, 12], passes=False, severity=Severity.MAJOR, delta=7)
s = score_per_line(val)
assert s.pass_at_0 is False
assert s.pass_at_4 is False
assert s.severity == "major"
def test_aggregate_fixture_basic() -> None:
metadata = [
_line_val(0, [5]*4, [5]*4, passes=True, severity=Severity.OK, delta=0),
_line_val(1, [5]*4, [6]*4, passes=False, severity=Severity.MINOR, delta=1),
_line_val(2, [5]*4, [9]*4, passes=False, severity=Severity.MAJOR, delta=4),
]
fixture = aggregate_fixture(
fixture_id="x",
settings_name="default",
metadata=metadata,
iterations_used=2,
stop_reason="max_iters_reached",
duration_ms=1500,
gemini_calls=3,
line_count_match=True,
)
assert fixture.pct_pass_at_2 == pytest.approx(2 / 3)
assert fixture.pct_pass_at_0 == pytest.approx(1 / 3)
assert fixture.mean_delta == pytest.approx((0 + 1 + 4) / 3)
assert fixture.max_delta == 4
-94
View File
@@ -1,94 +0,0 @@
"""
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for validation errors and credit checks.
"""
class EmailValidationError(Exception):
"""
Raised when email validation fails.
Attributes:
message: Human-readable error message
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
"""
def __init__(self, message: str, reason: str = "invalid"):
self.message = message
self.reason = reason
super().__init__(message)
class IPBlockedError(Exception):
"""
Raised when a request comes from a blocked IP address.
Attributes:
message: Human-readable error message
ip_address: The blocked IP address (may be partially masked)
"""
def __init__(self, message: str, ip_address: str = ""):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
-99
View File
@@ -1,99 +0,0 @@
"""
Lightweight i18n utility for backend translations.
Usage:
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
# In a route handler:
locale = get_locale_from_request(request)
message = t(locale, "auth.userNotFound")
message = t(locale, "jobs.notFound", job_id="abc123")
"""
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from starlette.requests import Request
SUPPORTED_LOCALES = ("en", "es", "de")
DEFAULT_LOCALE = "en"
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
def _load_translations(locale: str) -> dict[str, Any]:
"""Load and cache translation file for a locale."""
path = TRANSLATIONS_DIR / f"{locale}.json"
if not path.exists():
if locale != DEFAULT_LOCALE:
return _load_translations(DEFAULT_LOCALE)
return {}
with open(path, encoding="utf-8") as f:
result: dict[str, Any] = json.load(f)
return result
def t(locale: str, key: str, **kwargs: Any) -> str:
"""Get a translated string by dotted key path.
Args:
locale: Language code (en, es, de)
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
**kwargs: Interpolation variables (e.g., job_id="abc123")
Returns:
Translated string with variables interpolated, or the key itself if not found.
"""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
translations = _load_translations(locale)
# Walk the dotted path
value: Any = translations
for part in key.split("."):
if isinstance(value, dict) and part in value:
value = value[part]
else:
# Key not found — fall back to English
if locale != DEFAULT_LOCALE:
return t(DEFAULT_LOCALE, key, **kwargs)
return key # Return the key itself as last resort
if not isinstance(value, str):
return key
result: str = value
# Interpolate variables
if kwargs:
try:
result = result.format(**kwargs)
except KeyError:
pass # Missing variable — return template as-is
return result
def get_locale_from_request(request: Request) -> str:
"""Extract locale from Accept-Language header.
Returns the first supported locale found, or DEFAULT_LOCALE.
"""
accept_lang = request.headers.get("accept-language", "")
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
for part in accept_lang.split(","):
lang = part.split(";")[0].strip().split("-")[0].lower()
if lang in SUPPORTED_LOCALES:
return lang
return DEFAULT_LOCALE
def get_locale_prefix(locale: str) -> str:
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
return f"/{locale}"
+6 -111
View File
@@ -1,8 +1,5 @@
"""
FastAPI application entry point for karaoke generation backend.
(Trivial comment touch to trigger backend CI on ephemeral runners after
the 2026-05-17 dispatcher e2 fix; safe to remove on next backend edit.)
"""
import logging
from contextlib import asynccontextmanager
@@ -10,16 +7,10 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker, client_errors, bulk, parse_titles
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
from backend.services.nltk_preloader import preload_all_nltk_resources
from backend.services.langfuse_preloader import preload_langfuse_handler
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware
from backend.middleware.edge_auth import EdgeAuthMiddleware
from backend.workers.registry import worker_registry
from backend.version import VERSION
@@ -76,28 +67,7 @@ async def lifespan(app: FastAPI):
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Preload NLP models and resources to avoid cold start delays
# See docs/archive/2026-01-08-performance-investigation.md for background
# 1. SpaCy model (60+ second delay without preload)
try:
preload_spacy_model("en_core_web_sm")
except Exception as e:
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
# 2. NLTK cmudict (50-100+ second delay without preload)
try:
preload_all_nltk_resources()
except Exception as e:
logger.warning(f"NLTK preload failed (will load lazily): {e}")
# 3. Langfuse callback handler (200+ second delay without preload)
try:
preload_langfuse_handler()
except Exception as e:
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
# Validate OAuth credentials (non-blocking)
try:
await validate_credentials_on_startup()
@@ -105,42 +75,8 @@ async def lifespan(app: FastAPI):
logger.error(f"Credential validation failed: {e}")
yield
# Shutdown - wait for any active workers to complete before terminating
# This prevents Cloud Run from killing workers mid-processing.
#
# Cloud Run gen2 default termination grace period is 600s. We wait up to
# 480s, leaving ~120s for the cleanup pass below to write Firestore state
# before SIGKILL. Without that headroom, render jobs that miss the wait
# would sit in `rendering_video` forever waiting for an operator.
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 480s)...")
completed = await worker_registry.wait_for_completion(timeout=480)
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
f"Remaining workers: {worker_registry.get_active_workers()}"
)
# Park any still-active render jobs so the auto-retry scheduler can
# recover them. Safe to call unconditionally — it's a no-op if nothing
# render-related is in flight.
try:
from backend.workers.render_video_worker import park_active_render_jobs_for_shutdown
parked = park_active_render_jobs_for_shutdown()
if parked:
logger.warning(
f"Parked {parked} in-flight render job(s) for auto-retry "
f"due to shutdown"
)
except Exception as exc:
logger.error(f"Render-job shutdown park failed: {exc!r}")
else:
logger.info("No active workers, proceeding with shutdown")
# Shutdown
logger.info("Shutting down karaoke generation backend")
@@ -159,8 +95,8 @@ if tracing_enabled:
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@@ -168,58 +104,17 @@ app.add_middleware(
# Add audit logging middleware (captures all requests with request_id for correlation)
app.add_middleware(AuditLoggingMiddleware)
# Add tenant detection middleware (extracts tenant from subdomain/headers)
app.add_middleware(TenantMiddleware)
# Edge origin lock (added last = outermost, runs first). Rejects direct-to-origin
# requests that bypassed the Cloudflare edge, before tenant/audit middleware do
# any work. No-op unless EDGE_AUTH_MODE is warn/enforce. See
# backend/middleware/edge_auth.py.
app.add_middleware(EdgeAuthMiddleware)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(parse_titles.router, prefix="/api") # kjbox karaoke-filename parser
app.include_router(bulk.router, prefix="/api") # Bulk Mode: multi-job submission
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
# Exception handlers
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
return JSONResponse(
status_code=402,
content={
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
},
)
@app.get("/")
+1 -7
View File
@@ -1,11 +1,5 @@
"""Middleware package for FastAPI application."""
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
__all__ = [
"AuditLoggingMiddleware",
"TenantMiddleware",
"get_tenant_from_request",
"get_tenant_config_from_request",
]
__all__ = ["AuditLoggingMiddleware"]
-115
View File
@@ -1,115 +0,0 @@
"""
Edge origin-lock middleware.
Cloudflare fronts ``api.nomadkaraoke.com`` (WAF + rate limiting). To stop
attackers from bypassing the edge by hitting the Cloud Run origin directly
(``*.run.app`` / ``ghs.googlehosted.com``), Cloudflare injects a shared-secret
header (``X-Edge-Auth``) on every proxied request via a Transform Rule. This
middleware rejects public requests that lack the correct header.
A Cloud Run *domain mapping* can't use ``ingress=internal-and-cloud-load-balancing``
(that needs a load balancer), so this app-layer check is how we lock the origin.
Modes (env ``EDGE_AUTH_MODE``, default ``off``):
off - passthrough, no checks (deploy-dormant; behaviour unchanged).
warn - log requests missing/failing the header, but ALLOW them. Use to
confirm nothing legit is missing the header before enforcing.
enforce - reject requests missing/failing the header with 403.
Secret: env ``EDGE_ORIGIN_SECRET`` (same value Cloudflare injects). If enforce
mode is set but the secret is unconfigured, the middleware FAILS CLOSED (503) —
a security control must not silently allow direct-origin traffic. (Low risk:
Cloud Run won't start a revision whose secret ref is missing, and enforce is
only enabled after staging confirms the secret.) In warn mode it still allows.
Exemptions (always allowed, even in enforce mode):
* Health / root endpoints and Cloud Run probes, which hit the origin
directly (not via Cloudflare) and legitimately lack the header.
"""
import hmac
import logging
import os
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import JSONResponse, Response
logger = logging.getLogger(__name__)
# Header Cloudflare injects (must match CloudflareConfig.ORIGIN_AUTH_HEADER in
# infrastructure/modules/edge_security.py).
EDGE_AUTH_HEADER = "X-Edge-Auth"
# Paths always allowed without the edge header. These are non-sensitive and are
# probed directly on the origin by Cloud Run startup/liveness checks and GCP
# uptime monitoring, which don't traverse Cloudflare.
_EXEMPT_EXACT = {"/", "/api/health", "/api/health/"}
_EXEMPT_PREFIXES = ("/api/health/",)
_VALID_MODES = {"off", "warn", "enforce"}
def _mode() -> str:
mode = os.environ.get("EDGE_AUTH_MODE", "off").lower().strip()
return mode if mode in _VALID_MODES else "off"
def _is_exempt(path: str) -> bool:
if path in _EXEMPT_EXACT:
return True
return any(path.startswith(p) for p in _EXEMPT_PREFIXES)
class EdgeAuthMiddleware(BaseHTTPMiddleware):
"""Reject origin requests that didn't come through the Cloudflare edge."""
async def dispatch(self, request: Request, call_next) -> Response:
mode = _mode()
if mode == "off":
return await call_next(request)
if _is_exempt(request.url.path):
return await call_next(request)
secret = os.environ.get("EDGE_ORIGIN_SECRET", "")
if not secret:
# Misconfiguration: enforce is requested but no secret is wired up.
# In enforce mode, fail CLOSED (503) — a security control must not
# silently allow direct-origin traffic when it can't do its job.
# This is low-risk operationally: Cloud Run won't even start a
# revision whose --set-secrets ref is missing, and enforce is only
# turned on after staging confirms the secret. In warn mode we still
# allow (observational only).
logger.error(
"EDGE_AUTH_MODE=%s but EDGE_ORIGIN_SECRET is unset.", mode,
)
if mode == "enforce":
return JSONResponse(
status_code=503,
content={"detail": "Origin authentication unavailable"},
)
return await call_next(request)
provided = request.headers.get(EDGE_AUTH_HEADER, "")
# Constant-time comparison to avoid leaking the secret via timing.
ok = bool(provided) and hmac.compare_digest(provided, secret)
if ok:
return await call_next(request)
client = request.client.host if request.client else "?"
if mode == "warn":
logger.warning(
"edge-auth: request missing/invalid %s header (warn mode, "
"allowing). path=%s ip=%s",
EDGE_AUTH_HEADER, request.url.path, client,
)
return await call_next(request)
# enforce
logger.warning(
"edge-auth: blocked direct-to-origin request. path=%s ip=%s",
request.url.path, client,
)
return JSONResponse(status_code=403, content={"detail": "Forbidden"})
-192
View File
@@ -1,192 +0,0 @@
"""
Tenant detection middleware for white-label B2B portals.
This middleware extracts the tenant ID from incoming requests and attaches
the tenant configuration to the request state for use by downstream handlers.
Tenant detection priority:
1. X-Tenant-ID header (explicitly set by frontend)
2. `tenant` query parameter (for development/testing)
3. Host header subdomain detection (production flow)
The middleware is non-blocking - if no tenant is detected, the request
proceeds as a default Nomad Karaoke request (tenant_id = None).
"""
import logging
import os
from typing import Optional
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from backend.services.tenant_service import get_tenant_service
from backend.models.tenant import TenantConfig
logger = logging.getLogger(__name__)
# Only allow query param tenant override in non-production environments
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
os.environ.get("ENVIRONMENT", "").lower() == "production"
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
class TenantMiddleware(BaseHTTPMiddleware):
"""
Middleware that detects tenant from request and attaches config to state.
After this middleware runs, routes can access:
- request.state.tenant_id: str or None
- request.state.tenant_config: TenantConfig or None
Usage in route handlers:
```python
@router.get("/something")
async def handler(request: Request):
tenant_id = getattr(request.state, "tenant_id", None)
tenant_config = getattr(request.state, "tenant_config", None)
```
"""
async def dispatch(self, request: Request, call_next) -> Response:
# Extract tenant ID from request
tenant_id = self._extract_tenant_id(request)
# Load tenant config if tenant detected
tenant_config: Optional[TenantConfig] = None
if tenant_id:
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(tenant_id)
if tenant_config and not tenant_config.is_active:
# Tenant exists but is inactive - treat as default
logger.warning(f"Inactive tenant requested: {tenant_id}")
tenant_id = None
tenant_config = None
# Attach to request state
request.state.tenant_id = tenant_id
request.state.tenant_config = tenant_config
# Log tenant detection for debugging
if tenant_id:
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
# Process request
response = await call_next(request)
# Add tenant ID to response headers for debugging
if tenant_id:
response.headers["X-Tenant-ID"] = tenant_id
return response
def _extract_tenant_id(self, request: Request) -> Optional[str]:
"""
Extract tenant ID from request using priority-based detection.
Returns:
Tenant ID if detected, None for default Nomad Karaoke
"""
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
header_tenant = request.headers.get("X-Tenant-ID")
if header_tenant:
return header_tenant.lower().strip()
# Priority 2: Query parameter (for development/testing only)
# Disabled in production to prevent tenant spoofing
if not IS_PRODUCTION:
query_tenant = request.query_params.get("tenant")
if query_tenant:
return query_tenant.lower().strip()
# Priority 3: Host header subdomain detection
host = request.headers.get("Host", "")
return self._extract_tenant_from_host(host)
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
"""
Extract tenant ID from Host header subdomain.
Patterns handled:
- vocalstar.nomadkaraoke.com -> vocalstar
- vocalstar.gen.nomadkaraoke.com -> vocalstar
- localhost:3000 -> None (local dev)
- gen.nomadkaraoke.com -> None (main app)
- api.nomadkaraoke.com -> None (API)
Returns:
Tenant ID if subdomain matches tenant, None otherwise
"""
if not host:
return None
# Normalize
host_lower = host.lower()
# Remove port if present
if ":" in host_lower:
host_lower = host_lower.split(":")[0]
# Check if this is a nomadkaraoke.com domain
if "nomadkaraoke.com" not in host_lower:
return None
# Split into parts
parts = host_lower.split(".")
# Need at least 3 parts: subdomain.nomadkaraoke.com
if len(parts) < 3:
return None
# First part is the potential tenant ID
potential_tenant = parts[0]
# Skip known non-tenant subdomains
if potential_tenant in NON_TENANT_SUBDOMAINS:
return None
# Verify tenant exists in GCS
tenant_service = get_tenant_service()
if tenant_service.tenant_exists(potential_tenant):
return potential_tenant
return None
def get_tenant_from_request(request: Request) -> Optional[str]:
"""
Helper function to get tenant ID from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_from_request
@router.get("/something")
async def handler(request: Request):
tenant_id = get_tenant_from_request(request)
```
"""
return getattr(request.state, "tenant_id", None)
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
"""
Helper function to get tenant config from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_config_from_request
@router.get("/something")
async def handler(request: Request):
config = get_tenant_config_from_request(request)
if config and not config.features.audio_search:
raise HTTPException(403, "Audio search not available")
```
"""
return getattr(request.state, "tenant_config", None)
-110
View File
@@ -1,110 +0,0 @@
"""
Audio edit session model for persisting input audio editing progress.
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class AudioEditSessionSummary:
"""Compact summary of operations in an audio edit session, for list views."""
total_operations: int = 0
operations_breakdown: Dict[str, int] = field(default_factory=dict)
duration_change_seconds: float = 0.0
net_duration_seconds: float = 0.0
def to_dict(self) -> Dict[str, Any]:
return {
"total_operations": self.total_operations,
"operations_breakdown": self.operations_breakdown,
"duration_change_seconds": self.duration_change_seconds,
"net_duration_seconds": self.net_duration_seconds,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
return cls(
total_operations=data.get("total_operations", 0),
operations_breakdown=data.get("operations_breakdown", {}),
duration_change_seconds=data.get("duration_change_seconds", 0.0),
net_duration_seconds=data.get("net_duration_seconds", 0.0),
)
@dataclass
class AudioEditSession:
"""
A saved snapshot of an audio editing session.
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "manual" | "submit"
audio_duration_seconds: Optional[float] = None
original_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
edit_data_gcs_path: str = ""
# Hash of the edit data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"original_duration_seconds": self.original_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"edit_data_gcs_path": self.edit_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
original_duration_seconds=data.get("original_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+58 -220
View File
@@ -9,75 +9,58 @@ from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, validator
from karaoke_gen.utils import normalize_text
class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 8 main stages with up to 3 human interaction points:
The workflow has 8 main stages with 3 human interaction points:
1. Input & Setup (may include audio source selection)
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
3. Title/End Screen Generation
4. Countdown Padding Synchronization
5. Combined Human Review (BLOCKING) - lyrics review + instrumental selection
6. Video Finalization
7. Distribution
Note: As of 2026-01, lyrics review and instrumental selection are combined
into a single human review step. The user selects their instrumental
during the lyrics review phase, before the video is rendered.
5. Human Review (BLOCKING)
6. Instrumental Selection (BLOCKING)
7. Video Finalization
8. Distribution
"""
# Initial states
PENDING = "pending" # Job created, queued for processing
# Audio search states (for artist+title search mode)
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
DOWNLOAD_PENDING_RETRY = "download_pending_retry" # Download hit a transient issue (few/no seeders, tracker blip); auto-retrying for up to 24h
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Optional: Audio editing (before processing begins)
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
# Optional: duration cost confirmation (BLOCKING) - confirm credits before heavy processing
AWAITING_DURATION_CONFIRM = "awaiting_duration_confirm" # ⚠️ WAITING FOR USER - confirm duration-based cost
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
# Stage 2b: Lyrics processing (parallel track 2)
TRANSCRIBING = "transcribing" # AudioShake API transcription
CORRECTING = "correcting" # Automatic lyrics correction
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
# Stage 3: Title/End screens + Backing Vocals Analysis
# Stage 3: Title/End screens
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
# Stage 4: Countdown padding (automatic)
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
# Stage 5: Combined Human Review (BLOCKING) - lyrics + instrumental selection
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - combined review needed
IN_REVIEW = "in_review" # User is actively reviewing
REVIEW_COMPLETE = "review_complete" # User submitted lyrics + instrumental selection
# Stage 5: Human review (BLOCKING)
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
IN_REVIEW = "in_review" # User is actively reviewing lyrics
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
# Stage 5.5: Render video with corrected lyrics (post-review)
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
RENDER_PENDING_CAPACITY = "render_pending_capacity" # Waiting for GCE encoding capacity (auto-retry)
# Stage 6: Instrumental already selected (during combined review)
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # LEGACY - no longer used
INSTRUMENTAL_SELECTED = "instrumental_selected" # Instrumental was selected during combined review
# Stage 6: Instrumental selection (BLOCKING)
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
# Stage 7: Video generation and finalization
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
@@ -104,112 +87,71 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.DOWNLOAD_PENDING_RETRY, JobStatus.FAILED],
# DOWNLOAD_PENDING_RETRY = parked when a torrent download hits a transient
# issue (few/no seeders, tracker outage); auto-retried by the recover-stuck
# cron for up to 24h before failing permanently.
JobStatus.DOWNLOAD_PENDING_RETRY: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
# Can also enter optional audio edit phase if requires_audio_edit flag is set
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.FAILED],
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_DURATION_CONFIRM, JobStatus.FAILED],
# Duration cost confirmation (BLOCKING) - user confirms before heavy processing begins
JobStatus.AWAITING_DURATION_CONFIRM: [
JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING,
JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT,
JobStatus.FAILED, JobStatus.CANCELLED,
],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Lyrics flow
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Post-parallel processing (screens + backing vocals analysis)
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Post-parallel processing
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Combined human review flow (lyrics + instrumental selection)
# Human review flow
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [
JobStatus.RENDERING_VIDEO,
JobStatus.PREP_COMPLETE, # for prep-only jobs
JobStatus.RENDER_PENDING_CAPACITY, # GCE capacity exhausted before render started
JobStatus.FAILED,
],
# Video rendering (post-review) - instrumental was already selected during combined review
# RENDER_PENDING_CAPACITY = parked when GCE encoding capacity is exhausted; auto-retried by scheduler.
JobStatus.RENDERING_VIDEO: [
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.PREP_COMPLETE,
JobStatus.RENDER_PENDING_CAPACITY,
JobStatus.FAILED,
],
JobStatus.RENDER_PENDING_CAPACITY: [
JobStatus.RENDERING_VIDEO, # auto-retry succeeded, picked up again
JobStatus.REVIEW_COMPLETE, # admin / scheduler resets to re-trigger render
JobStatus.FAILED, # exceeded max wait or operator gave up
JobStatus.CANCELLED, # user cancelled while waiting
],
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
# Video rendering (post-review)
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# Instrumental selection flow
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
# Video generation flow
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
# Distribution flow
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
# PREP_COMPLETE allows finalise-only continuation
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
JobStatus.RENDER_PENDING_CAPACITY, # Operator parks a stale-failed job for auto-retry
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
],
# Legacy states (for backward compatibility)
JobStatus.QUEUED: [JobStatus.PENDING],
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
@@ -225,7 +167,6 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
@@ -255,14 +196,7 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -271,9 +205,6 @@ class Job(BaseModel):
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
tenant_id: str = "" # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -343,31 +274,7 @@ class Job(BaseModel):
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Creation-time decision params (for observability — what did the user actually choose?)
creation_params: Dict[str, Any] = Field(default_factory=dict)
# Anti-abuse: IP address at job creation time (for cross-account correlation)
creation_ip: Optional[str] = None
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
credit_refunded: bool = False
# Edit tracking (how many times a completed track has been re-edited)
edit_count: int = 0
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -458,43 +365,6 @@ class Job(BaseModel):
}
"""
# Processing metadata (immutable provenance data written by workers during processing)
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Immutable processing provenance data. Written once by each worker during processing.
Never modified after initial write. Used for analytics, debugging, and data export.
Structure:
{
"transcription": {
"provider": "audioshake",
"audioshake_task_id": "...",
"audioshake_asset_id": "...",
"language_detected": "en",
"word_count": 358,
"segment_count": 42,
},
"correction": {
"handlers_applied": [...],
"corrections_made": 12,
"correction_ratio": 0.034,
"reference_sources_found": ["genius", "spotify"],
"agentic_routing": "rule-based",
"duration_seconds": 33.2,
},
"separation": {
"provider": "modal",
"clean_model": "...",
"duration_seconds": 420.5,
},
"timing": {
"lyrics_worker_seconds": 125.3,
"audio_worker_seconds": 420.5,
...
},
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -602,20 +472,7 @@ class JobCreate(BaseModel):
prep_only: bool = False # Stop after review, don't run finalisation
finalise_only: bool = False # Skip prep, run only finalisation
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Credit pricing: how many credits to charge for this job at creation time.
# Search and upload jobs pass the default (1) and may top up later.
# URL jobs with a known duration pass ceil(duration/600) credits.
credits: int = 1
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -627,33 +484,14 @@ class JobCreate(BaseModel):
- server_version: Current server version
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
tenant_id: str = "" # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
"""Validate URL is not empty."""
@validator('url', 'artist', 'title')
def validate_inputs(cls, v):
"""Validate string inputs are not empty."""
if v is not None and isinstance(v, str) and not v.strip():
raise ValueError("Field cannot be empty string")
return v.strip() if isinstance(v, str) else v
@validator('artist', 'title')
def normalize_artist_title(cls, v):
"""Normalize artist/title text to standardize Unicode characters.
This ensures consistent data storage by converting:
- Curly quotes -> straight quotes
- Various dashes -> hyphen
- Unusual whitespace -> regular space
"""
if v is not None and isinstance(v, str):
if not v.strip():
raise ValueError("Field cannot be empty string")
# normalize_text handles stripping and Unicode normalization
return normalize_text(v)
return v
class JobResponse(BaseModel):
"""Job response model."""
-178
View File
@@ -1,178 +0,0 @@
"""
Referral system models.
Supports:
- Referral link creation and management
- Earnings tracking from referred purchases
- Payout processing via Stripe Connect
- API request/response shapes for referral endpoints
"""
from datetime import datetime
from typing import Optional, List
from pydantic import BaseModel, Field
# ============================================================================
# Firestore Document Models
# ============================================================================
class ReferralLinkStats(BaseModel):
"""Aggregated statistics for a referral link."""
clicks: int = 0
signups: int = 0
purchases: int = 0
total_earned_cents: int = 0
class ReferralLink(BaseModel):
"""
Referral link stored in Firestore.
Each user can have one referral link. Vanity links use a custom code
(e.g., "karaoke-king") while standard links use a generated code.
"""
code: str # Unique referral code (URL slug)
owner_email: str # Email of the referrer
display_name: Optional[str] = None # Public name shown on interstitial
custom_message: Optional[str] = None # Optional message on interstitial
# Reward configuration
discount_percent: int = 10 # Discount for referred user
kickback_percent: int = 20 # Earnings for referrer
discount_duration_days: int = 30 # How long the discount lasts
earning_duration_days: int = 365 # How long referrer earns from this link
# Stripe
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
# Flags
is_vanity: bool = False # Custom vanity code vs generated
enabled: bool = True # Can be disabled without deleting
# Timestamps
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
# Aggregated stats
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
class ReferralEarning(BaseModel):
"""
Record of a single earning from a referred purchase.
Created when a referred user completes a Stripe checkout.
"""
id: str
referrer_email: str
referred_email: str
referral_code: str
stripe_session_id: str
purchase_amount_cents: int
earning_amount_cents: int
status: str = "pending" # pending, paid, cancelled
created_at: datetime = Field(default_factory=datetime.utcnow)
paid_at: Optional[datetime] = None
payout_id: Optional[str] = None
class ReferralPayout(BaseModel):
"""
Record of a payout to a referrer via Stripe Connect.
Groups multiple earnings into a single transfer.
"""
id: str
referrer_email: str
stripe_transfer_id: str
amount_cents: int
earnings_included: List[str] # List of ReferralEarning IDs
status: str = "processing" # processing, completed, failed
created_at: datetime = Field(default_factory=datetime.utcnow)
# ============================================================================
# API Request Models
# ============================================================================
class CreateReferralLinkRequest(BaseModel):
"""Admin request to create a vanity referral link."""
vanity_code: Optional[str] = None
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int = 10
kickback_percent: int = 20
discount_duration_days: int = 30
earning_duration_days: int = 365
class UpdateReferralLinkRequest(BaseModel):
"""Request to update a referral link (user: name/message only, admin: all fields)."""
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: Optional[int] = None
kickback_percent: Optional[int] = None
discount_duration_days: Optional[int] = None
earning_duration_days: Optional[int] = None
enabled: Optional[bool] = None
# ============================================================================
# API Response Models
# ============================================================================
class ReferralLinkResponse(BaseModel):
"""Public referral link information."""
code: str
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int
kickback_percent: int
is_vanity: bool
enabled: bool
stats: ReferralLinkStats
created_at: datetime
class ReferralEarningResponse(BaseModel):
"""Public earning information (for referrer dashboard)."""
id: str
referred_email: str
referral_code: str
purchase_amount_cents: int
earning_amount_cents: int
status: str
created_at: datetime
paid_at: Optional[datetime] = None
class ReferralPayoutResponse(BaseModel):
"""Public payout information (for referrer dashboard)."""
id: str
amount_cents: int
earnings_count: int
status: str
created_at: datetime
class ReferralDashboardResponse(BaseModel):
"""Full referral dashboard data for the referrer."""
referral_link: ReferralLinkResponse
recent_earnings: List[ReferralEarningResponse]
recent_payouts: List[ReferralPayoutResponse]
total_earned_cents: int
total_paid_cents: int
pending_balance_cents: int
stripe_connect_onboarded: bool
class ReferralInterstitialResponse(BaseModel):
"""Data for the referral landing/interstitial page."""
valid: bool = True
referral_code: str = ""
referrer_display_name: Optional[str] = None
discount_percent: int = 0
discount_duration_days: int = 0
custom_message: Optional[str] = None
+6 -53
View File
@@ -10,19 +10,14 @@ class URLSubmissionRequest(BaseModel):
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
is_private: Optional[bool] = None # Private (non-published) track mode
# Duration-based pricing: supplied by the client from YouTube metadata
duration_seconds: Optional[float] = None # Known audio duration in seconds
acknowledged_credits: Optional[int] = None # Client's expected credit cost (anti-mismatch guard)
class UploadSubmissionRequest(BaseModel):
@@ -33,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -42,13 +37,12 @@ class UploadSubmissionRequest(BaseModel):
class CorrectionsSubmission(BaseModel):
"""
Request to submit corrected lyrics after human review.
This is the critical human-in-the-loop interaction point.
The corrections data comes from the lyrics-transcriber review interface.
"""
corrections: Dict[str, Any] # Full corrections JSON from frontend
user_notes: Optional[str] = None # Optional notes from reviewer
is_duet: Optional[bool] = None # When present, persists duet flag to state_data
@validator('corrections')
def validate_corrections_format(cls, v):
@@ -63,13 +57,13 @@ class CorrectionsSubmission(BaseModel):
class InstrumentalSelection(BaseModel):
"""
Request to select instrumental audio option.
This is the second critical human-in-the-loop interaction point.
User chooses between clean instrumental, instrumental with backing vocals,
or a custom instrumental (created via create-custom-instrumental endpoint).
"""
selection: str # "clean", "with_backing", or "custom"
@validator('selection')
def validate_selection(cls, v):
"""Validate selection is a valid option."""
@@ -79,26 +73,6 @@ class InstrumentalSelection(BaseModel):
return v
class CompleteReviewRequest(BaseModel):
"""
Request to complete the review with optional instrumental selection.
For the combined review flow, this includes the instrumental selection.
For legacy flows, the instrumental_selection can be omitted.
"""
instrumental_selection: Optional[str] = None # "clean", "with_backing", or "custom"
@validator('instrumental_selection')
def validate_instrumental_selection(cls, v):
"""Validate instrumental selection if provided."""
if v is None:
return v
valid_options = ['clean', 'with_backing', 'custom']
if v not in valid_options:
raise ValueError(f"instrumental_selection must be one of: {valid_options}")
return v
class MuteRegionRequest(BaseModel):
"""A region to mute in the backing vocals."""
start_seconds: float
@@ -147,24 +121,3 @@ class RetryJobRequest(BaseModel):
"""Request to retry a failed job."""
from_stage: Optional[str] = None # Optional: restart from specific stage
class ChangeVisibilityRequest(BaseModel):
"""Request to change job visibility (public/private)."""
target_visibility: str # "public" or "private"
@validator('target_visibility')
def validate_target(cls, v):
if v not in ('public', 'private'):
raise ValueError("target_visibility must be 'public' or 'private'")
return v
class ChangeVisibilityResponse(BaseModel):
"""Response from change visibility endpoint."""
status: str # "success", "processing", "error"
job_id: str
message: str
previous_visibility: str # "public" or "private"
new_visibility: str # "public" or "private"
reprocessing_required: bool
-107
View File
@@ -1,107 +0,0 @@
"""
Review session model for persisting lyrics review progress.
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class ReviewSessionSummary:
"""Compact summary of edits in a review session, for list views."""
total_segments: int = 0
total_words: int = 0
corrections_made: int = 0
changed_words: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"total_segments": self.total_segments,
"total_words": self.total_words,
"corrections_made": self.corrections_made,
"changed_words": self.changed_words,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
return cls(
total_segments=data.get("total_segments", 0),
total_words=data.get("total_words", 0),
corrections_made=data.get("corrections_made", 0),
changed_words=data.get("changed_words", []),
)
@dataclass
class ReviewSession:
"""
A saved snapshot of a lyrics review editing session.
Stored at: jobs/{job_id}/review_sessions/{session_id}
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "preview" | "manual"
audio_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
correction_data_gcs_path: str = ""
# Hash of the correction data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"correction_data_gcs_path": self.correction_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
-218
View File
@@ -1,218 +0,0 @@
"""
Tenant data models for white-label B2B portals.
Tenants are B2B customers who get their own branded portal with custom
configuration. Each tenant has:
- Custom branding (logo, colors, site title)
- Feature flags (enable/disable audio search, distribution, etc.)
- Default settings (theme, distribution mode)
- Auth configuration (allowed email domains, fixed tokens)
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
"""
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel, Field
class TenantBranding(BaseModel):
"""Branding configuration for a tenant's portal."""
logo_url: Optional[str] = Field(
None, description="URL to tenant's logo image (PNG with transparency preferred)"
)
logo_height: int = Field(40, description="Logo height in pixels for header display")
primary_color: str = Field(
"#ff5bb8", description="Primary brand color (hex format)"
)
secondary_color: str = Field(
"#8b5cf6", description="Secondary brand color (hex format)"
)
accent_color: Optional[str] = Field(
None, description="Accent color for highlights (hex format)"
)
background_color: Optional[str] = Field(
None, description="Custom background color (defaults to dark theme)"
)
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
site_title: str = Field(
"Karaoke Generator", description="Browser tab title"
)
tagline: Optional[str] = Field(
None, description="Optional tagline shown below logo"
)
class TenantFeatures(BaseModel):
"""Feature flags controlling what's available in the tenant's portal."""
# Audio input methods
audio_search: bool = Field(
True, description="Enable audio search (flacfetch integration)"
)
file_upload: bool = Field(True, description="Enable direct file upload")
youtube_url: bool = Field(True, description="Enable YouTube URL input")
# Distribution options
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
# Customization options
theme_selection: bool = Field(
True, description="Allow user to select theme (false = use default)"
)
color_overrides: bool = Field(
True, description="Allow user to customize colors"
)
# Output formats
enable_cdg: bool = Field(True, description="Generate CDG format output")
enable_4k: bool = Field(True, description="Generate 4K video output")
# Advanced features
admin_access: bool = Field(
False, description="Allow access to admin dashboard"
)
class TenantDefaults(BaseModel):
"""Default settings applied to all jobs for this tenant."""
theme_id: Optional[str] = Field(
None, description="Default theme ID (required if theme_selection=false)"
)
locked_theme: Optional[str] = Field(
None,
description="If set, users cannot change theme - always uses this theme ID"
)
distribution_mode: str = Field(
"all",
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
)
brand_prefix: Optional[str] = Field(
None, description="Prefix for output filenames (e.g., 'VSTAR')"
)
dropbox_path: Optional[str] = Field(
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
)
gdrive_folder_id: Optional[str] = Field(
None, description="Tenant-specific Google Drive folder ID for output uploads"
)
youtube_description_template: Optional[str] = Field(
None, description="Default YouTube description template"
)
class TenantAuth(BaseModel):
"""Authentication configuration for a tenant."""
allowed_email_domains: List[str] = Field(
default_factory=list,
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
)
require_email_domain: bool = Field(
True,
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
)
fixed_token_ids: List[str] = Field(
default_factory=list,
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
)
sender_email: Optional[str] = Field(
None,
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
)
class TenantConfig(BaseModel):
"""Complete configuration for a white-label tenant."""
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
subdomain: str = Field(
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
)
is_active: bool = Field(True, description="Whether tenant portal is active")
branding: TenantBranding = Field(default_factory=TenantBranding)
features: TenantFeatures = Field(default_factory=TenantFeatures)
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
auth: TenantAuth = Field(default_factory=TenantAuth)
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
def get_frontend_url(self) -> str:
"""Get the frontend URL for this tenant's portal."""
return f"https://{self.subdomain}"
def get_sender_email(self) -> str:
"""Get the email sender address for this tenant."""
if self.auth.sender_email:
return self.auth.sender_email
# Default pattern: {tenant_id}@nomadkaraoke.com
return f"{self.id}@nomadkaraoke.com"
def is_email_allowed(self, email: str) -> bool:
"""Check if an email address is allowed for this tenant."""
if not self.auth.allowed_email_domains:
# No domain restrictions
return True
email_lower = email.lower()
for domain in self.auth.allowed_email_domains:
if email_lower.endswith(f"@{domain.lower()}"):
return True
return not self.auth.require_email_domain
class TenantPublicConfig(BaseModel):
"""
Public tenant configuration returned to frontend.
This excludes sensitive auth details like token IDs.
"""
id: str
name: str
subdomain: str
is_active: bool
branding: TenantBranding
features: TenantFeatures
defaults: TenantDefaults
# Auth info limited to what frontend needs
allowed_email_domains: List[str] = Field(default_factory=list)
@classmethod
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
"""Create public config from full config."""
return cls(
id=config.id,
name=config.name,
subdomain=config.subdomain,
is_active=config.is_active,
branding=config.branding,
features=config.features,
defaults=TenantDefaults(
theme_id=config.defaults.theme_id,
locked_theme=config.defaults.locked_theme,
distribution_mode=config.defaults.distribution_mode,
# Don't expose brand_prefix or youtube_description_template
),
allowed_email_domains=config.auth.allowed_email_domains,
)
class TenantConfigResponse(BaseModel):
"""Response from GET /api/tenant/config endpoint."""
tenant: Optional[TenantPublicConfig] = Field(
None, description="Tenant config if found, null for default Nomad Karaoke"
)
is_default: bool = Field(
True, description="True if using default Nomad Karaoke config"
)
+52 -121
View File
@@ -6,10 +6,11 @@ Supports:
- Credit system for karaoke generation
- Role-based access control (user/admin)
- Stripe integration for payments
- Beta tester program with feedback collection
"""
from datetime import datetime, timezone
from datetime import datetime
from enum import Enum
from typing import Optional, List, Dict
from typing import Optional, List
from pydantic import BaseModel, Field
@@ -19,6 +20,14 @@ class UserRole(str, Enum):
ADMIN = "admin"
class BetaTesterStatus(str, Enum):
"""Status of beta tester participation."""
ACTIVE = "active" # Enrolled, free credits available
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
COMPLETED = "completed" # Feedback submitted
EXPIRED = "expired" # 24hr deadline passed without feedback
class CreditTransaction(BaseModel):
"""Record of a credit transaction."""
id: str
@@ -30,42 +39,17 @@ class CreditTransaction(BaseModel):
created_by: Optional[str] = None # Admin email if granted by admin
class PushSubscription(BaseModel):
"""
Web Push subscription for a user's device.
Stores the push subscription endpoint and encryption keys needed
to send push notifications to the user's browser/device.
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage (e.g., a consumer job notification appearing
on a tenant portal's service worker).
"""
endpoint: str # Push service endpoint URL
keys: Dict[str, str] # p256dh and auth keys for encryption
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
last_used_at: Optional[datetime] = None # Last time a notification was sent
class User(BaseModel):
"""
User model stored in Firestore.
Users are identified by email address. Authentication is via magic links
(no passwords). Credits are consumed when creating karaoke jobs.
For multi-tenant white-label portals, users are scoped to a tenant.
Users with tenant_id=None are default Nomad Karaoke users.
"""
email: str # Primary identifier
role: UserRole = UserRole.USER
credits: int = 0
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Stripe integration
stripe_customer_id: Optional[str] = None
@@ -85,35 +69,16 @@ class User(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
# Payment tracking (cached from stripe_payments, in cents)
total_spent: int = 0
# Optional profile fields for future use
display_name: Optional[str] = None
# i18n locale preference (set from Accept-Language on login, used for emails/push)
locale: Optional[str] = None
# Push notification subscriptions (Web Push API)
# Users can subscribe from multiple devices/browsers
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
# Feedback-for-credits program
has_submitted_feedback: bool = False
# Anti-abuse: IP address and device fingerprint used during signup
signup_ip: Optional[str] = None
device_fingerprint: Optional[str] = None
# Flag to prevent duplicate welcome credit grants (idempotency)
welcome_credits_granted: bool = False
# Referral system
referral_code: Optional[str] = None # User's own referral code
referred_by_code: Optional[str] = None # Code used when this user signed up
referred_at: Optional[datetime] = None # When the referral was applied
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
# Beta tester program
is_beta_tester: bool = False
beta_tester_status: Optional[BetaTesterStatus] = None
beta_enrolled_at: Optional[datetime] = None
beta_promise_text: Optional[str] = None # User's promise statement
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
beta_feedback_email_sent: bool = False
class MagicLinkToken(BaseModel):
@@ -122,9 +87,6 @@ class MagicLinkToken(BaseModel):
Tokens are short-lived (15 minutes) and single-use.
Stored in Firestore with automatic TTL.
For multi-tenant portals, the tenant_id determines which portal
the user will be redirected to after verification.
"""
token: str # Secure random token
email: str
@@ -134,18 +96,6 @@ class MagicLinkToken(BaseModel):
used_at: Optional[datetime] = None
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Referral attribution (travels with magic link for cross-device support)
referral_code: Optional[str] = None
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
credit_eval_reasoning: Optional[str] = None
credit_eval_error: Optional[str] = None
class Session(BaseModel):
@@ -154,8 +104,6 @@ class Session(BaseModel):
Sessions are created after successful magic link verification.
Sessions expire after 7 days of inactivity or 30 days absolute.
For multi-tenant portals, sessions are scoped to a tenant.
"""
token: str # Secure random session token
user_email: str
@@ -164,20 +112,14 @@ class Session(BaseModel):
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
is_active: bool = True
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Pydantic models for API requests/responses
class SendMagicLinkRequest(BaseModel):
"""Request to send a magic link email."""
email: str
device_fingerprint: Optional[str] = None
referral_code: Optional[str] = None # From referral interstitial
class SendMagicLinkResponse(BaseModel):
@@ -191,27 +133,12 @@ class VerifyMagicLinkRequest(BaseModel):
token: str
class ResendFromTokenRequest(BaseModel):
"""Request to resend a magic link based on a previously-issued (expired/used) token."""
token: str
class ResendFromTokenResponse(BaseModel):
"""Response after attempting to resend a magic link from a known token."""
status: str # "sent" | "no_token"
masked_email: Optional[str] = None
message: str
class VerifyMagicLinkResponse(BaseModel):
"""Response after verifying magic link."""
status: str
session_token: str
user: "UserPublic"
message: str
tenant_subdomain: Optional[str] = None
credits_granted: int = 0
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
class UserPublic(BaseModel):
@@ -222,16 +149,6 @@ class UserPublic(BaseModel):
display_name: Optional[str] = None
total_jobs_created: int = 0
total_jobs_completed: int = 0
tenant_id: Optional[str] = None
feedback_eligible: bool = False
total_spent: int = 0
referral_code: Optional[str] = None
has_active_referral_discount: bool = False
referral_discount_percent: Optional[int] = None
referral_discount_expires_at: Optional[str] = None # ISO string
referred_by_code: Optional[str] = None
created_at: Optional[str] = None
last_login_at: Optional[str] = None
class AddCreditsRequest(BaseModel):
@@ -257,18 +174,18 @@ class UserListResponse(BaseModel):
# ============================================================================
# User Feedback-for-Credits Models
# Beta Tester Program Models
# ============================================================================
class UserFeedback(BaseModel):
class BetaTesterFeedback(BaseModel):
"""
Feedback from any user (not just beta testers).
Feedback from a beta tester after using the service.
Stored in Firestore 'user_feedback' collection.
Users earn 1 free credit for submitting detailed feedback.
Stored in Firestore 'beta_feedback' collection.
"""
id: str
id: str # Unique feedback ID
user_email: str
job_id: Optional[str] = None # The job they're providing feedback on
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
@@ -287,11 +204,33 @@ class UserFeedback(BaseModel):
# Metadata
created_at: datetime = Field(default_factory=datetime.utcnow)
submitted_via: str = "web"
submitted_via: str = "web" # 'web' or 'email'
class UserFeedbackRequest(BaseModel):
"""Request to submit user feedback for credits."""
class BetaTesterEnrollRequest(BaseModel):
"""Request to enroll as a beta tester."""
email: str
promise_text: str = Field(
min_length=10,
description="User's promise to provide feedback (min 10 chars)"
)
accept_corrections_work: bool = Field(
description="User accepts they may need to review/correct lyrics"
)
class BetaTesterEnrollResponse(BaseModel):
"""Response after enrolling as beta tester."""
status: str
message: str
credits_granted: int
session_token: Optional[str] = None # If new user, provide session
class BetaFeedbackRequest(BaseModel):
"""Request to submit beta tester feedback."""
job_id: Optional[str] = None
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
ease_of_use_rating: int = Field(ge=1, le=5)
@@ -308,16 +247,8 @@ class UserFeedbackRequest(BaseModel):
would_use_again: bool = True
class UserFeedbackResponse(BaseModel):
"""Response after submitting user feedback."""
class BetaFeedbackResponse(BaseModel):
"""Response after submitting feedback."""
status: str
message: str
credits_granted: int = 0
class FeedbackEligibilityResponse(BaseModel):
"""Response for feedback eligibility check."""
eligible: bool
has_submitted: bool
jobs_completed: int
credits_reward: int = 1
bonus_credits: int = 0 # Bonus credits for great feedback
Loaded 100 of 5639 files, more files were not shown because too many files have changed in this diff. Show more