Compare commits

..
Author SHA1 Message Date
Andrew Beveridge d0a657610b Added admin API script 2026-01-04 20:32:46 -05:00
5334 changed files with 262307 additions and 967621 deletions

No files matched your search

@@ -1,109 +0,0 @@
# Plan: Admin Feedback Visibility
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1454-feedback-email-debug
**Status:** Draft
## Overview
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
1. **Admin UI page** to browse and review all submitted feedback
2. **Email notification** to admin@nomadkaraoke.com on each new submission
## Requirements
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
- [ ] Sortable by date (newest first by default) and filterable by search (email)
- [ ] Star ratings rendered visually (not just numbers)
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
- [ ] Email includes all feedback content (ratings, text fields, user email)
- [ ] Test data filtering works (exclude_test toggle)
- [ ] Sidebar navigation includes new "Feedback" item
## Technical Approach
### Admin API Endpoint
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
### Admin UI Page
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
### Email Notification
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
## Implementation Steps
### Step 1: Backend — Admin feedback list endpoint
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
2. [ ] Add `GET /api/admin/feedback` endpoint with:
- Pagination: `limit=50`, `offset=0`
- Search by user email
- `exclude_test` filter (reuse `is_test_email`)
- Sort by `created_at` descending
- Return: items, total, offset, limit, has_more
3. [ ] Add aggregate stats in response: average ratings, total count
### Step 2: Backend — Email notification on submission
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
- User email
- All 4 star ratings
- All text fields
- Would recommend / would use again
- Submitted timestamp
- Link to admin feedback page
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
### Step 3: Frontend — Admin API client
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
### Step 4: Frontend — Admin feedback page
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
- Search by email
- Pagination (50 per page)
- Refresh button
- Test data toggle (via `useAdminSettings`)
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
### Step 5: Testing
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
11. [ ] Backend unit test for `send_feedback_notification` email method
12. [ ] Frontend component test for feedback page (render, empty state)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
## Testing Strategy
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
- **Integration**: Submit feedback → verify email service called
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
## Open Questions
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
## Rollback Plan
- Backend: Remove endpoint from admin.py, revert email call in users.py
- Frontend: Delete feedback page, remove sidebar item
- No data migration needed — Firestore collection unchanged
-393
View File
@@ -1,393 +0,0 @@
# Plan: Admin Job Details Page
**Created:** 2026-01-09
**Branch:** feat/sess-20260109-1738-admin-job-details
**Status:** In Progress (Phase 1 Complete)
## Overview
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
## Requirements
### Must Have
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
- [ ] Download links for all existing files (generate signed URLs)
- [ ] View worker logs with filtering by level/worker
- [ ] Restart job from beginning (reset to PENDING)
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
### Should Have
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
- [ ] View/edit raw JSON fields (state_data, file_urls)
- [ ] Quick actions: send completion email, clear cache, impersonate user
### Nice to Have
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
- [ ] Diff view when editing fields
- [ ] Audit log of admin changes
## Technical Approach
### Routing Fix (COMPLETED)
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
- Uses `useSearchParams()` to detect `?id=` parameter
- Shows detail view when `selectedJobId` is present, list view otherwise
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
- Deleted old `/admin/jobs/detail/` directory
### Backend Enhancements
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
### Frontend Structure
The page will have collapsible sections:
1. **Header**: Job ID, status badge, artist/title, quick actions
2. **Overview Cards**: User, dates, progress, source info
3. **Stage Timeline**: Visual breakdown of each stage with times
4. **Files**: Downloadable links for all GCS files
5. **Logs**: Filterable worker logs
6. **Configuration**: Editable job settings
7. **Raw Data**: JSON views of state_data, file_urls, full job
### Job Reset Logic
Reset to specific states requires:
- Clearing downstream state_data keys
- Resetting status
- Optionally clearing associated files
- Adding timeline event for the reset
Valid reset targets:
| Target State | Clears | Use Case |
|--------------|--------|----------|
| PENDING | Everything except input | Full restart |
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
## Implementation Steps
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
### Phase 1: Fix Routing (COMPLETED)
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
2. [x] Combined list and detail views in single component
3. [x] Use `useSearchParams()` to detect `?id=` parameter
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
6. [x] Verified: All 9 E2E tests pass, build succeeds
### Phase 2: Backend - File Downloads with Signed URLs
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
10. [ ] Traverse job.file_urls recursively
11. [ ] Generate signed URLs for each GCS path
12. [ ] Return structured response with file metadata
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
### Phase 3: Frontend - Enhanced File Display
15. [ ] Create "Files" accordion section with grouped downloads
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
17. [ ] Show file name, type, download button
18. [ ] Handle missing/unavailable files gracefully
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
20. [ ] Verify: `npm run test:e2e` passes
### Phase 4: Backend - Job Update Endpoint
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
23. [ ] Define editable fields whitelist
24. [ ] Validate field values (e.g., status must be valid enum)
25. [ ] Log admin changes for audit trail
26. [ ] Verify: Backend tests pass
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
### Phase 5: Frontend - Editable Fields
28. [ ] Create "Configuration" section with editable fields
29. [ ] Use inline edit pattern (click to edit, enter to save)
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
31. [ ] Show validation errors inline
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
33. [ ] Verify: `npm run test:e2e` passes
### Phase 6: Backend - Job Reset Endpoint
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
36. [ ] Accept `target_state` parameter
37. [ ] Implement reset logic for each target state
38. [ ] Clear appropriate state_data keys
39. [ ] Optionally trigger workers after reset
40. [ ] Verify: Backend tests pass
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
### Phase 7: Frontend - Reset Actions
42. [ ] Add "Reset Job" dropdown/menu in header
43. [ ] Show available reset targets based on current state
44. [ ] Confirm dialog explaining what will be cleared
45. [ ] Show success/error toast after reset
46. [ ] **E2E Test**: Add reset confirmation dialog test
47. [ ] Verify: `npm run test:e2e` passes
### Phase 8: Enhanced Stage Timeline
48. [ ] Parse timeline events to calculate stage durations
49. [ ] Display stages as cards or timeline items
50. [ ] Show duration for each stage
51. [ ] Highlight current/blocking stages
52. [ ] **E2E Test**: Add timeline display test
### Phase 9: Final Verification
53. [ ] Run full backend test suite: `make test-backend`
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
55. [ ] Manual production testing checklist (see Testing Strategy)
56. [ ] Verify CI passes on PR
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
## Testing Strategy
Based on `docs/TESTING.md` guidelines, this feature requires:
### Test Types Required
| Type | Location | Purpose |
|------|----------|---------|
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
| Manual Production E2E | N/A | Final validation on staging/prod |
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
```python
def test_get_job_files_returns_signed_urls_for_all_files():
"""Test that all files in file_urls get signed URLs."""
def test_get_job_files_handles_nested_file_urls():
"""Test recursive traversal of stems.instrumental_clean, etc."""
def test_get_job_files_handles_missing_files_gracefully():
"""Test files that don't exist in GCS are omitted or marked."""
def test_get_job_files_requires_admin():
"""Test non-admin gets 403."""
def test_get_job_files_returns_404_for_missing_job():
"""Test invalid job_id returns 404."""
```
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
```python
def test_update_job_allows_editable_fields():
"""Test artist, title, user_email, theme_id can be updated."""
def test_update_job_rejects_non_editable_fields():
"""Test job_id, created_at, etc. cannot be updated."""
def test_update_job_validates_status_enum():
"""Test invalid status value returns 422."""
def test_update_job_validates_theme_id_exists():
"""Test non-existent theme_id returns 422."""
def test_update_job_requires_admin():
"""Test non-admin gets 403."""
def test_update_job_returns_updated_job():
"""Test response includes updated job data."""
```
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
```python
def test_reset_to_pending_clears_all_state_data():
"""Test full reset clears everything except input."""
def test_reset_to_audio_selection_clears_search_results():
"""Test audio selection reset clears audio_search_results."""
def test_reset_to_awaiting_review_clears_review_data():
"""Test review reset clears screens, videos, review data."""
def test_reset_to_instrumental_selection_clears_finals():
"""Test instrumental reset clears encoding results."""
def test_reset_adds_timeline_event():
"""Test reset adds admin action to timeline."""
def test_reset_invalid_target_returns_422():
"""Test invalid target_state returns validation error."""
def test_reset_requires_admin():
"""Test non-admin gets 403."""
def test_reset_from_complete_to_pending_works():
"""Test can reset completed job back to start."""
```
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
```typescript
import { test, expect } from '@playwright/test';
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
test.describe('Admin Job Detail Page', () => {
test.beforeEach(async ({ page }) => {
await setAdminAuthToken(page);
});
test('navigates from jobs list to detail page', async ({ page }) => {
// Mock jobs list and single job endpoints
await page.route('**/api/admin/jobs', route => route.fulfill({
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
}));
await page.route('**/api/jobs/test123', route => route.fulfill({
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
}));
await page.goto('/admin/jobs');
await page.click('text=test123');
await expect(page).toHaveURL('/admin/jobs/test123');
await expect(page.getByText('Job test123')).toBeVisible();
});
test('displays all job metadata sections', async ({ page }) => {
// Test overview cards, timeline, logs accordion, etc.
});
test('files section shows downloadable links', async ({ page }) => {
// Mock files endpoint, verify download buttons appear
});
test('edit field inline and save', async ({ page }) => {
// Click edit, change value, press Enter, verify PATCH called
});
test('reset job shows confirmation dialog', async ({ page }) => {
// Click reset, verify dialog, confirm, verify POST called
});
test('handles API errors gracefully', async ({ page }) => {
// Mock 500 response, verify error toast shown
});
});
```
### Test File Organization
```
backend/tests/
├── test_admin_job_detail.py # NEW - all backend tests for this feature
├── test_admin_api.py # Existing admin tests (keep separate)
└── conftest.py # Shared fixtures
frontend/e2e/
├── fixtures/
│ ├── test-helper.ts # Add setAdminAuthToken helper
│ └── mock-responses.ts # Add admin job detail mocks
└── regression/
└── admin-job-detail.spec.ts # NEW - E2E regression tests
```
### Mocking Guidelines
**Backend - What to mock:**
- `StorageService.generate_signed_url()` - return predictable test URLs
- `StorageService.file_exists()` - control which files "exist"
- Firestore operations via existing `mock_firestore` fixture
**Backend - What NOT to mock:**
- `JobManager` logic - test the real reset/update logic
- Validation logic - test real Pydantic validation
- Authorization checks - test real `require_admin` dependency
**Frontend - What to mock:**
- All API responses via Playwright route interception
- Time (if testing token expiry)
**Frontend - What NOT to mock:**
- React component behavior
- User interactions (click, type, etc.)
### CI Requirements
This feature will be validated by existing CI checks:
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
- `frontend-build` - Must pass (verifies new page compiles)
### Running Tests Locally
```bash
# Backend tests only (for this feature)
poetry run pytest backend/tests/test_admin_job_detail.py -v
# All backend tests
make test-backend
# Frontend E2E regression (includes new tests)
cd frontend && npm run test:e2e
# Single E2E test file
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
# With browser visible for debugging
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
```
### Manual Production Testing Checklist
After deployment, manually verify on production:
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
- [ ] All file download links work (valid signed URLs)
- [ ] Edit artist/title → refresh → changes persisted
- [ ] Reset job to pending → job status shows "pending"
- [ ] Logs section loads and filters work
- [ ] Error states handled (invalid job ID shows friendly error)
## Open Questions
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
- **Recommendation**: Delete it. No external links depend on it.
2. [ ] Should job reset delete GCS files or just clear state_data references?
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
3. [ ] Should edits be immediately saved or require explicit "Save" button?
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
## Rollback Plan
If issues arise:
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
2. Backend: New endpoints are additive, won't affect existing functionality
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
## Notes
- The existing detail page has good foundational code - we're enhancing rather than replacing
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
- Job reset is powerful - consider adding confirmation dialogs and logging
-120
View File
@@ -1,120 +0,0 @@
# Plan: Admin Lyrics Review Dashboard
**Created:** 2026-03-05
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
**Status:** Draft
## Overview
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
## Requirements
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
- [ ] Admin page showing list of recently reviewed jobs with key metrics
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
- [ ] Edit log replay/stepper UI to walk through user changes one by one
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
## Technical Approach
### Backend: 2 new admin endpoints
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
- Filter by status (completed jobs, or optionally all post-review statuses)
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
- Support pagination, search, exclude_test filtering
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
- Returns immutable copy of:
- Original corrections (`corrections.json` from GCS)
- Updated corrections (`corrections_updated.json` from GCS)
- Edit log (`edit_log_{session_id}.json` from GCS)
- Annotations (`annotations.json` from GCS)
- Job metadata (artist, title, user_email, timestamps)
- Audio URL (signed, for playback in review UI)
- This is a **read-only** endpoint - no job state mutations
- Uses `require_admin` dependency (not `require_review_auth`)
- Does NOT transition job status or modify any state
### Frontend: New admin page + Read-only review mode
1. **`/admin/edit-reviews/page.tsx`** - List page
- Table of recently reviewed jobs
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
- "View Review" button → opens read-only lyrics review
- "View Edit Log" button → opens edit replay view
- "View Raw Data" button → shows raw JSON
2. **Read-only review viewer** - Reuse LyricsAnalyzer
- Load data from the new admin endpoint (not the regular review endpoint)
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
- Header shows "Admin Review - Read Only" indicator
- No submit/save buttons, no state transitions
- Audio playback still works (via signed URL)
3. **Edit log replay** - New component
- Timeline/stepper showing each edit entry
- For each entry: operation type, text before → text after, timestamp
- Step forward/backward through edits
- Show feedback reason if user provided one
- Summary stats: total edits, edits with feedback, correction types breakdown
4. **Raw data viewer** - JSON viewer tabs
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
- Tab 2: Annotations
- Tab 3: Corrections diff (original vs updated)
### Key safety guarantee
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
## Implementation Steps
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
9. [ ] **Tests: Backend unit tests** for new admin endpoints
10. [ ] **Tests: Frontend component tests** for new page
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
## Testing Strategy
- Unit tests for backend endpoints (mock Firestore + GCS)
- Verify admin auth is required on all new endpoints
- Verify no write operations occur on read-only endpoints
- Frontend component tests for key interactions
## Open Questions
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
## Rollback Plan
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
-151
View File
@@ -1,151 +0,0 @@
# Plan: AI-Powered Credit Gating
**Created:** 2026-03-23
**Branch:** feat/sess-20260323-1716-ai-credit-gating
**Status:** Implemented
## Overview
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
## Requirements
- [ ] Welcome credit grants go through AI evaluation before granting
- [ ] Feedback credit grants go through AI evaluation before granting
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
- [ ] Approved users get credits as before (seamless experience)
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
- [ ] Rejection email invites them to reply if it's a mistake
- [ ] Decision is logged to Firestore for audit trail
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
- [ ] Admin can override decisions via existing admin credit grant endpoint
## Technical Approach
### Architecture
```
User verifies email / submits feedback
↓
Collect abuse signals (sync, fast):
- User's fingerprint + IP
- Other accounts with same fingerprint (from correlations)
- Other accounts with same IP
- IP geolocation info
- Recent signup count from same IP/fingerprint (last 24h)
- User agent string
- For feedback: the feedback content itself
↓
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
- Present all signals
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
↓
If "grant": grant credits normally, log decision
If "deny": skip credit grant, send rejection email, log decision
If AI call fails: grant credits (fail-open), log error
```
### Key Design Decisions
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
## Implementation Steps
### Step 1: Add google-generativeai dependency
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
- Verify it works with Vertex AI auth in the nomadkaraoke project
### Step 2: Create credit evaluation service
- New file: `backend/services/credit_evaluation_service.py`
- `CreditEvaluationService` class with:
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
- Builds prompt with signals
- Calls Gemini for decision
- Logs decision to Firestore
- Returns structured result
### Step 3: Build the Gemini prompt
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
- Present all signals in a structured format
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
### Step 4: Create rejection email template
- Add `send_credit_denied_email(email, reasoning)` to EmailService
- Friendly-but-firm tone
- CC andrew@beveridge.uk
- Invite reply if it's a mistake
- Use existing email template infrastructure (branding, styling)
### Step 5: Integrate into welcome credit flow
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
- If denied: send rejection email, return False
- If error: log warning, grant anyway (fail-open)
### Step 6: Integrate into feedback credit flow
- Modify feedback endpoint in users.py
- Before granting credits: evaluate with feedback content included
- If denied: return different response message, send rejection email
- If error: grant anyway
### Step 7: Add Firestore collection for audit trail
- Collection: `credit_evaluations`
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
- TTL: keep indefinitely (small volume, high audit value)
### Step 8: Tests
- Unit tests for signal collection
- Unit tests for prompt building
- Unit tests for decision parsing (grant/deny/error cases)
- Integration test for the evaluation service with mocked Gemini
- Integration test for welcome credit flow with evaluation
- Integration test for feedback credit flow with evaluation
- Test fail-open behavior
### Step 9: Configuration
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
- Add to config.py
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `pyproject.toml` | Modify | Add google-generativeai dependency |
| `backend/config.py` | Modify | Add credit evaluation config vars |
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
| `backend/services/email_service.py` | Modify | Add credit denied email template |
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
## Testing Strategy
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
- **Integration tests**: Full evaluation flow with mocked Gemini responses
- **E2E tests**: Manual verification in production with test accounts
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
## Open Questions
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
## Rollback Plan
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
@@ -1,150 +0,0 @@
# Plan: Block Disposable Email Registration
**Created:** 2026-03-19
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
**Status:** Draft
## Overview
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
## Context
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
- Each new account gets 2 free welcome credits automatically
## Requirements
- [x] Reject disposable email domains at the magic link endpoint (backend)
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
- [x] Add `emailhook.site` to the default disposable domains list
- [x] Add backend unit tests for the integration
- [x] Add frontend test coverage for the new error state
## Design Decision: Explicit Rejection vs Silent Failure
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
1. We *want* the user to know their disposable email won't work (so they can use a real one)
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
3. The friendly message serves our goal of converting them to real email users
## Implementation Steps
### Step 1: Backend — Add disposable email check to magic link endpoint
**File:** `backend/api/routes/users.py`
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
```python
# Check for disposable email domains
email_validation = get_email_validation_service()
if email_validation.is_disposable_domain(email):
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
raise HTTPException(
status_code=422,
detail="disposable_email_not_allowed"
)
```
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
### Step 2: Backend — Also check email and IP blocklists
In the same location, add checks for explicitly blocked emails and IPs:
```python
if email_validation.is_email_blocked(email):
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
# Silent reject — return generic success to prevent enumeration
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
ip_address = http_request.client.host if http_request.client else None
if ip_address and email_validation.is_ip_blocked(ip_address):
logger.warning(f"Blocked IP signup attempt: {ip_address}")
return SendMagicLinkResponse(
status="success",
message="If this email is registered, you will receive a sign-in link shortly."
)
```
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
### Step 3: Backend — Add `emailhook.site` to default disposable domains
**File:** `backend/services/email_validation_service.py`
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
### Step 4: Frontend — Handle the disposable email error in AuthDialog
**File:** `frontend/components/auth/AuthDialog.tsx`
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
```tsx
const success = await sendMagicLink(email.trim().toLowerCase())
if (success) {
setStep("sent")
}
// If error contains our specific code, override with friendly message
// (the error state is already set by the auth store)
```
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
The friendly message:
> **We don't support disposable email addresses**
>
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
>
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
### Step 5: Backend tests
**File:** `backend/tests/test_email_validation_integration.py` (new)
Test cases:
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
2. Magic link with legitimate email proceeds normally
3. Magic link with blocked email returns silent success (no actual email sent)
4. Magic link with blocked IP returns silent success
### Step 6: Frontend tests
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
Test cases:
1. Submitting a disposable email shows the friendly rejection message
2. The friendly message contains the key copy (no spam promise, use real email)
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
## Testing Strategy
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
## Rollback Plan
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
-236
View File
@@ -1,236 +0,0 @@
# Plan: Combined Lyrics + Instrumental Review Phase
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Overview
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
### Key Insight
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
### Benefits
1. **Better UX** - One interaction point instead of two
2. **Faster turnaround** - Users complete both reviews in one session
3. **Reduced drop-off** - Users less likely to abandon between phases
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
5. **Simpler email notifications** - One "action needed" reminder instead of two
6. **Cleaner codebase** - No conditional paths or legacy state handling
## Requirements
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
- [x] Functional: Single email notification for "review needed" instead of two separate ones
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
- [x] Functional: Admin "reset to review" action works with combined flow
- [x] Non-functional: No regression in existing test coverage
- [x] Non-functional: Clean removal of old code paths (no dead code)
## Technical Approach
### Strategy: "Combined Review - Clean Cutover"
**State Machine - Before:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
**State Machine - After:**
```
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
```
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
### Key Changes
1. **Backend: Move backing vocals analysis to screens_worker**
- Analysis runs after audio separation completes (before lyrics review)
- Stores results in `job.state_data['backing_vocals_analysis']`
- Remove analysis from render_video_worker entirely
2. **Backend: Replace review completion endpoint**
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
- Remove the old endpoint that didn't require instrumental
3. **Backend: Simplify render_video_worker**
- Always expect `instrumental_selection` in state_data
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
4. **Frontend: Single combined review page**
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
- Remove separate `/instrumental` route entirely
- Single "Complete Review" button submits both
5. **Backend: Remove instrumental-only endpoints**
- Remove `GET /api/jobs/{job_id}/instrumental-options`
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
- Remove `POST /api/jobs/{job_id}/select-instrumental`
- Keep custom instrumental creation for future advanced use
6. **Email notifications: Simplify**
- Remove "Instrumental selection needed" email trigger entirely
- Update "Review needed" email to mention instrumental selection
- Remove `instrumental_token` generation (no longer needed)
## Implementation Steps
### Phase 1: Backend Changes
1. [x] **Move backing vocals analysis to screens_worker**
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
- Remove analysis code from render_video_worker.py entirely
2. [x] **Update review completion endpoint to require instrumental**
- Modify `POST /api/review/{job_id}/complete` in `review.py`
- Add required `instrumental_selection` field to request body
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
3. [x] **Add instrumental data to correction-data endpoint**
- Modify `GET /api/review/{job_id}/correction-data` to include:
- `instrumental_options`: List of {id, label, audio_url}
- `backing_vocals_analysis`: Analysis data from state_data
4. [x] **Simplify render_video_worker**
- Remove conditional check for instrumental pre-selection
- Always transition directly to `INSTRUMENTAL_SELECTED`
- Always trigger video worker immediately after render
5. [x] **Update state machine**
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
6. [x] **Remove instrumental-only endpoints**
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
- Remove `require_instrumental_auth` dependency
- Remove `instrumental_token` generation in job_manager.py
### Phase 2: Frontend Changes
7. [x] **Create combined review page**
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
- Instrumental selection UI appears after preview video section
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
8. [x] **Create InstrumentalSelectorEmbedded component**
- Created embedded version of InstrumentalSelector
- Accepts options, analysis, value, onChange props
- Supports compact mode for modal embedding
9. [x] **Update review completion flow**
- "Complete Review" button requires instrumental selection
- Validation error shown if no instrumental selected
- Single API call submits both lyrics and instrumental selection
10. [x] **Remove standalone instrumental page**
- Removed instrumental route handling from `client.tsx`
- Removed `getExpectedStates` case for "instrumental"
- InstrumentalSelector component stubbed out (deprecated)
- Kept `selectInstrumental` API for finalise-only jobs
11. [x] **Update lib/api.ts**
- Modified `completeReview()` to require instrumental selection
- Kept `selectInstrumental()` for finalise-only jobs
- Removed other instrumental-only functions
### Phase 3: Email & Notification Cleanup
12. [x] **Remove instrumental notification trigger**
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
- Updated docstrings to clarify finalise-only vs normal flow
- Instrumental reminder scheduling still works for finalise-only jobs
13. [x] **Update review email template**
- Update copy to mention "review lyrics and select instrumental"
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
### Phase 4: Testing & Cleanup
14. [x] **Update/remove tests for old flow**
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
- Updated worker tests to import from screens_worker (where analysis moved)
- Fixed conftest.py mocks to return AuthResult objects
15. [x] **Add tests for combined flow**
- Existing tests already cover combined completion endpoint
- Test infrastructure validates combined review flow works
- 1741 tests pass (only 1 pre-existing unrelated failure)
16. [x] **Update documentation**
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
17. [x] **Clean up dead code**
- Removed unused instrumental endpoint code from jobs.py
- Stubbed InstrumentalSelector component (deprecated)
- Kept select-instrumental endpoint for finalise-only jobs
## Files to Modify/Delete
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
## Testing Strategy
### Unit Tests
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
### Integration Tests
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
- `test_non_interactive_job`: CLI jobs skip review entirely as before
### E2E Tests
- `test_combined_review_ui`: User completes both reviews in single session
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
| CLI regression | Test `non_interactive` flag behavior explicitly |
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
## Migration Notes
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
1. Admin resets job to `AWAITING_REVIEW` via admin panel
2. User completes combined review
3. Job proceeds normally
This is acceptable because:
- Very few jobs should be in this state at any time
- Admin reset is already a supported operation
- Clean cutover is worth the minor migration friction
@@ -1,321 +0,0 @@
# Plan: Consolidate YouTube Download Paths
**Created:** 2026-01-20
**Status:** Planning
## Problem Statement
YouTube downloads currently have **3 different code paths** with different behaviors:
1. **Audio Search → Select YouTube** (`audio_search.py`)
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
- Runs on Cloud Run, blocked by YouTube bot detection
- Never uses remote flacfetch even when configured
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
- Works fine locally, not relevant to cloud issues
### Current Issues
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
3. **Inconsistent error handling** - Different paths handle errors differently
4. **Testing gaps** - No integration tests verifying the full flow
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
## Goals
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
5. **Remove dead code** - Clean up redundant implementations
## Proposed Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Entry Points │
├──────────────────────┬──────────────────────┬───────────────────────┤
│ Audio Search Select │ Direct URL Submit │ File Upload │
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
│ │ │
▼ ▼ │
┌──────────────────────────────────────────────┐ │
│ YouTubeDownloadService │ │
│ (NEW: backend/services/youtube_service.py) │ │
│ │ │
│ - download_by_url(url, job_id) → gcs_path │ │
│ - download_by_id(video_id, job_id) → gcs_path│ │
│ │ │
│ Internally: │ │
│ 1. Check if remote flacfetch enabled │ │
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
│ 3. If no: Use local yt_dlp with clear error │ │
│ messaging about bot detection risk │ │
└──────────────────┬───────────────────────────┘ │
│ │
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ GCS │
│ uploads/{job_id}/audio/ │
└─────────────────────────────────────────────────────────────────────┘
```
## Implementation Steps
### Phase 1: Create YouTubeDownloadService
**File:** `backend/services/youtube_download_service.py`
```python
class YouTubeDownloadService:
"""
Single point of entry for all YouTube downloads in the cloud backend.
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
on the flacfetch VM which has YouTube cookies and avoids bot detection.
When remote is not configured, falls back to local yt_dlp with a warning
that downloads may be blocked.
"""
def __init__(self):
self._flacfetch_client = get_flacfetch_client()
async def download(
self,
url: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""
Download YouTube audio and upload to GCS.
Args:
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
job_id: Job ID for GCS path
artist: Optional artist name for filename
title: Optional title for filename
Returns:
GCS path (not gs:// prefix, just the path portion)
Raises:
YouTubeDownloadError: If download fails
"""
video_id = self._extract_video_id(url)
if self._flacfetch_client:
return await self._download_remote(video_id, job_id, artist, title)
else:
logger.warning(
"Remote flacfetch not configured. Attempting local download, "
"but this may fail due to YouTube bot detection on Cloud Run IPs."
)
return await self._download_local(url, job_id, artist, title)
async def download_by_id(
self,
video_id: str,
job_id: str,
artist: Optional[str] = None,
title: Optional[str] = None,
) -> str:
"""Download by video ID instead of URL."""
url = f"https://www.youtube.com/watch?v={video_id}"
return await self.download(url, job_id, artist, title)
```
### Phase 2: Refactor Audio Search Path
**File:** `backend/api/routes/audio_search.py`
Changes to `_download_and_start_processing()`:
```python
# BEFORE: Complex branching with multiple paths
if is_torrent_source and is_remote_enabled:
# Torrent path...
elif is_remote_enabled and source_id and source_name:
# YouTube remote path (buggy source_id extraction)...
else:
# Local path with another YouTube branch...
# AFTER: Clean separation by source type
if is_torrent_source:
result = await _download_torrent(source_name, source_id, target_file, job_id)
elif source_name == 'YouTube':
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
elif source_name == 'Spotify':
result = await _download_spotify(source_id, job_id, artist, title)
else:
raise DownloadError(f"Unknown source: {source_name}")
```
### Phase 3: Refactor Direct URL Path
**File:** `backend/api/routes/file_upload.py`
Changes to `create_job_from_url()`:
```python
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
# This always runs on Cloud Run and gets blocked
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
if is_youtube_url(body.url):
youtube_service = get_youtube_download_service()
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
# Update job with audio already downloaded
job.input_media_gcs_path = gcs_path
# Skip audio download in worker
```
**File:** `backend/workers/audio_worker.py`
Changes to `download_audio()`:
```python
# BEFORE: Has YouTube download logic with download_from_url()
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
async def download_audio(job_id: str, job: Job) -> str:
"""Download audio from GCS (already uploaded by service layer)."""
if not job.input_media_gcs_path:
raise Exception("No audio path - this shouldn't happen for URL jobs")
# Download from GCS to local for processing
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
storage_service.download_file(job.input_media_gcs_path, local_path)
return local_path
```
### Phase 4: Clean Up Dead Code
Remove or consolidate:
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
### Phase 5: Comprehensive Testing
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
```python
class TestYouTubeDownloadService:
def test_download_uses_remote_when_configured(self):
"""When FLACFETCH_API_URL is set, should use remote client."""
def test_download_falls_back_to_local_when_remote_disabled(self):
"""When remote not configured, should use local yt_dlp."""
def test_download_extracts_video_id_from_various_url_formats(self):
"""Should handle youtube.com, youtu.be, shorts, etc."""
def test_download_returns_correct_gcs_path_format(self):
"""Should return path without gs:// prefix."""
def test_download_handles_remote_failure_gracefully(self):
"""If remote fails, should raise clear error, not fall back silently."""
```
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
```python
class TestYouTubeDownloadFlow:
@pytest.mark.integration
async def test_audio_search_youtube_selection_downloads_correctly(self):
"""Full flow: search → select YouTube → verify GCS upload."""
@pytest.mark.integration
async def test_direct_url_submission_downloads_correctly(self):
"""Full flow: submit YouTube URL → verify GCS upload."""
@pytest.mark.integration
async def test_youtube_download_with_remote_flacfetch(self):
"""Verify remote flacfetch integration works end-to-end."""
```
#### Flacfetch Tests (in flacfetch repo)
```python
class TestYouTubeDownloadByIdEndpoint:
def test_download_by_id_with_youtube_source(self):
"""POST /download-by-id with source_name=YouTube works."""
def test_download_by_id_uploads_to_gcs_when_requested(self):
"""When gcs_path provided, should upload and return gs:// path."""
def test_download_by_id_handles_bot_detection_error(self):
"""Should return clear error if YouTube blocks the request."""
```
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
## Migration Strategy
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
2. **Phase 3** changes the URL submission flow - may need careful rollout
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
4. **Phase 5** tests should be written alongside each phase
## Quick Fix vs Full Refactor
### Quick Fix (do now)
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
- This unblocks the "search → select YouTube" flow
### Full Refactor (this plan)
- Consolidate all YouTube downloads into `YouTubeDownloadService`
- Fix the "direct URL submission" flow to use remote flacfetch
- Remove dead code and simplify branching
- Add comprehensive tests
## Risk Assessment
| Risk | Mitigation |
|------|------------|
| Breaking existing flows | Comprehensive tests before deployment |
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
| YouTube bot detection | Remote flacfetch with cookies is the solution |
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
## Success Criteria
1. **One code path** for YouTube downloads in cloud backend
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
3. **All tests pass** with >80% coverage on new code
4. **Production verified** with real YouTube downloads
5. **No dead code** remaining in the codebase
## Open Questions
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
2. Should direct URL submission be synchronous (wait for download) or async (background)?
3. How do we handle YouTube cookies management for remote flacfetch?
## Related Documentation
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
@@ -1,111 +0,0 @@
# Plan: Credit Refund on Job Deletion and Cancellation
**Created:** 2026-03-04
**Branch:** feat/sess-20260304-1308-investigate-user-credits
**Status:** Implemented
## Overview
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
## Requirements
- [x] Job failures already refund credits (existing behavior, no change needed)
- [ ] Job cancellation refunds 1 credit
- [ ] Job deletion refunds 1 credit (if not already completed)
- [ ] No refund for completed jobs (user got their video)
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
- [ ] Refund failure must not block the cancel/delete operation
## Technical Approach
### Key Insight: Idempotency via Job Field
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
### What Should and Shouldn't Be Refunded
| Job Status at Time of Action | Delete | Cancel | Notes |
|------------------------------|--------|--------|-------|
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
### Refund Helper
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
1. Checks `job.credit_refunded` — skip if already refunded
2. Checks `job.user_email` — skip if no user
3. Checks if admin — skip if admin job
4. Calls `user_service.refund_credit()` with appropriate reason
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
6. Wraps everything in try/except so failures don't block the parent operation
## Implementation Steps
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
- Add `credit_refunded: bool = False` to the Job model
- No migration needed — Firestore is schemaless, defaults to False for existing docs
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
- Add `credit_refunded` check at the top
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
- The job object is already fetched at line 1017
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
- Need to fetch the job (already done when `delete_files=True`, but not always)
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
- For FAILED/CANCELLED: skip because refund was already issued during that transition
- For COMPLETE/PREP_COMPLETE: skip because user received their output
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
- The helper handles this, but also update the Firestore doc
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
- Test: cancel_job refunds credit for non-admin user
- Test: cancel_job skips refund for admin jobs
- Test: delete_job refunds credit for non-complete, non-terminal jobs
- Test: delete_job does NOT refund for completed jobs
- Test: delete_job does NOT double-refund for already-failed jobs
- Test: delete_job does NOT double-refund for already-cancelled jobs
- Test: refund failure doesn't block cancel/delete
- Test: credit_refunded flag prevents double refund
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
## Testing Strategy
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
## Open Questions
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
## Rollback Plan
- The `credit_refunded` field is additive (defaults to False for old jobs)
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
- Any refunds already issued are just bonus credits to users, no harm done
@@ -1,204 +0,0 @@
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
**Created:** 2026-01-19
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
**Status:** Draft
## Overview
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
This plan covers:
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
2. Archiving the GitHub repo
3. Deprecating the PyPI package (via README update published as final version)
4. Reorganizing vendored code to a permanent location in karaoke-gen
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
## Requirements
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
- [ ] python-lyrics-transcriber GitHub repo archived
- [ ] Final PyPI release with deprecation notice in README
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
- [ ] Tests pass after reorganization
- [ ] karaoke-gen pyproject.toml updated for new path
## Technical Approach
### New Path Structure
Move the vendored code to `libs/lyrics_transcriber/`:
```
karaoke-gen/
├── libs/
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
│ ├── __init__.py
│ ├── cli/
│ ├── core/
│ ├── correction/
│ ├── lyrics/
│ ├── output/
│ ├── review/
│ ├── storage/
│ ├── transcribers/
│ ├── types.py
│ └── utils/
├── karaoke_gen/
├── backend/
├── frontend/
└── ...
```
### Why `libs/lyrics_transcriber/` instead of alternatives?
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
### pyproject.toml Changes
```toml
packages = [
{ include = "karaoke_gen" },
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
{ include = "backend" }
]
include = [
"karaoke_gen/nextjs_frontend/out",
"karaoke_gen/nextjs_frontend/out/**/*",
"libs/lyrics_transcriber/output/fonts/*", # Updated path
"libs/lyrics_transcriber/output/cdgmaker/images/*",
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
]
```
### Files to Remove from Vendored Code
From current `lyrics_transcriber_temp/`:
- `pyproject.toml` - Package config for standalone release
- `poetry.lock` - Lock file for standalone package
- `README.md` - Will be replaced by deprecation notice
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
- `Dockerfile` - Standalone container build
- `.github/` - CI/CD for standalone package
- `.cursor/` - Cursor AI config
- `.specify/` - Specify config
- `.gitignore` - Not needed (karaoke-gen has its own)
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
- `scripts/` - Development scripts for standalone package
- `specs/` - Specification files
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
## Implementation Steps
### Phase 1: Update and Archive External Project (Manual Steps)
1. [ ] Clone python-lyrics-transcriber repo locally
2. [ ] Create deprecation notice README:
```markdown
# python-lyrics-transcriber (DEPRECATED)
**This project has been deprecated and archived.**
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
## Migration
For karaoke video generation with synchronized lyrics:
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
- Web app: https://gen.nomadkaraoke.com
## Historical Context
This library was originally developed as a standalone tool for creating synchronized
lyrics files. It has now been integrated into the karaoke-gen platform which provides
a complete end-to-end solution for karaoke video generation.
The last standalone version was 0.81.0. No further releases will be made to PyPI.
```
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
### Phase 2: Reorganize Vendored Code in karaoke-gen
5. [ ] Create `libs/` directory
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
8. [ ] Update `pyproject.toml` packages and include paths
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
### Phase 3: Verify and Test
10. [ ] Run `poetry install` to verify package configuration
11. [ ] Run `make test` to ensure all tests pass
12. [ ] Verify imports work:
```python
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
```
### Phase 4: Handle Tests from lyrics_transcriber
13. [ ] Evaluate lyrics_transcriber tests:
- **74 test files** in `lyrics_transcriber_temp/tests/`
- These test the internal lyrics_transcriber library functionality
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
- Skip integration/performance/manual tests that were for standalone development
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
| `pyproject.toml` | Modify | Update package paths |
| `scripts/benchmark_encoding.py` | Modify | Update import path |
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
## Testing Strategy
### Immediate Testing
- Run `make test` after reorganization - all existing tests must pass
- Verify Poetry can build/install the package correctly
### Test Migration Decision
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
- `unit/correction/` - Correction logic tests (important)
- `unit/lyrics/` - Lyrics provider tests
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
- `integration/` - Integration tests
- `performance/` - Performance benchmarks
**Recommendation:** Migrate a subset of critical unit tests, particularly:
- `test_corrector.py` - Core correction logic
- `test_anchor_sequence*.py` - Anchor sequence matching
- Handler tests in `handlers/`
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
## Open Questions
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
## Rollback Plan
If issues arise:
1. Git revert the consolidation commits
2. `lyrics_transcriber_temp/` can be restored from git history
3. The external python-lyrics-transcriber repo can be unarchived if needed
## Notes
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
- Only the internal project structure changes, not the API
- This is primarily a cleanup/organization task with some external repo admin work
@@ -1,110 +0,0 @@
# Plan: Disable Auto-Correction (Agentic and Heuristic)
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1914-disable-auto-correction
**Status:** Draft
## Overview
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
## Requirements
- [x] Disable agentic AI correction (LLM-based via Gemini)
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
- [x] Preserve the transcription → human review → video generation flow
- [x] Minimal code changes (configuration-based if possible)
- [x] Easy to re-enable in the future if correction quality improves
## Technical Approach
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
**Current behavior:**
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
**Proposed changes:**
### Option A: Add a new `skip_correction` config (Recommended)
Add a single boolean `skip_correction` setting that:
- When `true`: Skips the entire correction process, returning raw transcription
- When `false` (future): Falls back to existing agentic/heuristic behavior
This is cleaner than disabling both systems separately because:
1. Single config controls both
2. Clear intent ("skip correction" vs "disable two different systems")
3. Easy to understand for future maintainers
### Option B: Just disable both individually
- Set `use_agentic_ai` default to `false`
- Disable default heuristic handlers by setting `enabled_handlers=[]`
Option A is cleaner, but Option B requires fewer code changes.
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
## Implementation Steps
1. [x] **Add `skip_correction` config to backend/config.py**
- Add `skip_correction: bool` setting, default `true`
- Comment explaining this disables both agentic and heuristic correction
2. [x] **Update lyrics_worker.py to pass skip_correction**
- Pass the config value through to LyricsProcessor/LyricsTranscriber
3. [x] **Update corrector.py to support skip_correction**
- When `skip_correction=True`, return immediately with uncorrected segments
- No handler processing, no agentic AI
- Still need to do anchor/gap finding? No - skip entirely
4. [x] **Update LyricsProcessor to support skip_correction**
- Pass through to LyricsTranscriber
5. [x] **Update LyricsTranscriber controller to support skip_correction**
- Pass through to corrector or skip corrector.run() entirely
6. [x] **Update docs/README.md**
- Note that auto-correction is disabled
- Update "Agentic AI correction" status to "Disabled"
7. [x] **Run tests to verify no regressions**
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
| `docs/README.md` | Modify | Update status to reflect disabled correction |
## Testing Strategy
- **Unit tests**: No new tests needed - this is a config change
- **Existing tests**: Run `make test` to ensure no regressions
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
## Open Questions
None - the approach is straightforward.
## Rollback Plan
To re-enable auto-correction:
1. Set `SKIP_CORRECTION=false` environment variable, OR
2. Change the default back to `false` in `backend/config.py`
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
## Alternative Considered
Instead of adding a new config, we could:
1. Set `USE_AGENTIC_AI=false` (disables agentic)
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
@@ -1,99 +0,0 @@
# Plan: Encoding Worker Immutable Deployment Pattern
## Status: In Progress
## Problem Statement
The GCE encoding worker has recurring version mismatch issues because:
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
2. Wheel selection uses `sort -V` which can fail if image is outdated
3. CI only restarts service - doesn't verify version strictly
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
5. 51 old wheel files accumulate in GCS bucket
## Solution: Immutable Deployment Pattern
### Core Principles
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
3. **Version Manifest**: CI writes expected version, service MUST match
4. **Strict Verification**: CI FAILS if deployed version doesn't match
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
## Implementation Steps
### Phase 1: GCS Structure & Scripts
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
- [x] **Step 1.4**: Create README.md with documentation
### Phase 2: CI Workflow Updates
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
- [x] **Step 2.3**: Update CI to write version.txt manifest
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
### Phase 3: Packer Image Updates
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
### Phase 4: Documentation & Cleanup
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
- [x] **Step 4.2**: Update infrastructure/packer/README.md
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
## New GCS Structure
```
gs://karaoke-gen-storage-nomadkaraoke/
├── wheels/
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
│ └── karaoke_gen-{version}.whl # Versioned for rollback
│
└── encoding-worker/
├── bootstrap.sh # Reference copy
├── startup.sh # CI-managed, downloaded on boot
└── version.txt # Expected version number
```
## Files to Create/Modify
| File | Action | Purpose |
|------|--------|---------|
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
| `.github/workflows/ci.yml` | Modify | New deployment flow |
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
## Testing Strategy
1. **Unit test**: Verify startup.sh logic locally
2. **Integration test**: After merge, verify:
- CI uploads files to correct GCS paths
- Service starts with correct version
- Health endpoint returns expected version
3. **Manual verification**: Check VM logs after first deployment
## Rollback Plan
If issues occur:
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
2. Previous versioned wheels remain in GCS for manual rollback
3. Old Packer images remain for VM rollback
## Open Questions
None - design is complete.
## Notes
- Bootstrap.sh is ~10 lines and should never need changes
- All logic changes go through startup.sh in GCS
- Version verification is strict - CI fails on mismatch
@@ -1,166 +0,0 @@
# Plan: Fix YouTube Remote Download Missing GCS Path
**Created:** 2026-01-20
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
**Status:** Complete
## Overview
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
## Root Cause
In `backend/api/routes/audio_search.py` lines 354-363:
```python
# Use download_by_id if we have source_id and remote is enabled
if source_id and source_name and is_remote_enabled:
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
target_file=target_file,
download_url=download_url,
# <-- MISSING: gcs_path parameter
)
```
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
## Requirements
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
- [x] Backend should receive a `gs://` path from flacfetch
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
- [x] Maintain backwards compatibility (local-only deployments should still work)
- [x] Add test coverage for this code path
## Technical Approach
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
The current code structure has two branches:
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
- If remote disabled: download locally and upload to GCS from Cloud Run
## Implementation Steps
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
4. [x] Update logging to clarify the download path being used
5. [x] Bump version in `pyproject.toml`
6. [x] Run tests to verify no regressions
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
| `pyproject.toml` | Modify | Bump version |
## Detailed Code Changes
### audio_search.py (lines 349-393)
Current structure:
```python
else:
# Local download (YouTube or fallback) <-- WRONG: not always local!
temp_dir = tempfile.mkdtemp(...)
if source_id and source_name and is_remote_enabled:
result = audio_search_service.download_by_id(
...
output_dir=temp_dir, # <-- BUG: unused for remote
# Missing gcs_path!
)
```
Fixed structure:
```python
else:
# Non-torrent download (YouTube or fallback)
if is_remote_enabled and source_id and source_name:
# Remote download via flacfetch VM - upload directly to GCS
gcs_destination = f"uploads/{job_id}/audio/"
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
result = audio_search_service.download_by_id(
source_name=source_name,
source_id=source_id,
output_dir="", # Not used for remote
target_file=target_file,
download_url=download_url,
gcs_path=gcs_destination, # <-- FIX: Add this!
)
# Handle GCS path response (same logic as torrent branch)
if result.filepath.startswith("gs://"):
parts = result.filepath.replace("gs://", "").split("/", 1)
if len(parts) == 2:
audio_gcs_path = parts[1]
else:
audio_gcs_path = result.filepath
filename = os.path.basename(result.filepath)
else:
# Fallback: upload manually
filename = os.path.basename(result.filepath)
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
with open(result.filepath, 'rb') as f:
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
else:
# Local download (remote disabled or no source_id)
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
# ... existing local download logic ...
```
## Testing Strategy
### Unit Tests
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
```python
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
"""
Test that YouTube downloads via remote flacfetch include GCS path.
This is the same bug pattern as the torrent download bug fixed in 2025-12:
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
"""
# Setup: YouTube source with remote enabled
# Verify: download_by_id called with gcs_path parameter
```
### Manual Testing (post-deploy)
1. Create a job and select a YouTube audio source
2. Verify job completes successfully
3. Check GCS path is properly set in job data
## Open Questions
None - the fix is straightforward and follows the pattern established for torrent sources.
## Rollback Plan
This is a minor code change. If issues occur:
1. Revert the PR
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
## Related Documentation
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
-407
View File
@@ -1,407 +0,0 @@
# Plan: Add Retry Logic for Flacfetch Download Failures
**Created:** 2026-02-10
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
**Status:** Draft
## Overview
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
Investigation revealed **multiple related issues** when flacfetch network calls fail:
1. **No retry logic** - A single network hiccup causes immediate job failure
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
3. **Broken retry button** - Can't retry because no source info was persisted
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
### Root Cause
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
### Solution
This plan fixes all four issues:
- Add retry logic to flacfetch client (network resilience)
- Persist download info before attempting download (enables retry)
- Update retry endpoint to handle audio search jobs
- Show error messages in frontend when retry fails
## Requirements
### Functional Requirements - Flacfetch Retry Logic
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
- [x] Use exponential backoff spread over ~5 minutes
- [x] Configurable max retries via environment variable
- [x] Log each retry attempt with context
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
- [x] Do retry on HTTP 5xx errors (server errors - transient)
### Functional Requirements - Persist Download Info
- [x] Save source_id, source_name, target_file, download_url to job document
- [x] Save BEFORE attempting download (not after)
- [x] Store in top-level fields for easy retry access
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
### Functional Requirements - Retry Endpoint
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
- [x] Check for saved download params (source_id, source_name)
- [x] Re-download using saved params
- [x] Clear error state before retrying
- [x] Trigger appropriate workers after successful download
### Functional Requirements - Frontend Error Handling
- [x] Show toast notification when retry fails
- [x] Display actual error message from API
- [x] Don't just console.error - make it visible to user
### Non-Functional Requirements
- [x] Minimal code changes (use existing patterns)
- [x] Clear error messages that include retry context
- [x] Backwards compatible (no breaking changes)
- [x] Should not significantly increase total timeout duration
## Technical Approach
### Architecture Decisions
**Option 1: Tenacity library (Recommended)**
- ✅ Industry standard, well-tested
- ✅ Declarative retry configuration
- ✅ Built-in exponential backoff
- ✅ Already used in project (check `poetry.lock`)
- ❌ New dependency if not already present
**Option 2: Manual retry loop**
- ✅ No new dependencies
- ✅ Full control over logic
- ❌ More code to maintain
- ❌ Easy to get wrong (edge cases)
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
### Retry Strategy
```python
# Retry on network errors only (RequestError)
# Do NOT retry on:
# - HTTP 4xx (client errors like 400, 401, 404)
# - Already completed downloads
# Do retry on:
# - httpx.RequestError (connection timeout, refused, DNS failure)
# - HTTP 5xx (server errors - service temporarily down)
Max retries: 8 (total 9 attempts)
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
- Attempt 1: immediate
- Attempt 2: wait 10s
- Attempt 3: wait 20s
- Attempt 4: wait 40s
- Attempt 5: wait 60s (capped)
- Attempt 6: wait 60s (capped)
- Attempt 7: wait 60s (capped)
- Attempt 8: wait 60s (capped)
- Attempt 9: wait 60s (capped)
Total max time: ~370s (6+ minutes) of backoff + request timeouts
This gives the flacfetch service time to recover from restarts,
network issues, or temporary VM problems.
```
### Configuration
Add to `backend/config.py`:
```python
# Flacfetch client retry settings
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
```
### Error Message Improvements
Change from:
```
Download by ID request failed:
```
To:
```
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
```
## Implementation Steps
### Phase 1: Check Dependencies
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
3. [ ] Run `poetry lock` if dependencies changed
### Phase 2: Add Configuration
4. [ ] Add retry configuration to `backend/config.py`
- `flacfetch_retry_max_attempts = 9`
- `flacfetch_retry_min_wait = 10.0`
- `flacfetch_retry_max_wait = 60.0`
### Phase 3: Implement Flacfetch Retry Logic
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
6. [ ] Create retry decorator with:
- Retry on `httpx.RequestError`
- Retry on `httpx.HTTPStatusError` with 5xx status
- Stop on `httpx.HTTPStatusError` with 4xx status
- Exponential backoff with cap at 60s
- Before-sleep logging callback
7. [ ] Apply decorator to `download_by_id()` method
8. [ ] Improve error messages to include:
- Number of attempts made
- Original exception type and message
### Phase 4: Persist Download Info Before Download
9. [ ] Update `audio_search.py:_download_and_start_processing()`
10. [ ] After getting source info from selected result (line 307-311)
11. [ ] **Save to job document BEFORE download attempt:**
```python
job_manager.update_job(job_id, {
'audio_source_type': 'audio_search',
'source_name': source_name,
'source_id': source_id,
'target_file': target_file,
'download_url': download_url,
})
```
12. [ ] Then proceed with download (existing code)
13. [ ] Add to Job model if needed (check if fields exist)
### Phase 5: Update Retry Endpoint for Audio Search
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
15. [ ] Add new case before "no input audio" error (line ~1088)
16. [ ] Check if job has saved download params:
```python
elif job.audio_source_type == 'audio_search' and job.source_id:
# Has audio search download info - retry download
```
17. [ ] Re-call audio search download logic with saved params
18. [ ] Or refactor download into reusable function
### Phase 6: Frontend Error Handling
19. [ ] Update `frontend/components/job/JobActions.tsx`
20. [ ] Add toast notification library (check if exists)
21. [ ] In `handleRetry()` catch block, show toast:
```typescript
catch (error) {
const message = error.response?.data?.detail || "Failed to retry job"
toast.error(message)
console.error("Failed to retry job:", error)
}
```
### Phase 7: Testing - Flacfetch Retry
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
24. [ ] Test no retry on 4xx errors
25. [ ] Test retry on 5xx errors
26. [ ] Test max retries exhausted
27. [ ] Verify logging output includes attempt numbers
### Phase 8: Testing - End-to-End
28. [ ] Test audio search job with flacfetch down
29. [ ] Verify source info is saved even when download fails
30. [ ] Verify retry button works after source info saved
31. [ ] Test retry succeeds when flacfetch comes back up
32. [ ] Verify error toast appears when retry fails
### Phase 9: Documentation
33. [ ] Update `backend/services/flacfetch_client.py` docstring
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
- "Always add retries for external service calls"
- "Save operation parameters before attempting operations"
- "Persist enough state to enable retry from any failure point"
35. [ ] Update `docs/ARCHITECTURE.md` if needed
36. [ ] Document the new job fields in API.md
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/config.py` | Modify | Add retry configuration variables |
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
| `docs/API.md` | Modify | Document new job fields |
## Testing Strategy
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
1. **Test retry on RequestError**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_network_error():
"""Should retry 3 times on connection errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
# Fail 2 times, then succeed
mock_post.side_effect = [
httpx.RequestError("Connection timeout"),
httpx.RequestError("Connection refused"),
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
]
result = await client.download_by_id("YouTube", "abc123")
assert result == "test123"
assert mock_post.call_count == 3
```
2. **Test no retry on 4xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_no_retry_on_client_error():
"""Should NOT retry on 400/401/404 errors."""
client = FlacfetchClient(base_url="http://test", api_key="key")
with patch("httpx.AsyncClient.post") as mock_post:
response = Mock(status_code=404, text="Not found")
response.raise_for_status.side_effect = httpx.HTTPStatusError(
"Not found", request=Mock(), response=response
)
mock_post.return_value = response
with pytest.raises(FlacfetchServiceError, match="404"):
await client.download_by_id("YouTube", "abc123")
assert mock_post.call_count == 1 # No retries
```
3. **Test retry on 5xx errors**
```python
@pytest.mark.asyncio
async def test_download_by_id_retries_on_server_error():
"""Should retry on 500/503 errors."""
# Similar to test 1 but with HTTP 503 errors
```
4. **Test max retries exhausted**
```python
@pytest.mark.asyncio
async def test_download_by_id_fails_after_max_retries():
"""Should fail with clear message after max retries."""
# All attempts fail, verify error message includes attempt count
```
### Integration Tests
5. **Test with actual flacfetch service** (if available in test environment)
- Verify retry doesn't break normal operation
- Test with deliberate network issues
### Manual Testing
6. **Simulate network failure**
```bash
# Temporarily stop flacfetch VM
gcloud compute instances stop flacfetch-service --zone=us-central1-a
# Submit a test job
# Observe logs showing retry attempts
# Restart VM during retries
gcloud compute instances start flacfetch-service --zone=us-central1-a
# Job should succeed
```
7. **Test with intentional delays**
- Add sleep in flacfetch API handler
- Verify retries don't cause issues
## Open Questions
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
- [ ] Should we also add retries to `wait_for_download()`?
- Probably NO - that already has its own polling/timeout logic
- [ ] Should we add circuit breaker pattern?
- Probably NO for now - keep it simple
- Could add in future if we see cascading failures
- [ ] Should retry logic be applied to other flacfetch methods?
- `start_download()` - probably YES (same pattern)
- `search()` - maybe YES (search can also have network issues)
- `get_download_status()` - probably NO (polling already handles this)
- **Decision**: Apply to `start_download()` too, skip `search()` for now
- [ ] Which toast library should frontend use?
- Check if already exists (react-hot-toast, sonner, etc.)
- If not, add lightweight library
- [ ] Should we save download params to state_data or top-level fields?
- Top-level fields more visible and easier to query
- **Decision**: Top-level fields (source_id, source_name, etc.)
## Rollback Plan
If this causes issues:
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
- Disables retries without code change
- Can deploy in ~2 minutes
2. **Full rollback**: Revert the PR
- Git revert the merge commit
- Deploy previous version
3. **Monitoring**: Watch for:
- Increased request duration to flacfetch
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
- Duplicate download attempts causing issues
## Success Criteria
- [ ] Jobs with transient flacfetch failures succeed instead of failing
- [ ] Error messages are more informative (show attempts and root cause)
- [ ] No increase in false positives (jobs that should fail, now hang)
- [ ] All tests pass
- [ ] No impact on successful jobs (retry overhead is minimal)
## Related Issues
### Job 749141f8 - The Incident
**What happened:**
- User searched for audio via audio search
- Selected "Elizabeth Moen - Songbird"
- Backend tried to download via flacfetch
- Flacfetch call failed with network error (connection timeout/refused)
- Job immediately failed
- No source info saved → Retry button broken
- User sees "Retry" button but clicking does nothing (silent 400 error)
**How this fix helps:**
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
2. ✅ Persist download info → Even if all retries fail, user can retry later
3. ✅ Retry endpoint fix → Retry button will actually work
4. ✅ Frontend toast → User sees helpful error if retry still fails
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
### Future Work (Not in This PR)
Similar pattern should be applied to other external service calls:
- AudioShake API (lyrics transcription)
- Modal API (audio separation)
- YouTube direct API calls (not via flacfetch)
- Email sending (SendGrid)
## References
- [Tenacity documentation](https://tenacity.readthedocs.io/)
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
@@ -1,149 +0,0 @@
# Plan: Improve Job Observability
**Created:** 2026-03-07
**Branch:** feat/sess-20260307-2216-improve-job-observability
**Status:** Draft
## Overview
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
## Problem Analysis
### What we found investigating two edited jobs:
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
## Requirements
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
- [ ] No breaking changes to existing timeline format (additive only)
## Technical Approach
### 1. Add `metadata` field to `TimelineEvent`
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
Key events that gain metadata:
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
### 2. Write Firestore job logs from API endpoints
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
```python
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
"""Write a log entry to the job's Firestore log subcollection."""
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
FirestoreService().append_log_to_subcollection(job_id, entry)
```
### 3. Enhance distribution logging in video_worker_orchestrator
After each distribution upload succeeds, log the details with structured metadata:
- YouTube: video_id, video_url, channel
- Dropbox: folder_path, sharing_link
- GDrive: file_ids dict
- Brand code: code, prefix, allocated vs recycled
### 4. Snapshot previous outputs on edit
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
### 5. Log cleanup results from edit/admin endpoints
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
## Implementation Steps
### Step 1: Add `metadata` to `TimelineEvent` model
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
### Step 2: Create `log_to_job` helper
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
- [ ] Catches and suppresses errors (logging should never break the caller)
### Step 3: Update `transition_to_state` to accept metadata
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
### Step 4: Enhance edit endpoint logging
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
### Step 5: Enhance admin delete-outputs endpoint logging
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
### Step 6: Enhance distribution logging in video_worker_orchestrator
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
### Step 7: Add output details to `complete` timeline event
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
### Step 8: Tests
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
- [ ] Unit test: `log_to_job` writes to subcollection
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
- [ ] Unit test: `complete` transition includes output metadata
- [ ] Verify existing tests still pass
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
## Testing Strategy
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
## What This Does NOT Include
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
## Rollback Plan
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
## Open Questions
None — the approach is straightforward and low-risk.
@@ -1,218 +0,0 @@
# Plan: Job Email Notifications & Admin Message Tools
**Created:** 2026-01-06
**Branch:** feat/sess-20260106-0116-add-email-button
**Status:** Draft
## Overview
Add comprehensive email notification system for karaoke-gen that:
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
2. **Uses cloud-stored templates** for the job completion message
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
## Requirements
### Functional Requirements
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
- [ ] Store email templates in GCS (editable without code deploy)
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
### Non-Functional Requirements
- [ ] Email sending must not block job completion (async/background)
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
- [ ] Template changes should take effect without restart (no long-term caching)
## Technical Approach
### 1. Template Storage (GCS)
Store templates in `gs://karaoke-gen-storage/templates/`:
- `job-completion.txt` - Plain text template for job completion
- `job-completion.html` - HTML version (optional, for rich emails)
- `action-needed-lyrics.txt` - Reminder for lyrics review
- `action-needed-instrumental.txt` - Reminder for instrumental selection
**Template Variables:**
- `{name}` - User's display name or "there" if unknown
- `{youtube_url}` - YouTube video URL
- `{dropbox_url}` - Dropbox folder URL
- `{artist}` - Artist name
- `{title}` - Song title
- `{job_id}` - Job ID
- `{review_url}` - Lyrics review URL (for action-needed emails)
- `{instrumental_url}` - Instrumental selection URL
- `{feedback_url}` - Feedback form URL (for completion emails)
### 2. Email Service Extensions
Extend `backend/services/email_service.py`:
- Add `send_job_completion()` method with CC support
- Add `send_action_reminder()` method
- Add `render_template()` helper to fetch & render GCS templates
### 3. Auto-Email Triggers
**Job Completion:**
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
- Implementation: Call email service async in background task
**Action Reminders (5-min idle detection):**
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
- Record `blocking_state_entered_at` timestamp in `state_data`
- Schedule a Cloud Tasks task for 5 minutes later
- The scheduled task checks if job is still in blocking state and user hasn't started review
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
### 4. Admin UI (Frontend)
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
- "Copy Message" button - fetches rendered template, copies to clipboard
- "Send Email" button - opens dialog for email input, calls API to send
### 5. Backend API Endpoints
New endpoints in `backend/api/routes/admin.py`:
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
New internal endpoint for scheduled reminder:
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
## Implementation Steps
### Phase 1: Template Infrastructure
1. [ ] Upload initial template to GCS from user's existing template file
2. [ ] Create `template_service.py` to fetch/render templates from GCS
3. [ ] Add unit tests for template rendering
### Phase 2: Email Extensions
4. [ ] Extend `email_service.py` with CC support in `send_email()`
5. [ ] Add `send_job_completion()` method
6. [ ] Add `send_action_reminder()` method
7. [ ] Add unit tests for new email methods
### Phase 3: Auto-Email on Completion
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
9. [ ] Add integration test for completion email trigger
### Phase 4: Idle Reminder System
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
12. [ ] Schedule 5-minute delayed task when entering blocking states
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
14. [ ] Add tests for idle reminder flow
### Phase 5: Admin API
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
17. [ ] Add tests for admin endpoints
### Phase 6: Admin UI
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
19. [ ] Add "Copy Message" button (admin-only) with loading state
20. [ ] Add "Send Email" button (admin-only) with email input dialog
21. [ ] Add E2E test for admin buttons
### Phase 7: Documentation
22. [ ] Document template management in docs/DEVELOPMENT.md
23. [ ] Update docs/API.md with new endpoints
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
## Testing Strategy
**Unit Tests:**
- Template rendering with all variable combinations
- Email service CC support
- Completion/reminder email generation
- Idle detection logic
**Integration Tests:**
- Full flow: job completion → email sent
- Idle reminder: job enters blocking state → 5 min → reminder sent
- Admin endpoint authentication
**Manual Testing:**
- Verify emails render correctly in Gmail/Outlook
- Test admin buttons on completed job
- Test clipboard copy works across browsers
## Open Questions
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
## Rollback Plan
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
2. **Template fetch failures** - Fall back to hardcoded template in code
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
## Template Content
Initial template (from user's file, with feedback link added):
```
Hi {name},
Thanks for your order!
Here's the link for the karaoke video published to YouTube:
{youtube_url}
Here's the dropbox folder with all the finished files and source files, including:
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
- "(Title).mov"/"(End).mov": title card and end screen videos
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
- lyrics/*.txt song lyrics from various sources in plain text format
{dropbox_url}
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
{feedback_url}
Thanks again and have a great day!
-Andrew
```
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
## Updating Templates
To update email templates after deployment:
```bash
# Upload new template
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
# Templates are fetched fresh on each use (no caching in code)
# Changes take effect immediately
```
-279
View File
@@ -1,279 +0,0 @@
# Plan: Private (Non-Published) Tracks
**Created:** 2026-02-24
**Branch:** feat/sess-20260224-1149-private-tracks
**Status:** Implemented
## Overview
Add a `is_private` flag to jobs that changes distribution behavior:
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
- **YouTube**: Skipped entirely
- **Google Drive**: Skipped entirely
- **Email**: Completion email includes Dropbox link but no YouTube section
- **Discord**: Still notified (for internal tracking)
Available to all users at job creation and toggleable by admin on existing jobs.
## Requirements
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
- [ ] Discord notifications still fire for private jobs
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
## Technical Approach
### Core Design: Single Boolean Flag with Runtime Overrides
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
- **Admin can flip one field** and everything adapts
### Key Helper Function
```python
# In job_defaults_service.py
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
"""Return distribution settings, applying private overrides if is_private=True."""
if getattr(job, 'is_private', False):
settings = get_settings()
return EffectiveDistributionSettings(
dropbox_path=settings.default_private_dropbox_path,
brand_prefix=settings.default_private_brand_prefix,
enable_youtube_upload=False,
gdrive_folder_id=None,
# Keep other settings from job (discord, youtube_description, etc.)
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=None,
)
# Non-private: return job's own settings
return EffectiveDistributionSettings(
dropbox_path=getattr(job, 'dropbox_path', None),
brand_prefix=getattr(job, 'brand_prefix', None),
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
youtube_description=getattr(job, 'youtube_description_template', None),
)
```
This helper is used in both distribution paths (orchestrator and native fallback).
### Distribution Paths That Need Changes
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
### Email Template Change
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
```python
# Remove YouTube section if no URL provided
if not variables.get("youtube_url"):
result = re.sub(
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
'\n',
result,
flags=re.DOTALL
)
variables["youtube_url"] = ""
```
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
### Admin Toggle Auto-Delete
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
## Implementation Steps
### Phase 1: Backend Model & Config (foundation)
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
- Add `is_private: bool = False` field after `made_for_you` field
- Add to `JobCreate` model as well
2. [ ] **Add private distribution config** (`backend/config.py`)
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
### Phase 2: Backend API (job creation + admin)
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
- Accept and pass through `is_private` field to `JobCreate`
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
- Accept `is_private` Form parameter
- Pass through to `JobCreate`
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
- Accept `is_private` in `AudioSearchRequest`
- Pass through to `JobCreate`
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
### Phase 3: Distribution Pipeline
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
### Phase 4: Email Template
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
### Phase 5: Frontend
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
- Show for all users (not just admin)
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
- Add `is_private?: boolean` to all job creation option interfaces
- Add `is_private` to `JobUpdateRequest` interface
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
- Show current state with visual indicator (e.g., lock icon for private jobs)
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
### Phase 6: Environment Configuration
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
- These should be configured in Cloud Run service and CI test environment
### Phase 7: Testing
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
- Test `get_effective_distribution_for_job()` with private and non-private jobs
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
- Test `_upload_to_dropbox()` uses private path and brand prefix
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
- Test `_handle_native_distribution()` respects private overrides
- Test non-private jobs are completely unaffected (regression tests)
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
- Test YouTube section removed from email when youtube_url is None
- Test YouTube section present when youtube_url is provided
- Test Dropbox section still present for private jobs
- Test full email rendering for private job scenario
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
- Test admin can update is_private field
- Test toggling is_private to True on completed job triggers output deletion
- Test toggling is_private to True on non-completed job does NOT delete outputs
- Test toggling is_private to False does nothing special
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
- Test private checkbox appears in job creation form
- Test private toggle in admin dashboard
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
## Testing Strategy
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
- `get_effective_distribution_for_job()` with private=True and False
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
- Native distribution fallback applies same overrides
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
### Email Template Tests (~5 tests)
- YouTube section stripped when youtube_url is None
- YouTube section preserved when youtube_url has value
- Dropbox section always present
- Full private job email rendering (Dropbox + no YouTube)
- Full non-private job email rendering (both present) - regression
### Admin Toggle Tests (~5 tests)
- is_private editable via PATCH
- Toggle to private on completed job deletes outputs
- Toggle to private on pending/in-progress job does NOT delete
- Toggle to non-private does nothing special
### Frontend Tests
- E2E: Private checkbox visible and functional in job creation
- E2E: Admin toggle visible and includes confirmation dialog
### Manual Verification
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
- Verify completion email has Dropbox link but no YouTube section
- Toggle existing completed job to private, verify outputs deleted
## Regression Risk Analysis
| Risk | Mitigation |
|------|-----------|
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
## Open Questions
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
- [x] Discord? → **Still notify** (useful for internal tracking)
## Rollback Plan
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
@@ -1,239 +0,0 @@
# Plan: Song Lookup & Autocomplete + Community Version Detection
**Created:** 2026-03-02
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
**Status:** Implementation Complete
## Overview
Add two features to the karaoke-gen job creation flow:
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
## Requirements
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
- [ ] Community version check triggers when both artist and title are filled
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
- [ ] Community check does not block job submission — just informs the user
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
## Architecture Decisions
### How to access karaoke-decide catalog data
**Decision: Proxy through karaoke-gen backend**
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
| Option | Pros | Cons |
|--------|------|------|
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
### How to detect community karaoke versions
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
### Autocomplete UI component
**Decision: Use existing shadcn Command (cmdk) + Popover components**
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
- Keyboard navigation (arrow keys, enter to select)
- Debounced search-as-you-type
- Clean dropdown with loading states
- Works inline within forms (not modal)
## Technical Approach
### Data flow
```
User types in artist field
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
→ user selects "Fleetwood Mac" → field populated with canonical name
User types in title field
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
→ user selects "Dreams" → field populated with canonical title
Both fields filled → community check fires
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
→ if community tracks found with YouTube URLs, return them
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
```
### Caching strategy
| Endpoint | Cache TTL | Scope |
|----------|-----------|-------|
| Artist search proxy | 5 min | In-memory (backend) |
| Track search proxy | 5 min | In-memory (backend) |
| Community check | 1 hour | In-memory (backend) |
### karaoke-decide API endpoints used
These are all **public, no auth required**:
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
## Implementation Steps
### Phase 1: Backend — Catalog proxy + community check endpoints
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
- HTTP client (httpx async) calling karaoke-decide API
- In-memory TTL cache for responses (5 min)
- Configurable karaoke-decide base URL
- Graceful error handling (return empty results on failure)
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
- Add async support (httpx instead of requests)
- In-memory TTL cache (1 hour)
- Return only community tracks with YouTube URLs
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
- `POST /api/catalog/community-check` — scrape karaokenerds.com
- All endpoints require auth (same as job creation)
- Pydantic request/response models
4. [ ] **Register routes in main.py** — add `catalog.router` to app
5. [ ] **Add config** — `backend/config.py`
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
### Phase 2: Frontend — Autocomplete component
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
- Wraps an Input field with a Popover + Command dropdown
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
- Debounced search (300ms) triggered on input change
- Shows loading spinner while fetching
- Shows "No results" when search returns empty
- Keyboard navigation (up/down arrows, enter to select, escape to close)
- Clicking outside closes dropdown
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
- When user types without selecting, uses their raw text (autocomplete is optional)
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
### Phase 3: Frontend — Integrate autocomplete into job creation
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
- Fetch suggestions from `api.searchArtists(query)`
- On select: populate artist field with canonical name
- Show artist name + optional disambiguation in dropdown items
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
- On select: populate title field with canonical name
- Show track name + artist name + duration in dropdown items
### Phase 4: Frontend — Community version detection
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
- Shows when community karaoke version(s) found
- Displays: "A karaoke version of this song already exists!"
- Lists community tracks with brand name and YouTube link
- "Watch on YouTube" button for top community version
- Dismissible (X button or "Continue anyway" link)
- Styled with green border (matching kjbox community track styling)
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
- Debounced check (500ms) when both artist AND title fields have values
- Only triggers when values change (not on every keystroke)
- Shows `CommunityVersionBanner` between the form fields and the submit button
- Does NOT block form submission
- Clears when artist or title changes
### Phase 5: Testing
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
- Test proxy endpoints with mocked karaoke-decide responses
- Test community check with mocked karaokenerds HTML
- Test error handling (karaoke-decide down, karaokenerds down)
- Test caching behavior
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
- Test HTML parsing, community detection, YouTube URL cleanup
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
- Test suggestion fetching and display
- Test keyboard navigation
- Test selection behavior
- Test free-text input (no selection)
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
- Test autocomplete appears on typing
- Test selecting from autocomplete populates field
- Test community banner appears for known songs
- Test form submission works with and without autocomplete selection
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
| `backend/main.py` | Modify | Register catalog router |
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
## Decisions (Resolved)
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
## Rollback Plan
- Feature is entirely additive — no existing functionality is modified
- Autocomplete fields fall back to plain text Input if API calls fail
- Community check is non-blocking — if it fails, form works normally
- All new endpoints are behind the existing auth middleware
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
@@ -1,329 +0,0 @@
# Plan: Unified Combined Review for Local CLI
**Created:** 2026-01-24
**Branch:** feat/sess-20260124-1444-combined-review-flow
**Status:** Draft
## Problem Statement
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
- Review code lives inside `correct_lyrics()` method in controller.py
- When auto-correction is disabled, `correct_lyrics()` is never called
- Users never get a chance to review lyrics
2. **Standalone instrumental UI still exists for CLI**
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
- CLI launches this standalone server after lyrics review
- The combined review flow we built only works for the web app
## Goals
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
4. **Keep it simple** - Minimal changes to achieve the goal
## Architecture Overview
### Current State (Broken)
```
CLI Flow:
controller.py::process()
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
→ ReviewServer starts (lyrics only)
→ User reviews
→ ReviewServer stops
→ gen_cli.py::run_instrumental_review()
→ InstrumentalReviewServer starts (instrumental only)
→ User selects instrumental
→ InstrumentalReviewServer stops
Cloud Web App Flow:
Job enters AWAITING_REVIEW
→ User visits /app/jobs/{id}/review
→ Combined UI: lyrics + instrumental
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
```
### Target State
```
CLI Flow (and Cloud CLI):
controller.py::process()
→ _run_human_review() [NEW - always called when review enabled]
→ ReviewServer starts (combined: lyrics + instrumental)
→ User reviews lyrics AND selects instrumental
→ ReviewServer stops, returns both results
→ Continue with selected instrumental
Cloud Web App Flow: (unchanged)
Same as before
```
## Key Insight
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
3. Call the review flow from the right place (not inside `correct_lyrics()`)
4. Delete `InstrumentalReviewServer` entirely
## Implementation Steps
### Phase 1: Fix ReviewServer to Support Instrumental Selection
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
```python
def __init__(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None, # NEW
backing_vocals_analysis: dict | None = None, # NEW
...
):
```
Update `_correction_data()` endpoint to include these in response:
```python
@app.get("/api/review/{job_id}/correction-data")
def _correction_data(job_id: str):
return {
"lyrics_data": self.correction_result.to_dict(),
"instrumental_options": self.instrumental_options, # NEW
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
}
```
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
File: `karaoke_gen/lyrics_transcriber/review/server.py`
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
```python
@app.post("/api/review/{job_id}/complete-review")
def _complete_review(job_id: str, request: ReviewCompleteRequest):
# Existing: store corrected lyrics
self.corrected_lyrics = request.corrected_lyrics
# NEW: store instrumental selection
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
self.review_completed = True
```
**1.3. Expose instrumental selection after review completes**
The calling code needs to get the selection. Update the return value or add an attribute:
```python
# After review loop completes:
return {
"corrected_lyrics": self.corrected_lyrics,
"instrumental_selection": self.instrumental_selection
}
```
### Phase 2: Fix controller.py to Call Review at the Right Time
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Create new method `_run_human_review()` that can be called independently:
```python
def _run_human_review(
self,
correction_result: CorrectionResult,
instrumental_options: list[dict] | None = None,
backing_vocals_analysis: dict | None = None
) -> tuple[CorrectionResult, str | None]:
"""Run human review server and return corrected result + instrumental selection."""
if not self.output_config.review_enabled:
return correction_result, None
server = ReviewServer(
correction_result=correction_result,
instrumental_options=instrumental_options,
backing_vocals_analysis=backing_vocals_analysis,
...
)
result = server.start() # Blocks until review complete
return result["corrected_lyrics"], result.get("instrumental_selection")
```
**2.2. Update `process()` to always call review when appropriate**
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
Current flow (simplified):
```python
def process(self):
self.transcribe()
self.separate_audio()
if self.output_config.run_correction:
self.correct_lyrics() # Review is inside here!
# ...
```
New flow:
```python
def process(self):
self.transcribe()
self.separate_audio()
# Build correction result (with or without auto-correction)
if self.output_config.run_correction:
correction_result = self._run_auto_correction()
else:
correction_result = self._create_uncorrected_result()
# Human review is separate from auto-correction
if self.output_config.review_enabled:
instrumental_options = self._get_instrumental_options()
backing_vocals_analysis = self._analyze_backing_vocals()
corrected, instrumental_selection = self._run_human_review(
correction_result,
instrumental_options,
backing_vocals_analysis
)
self.results.correction_result = corrected
self.results.instrumental_selection = instrumental_selection
```
### Phase 3: Update gen_cli.py to Use Combined Review
**3.1. Remove standalone instrumental review call**
File: `karaoke_gen/utils/gen_cli.py`
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
Before:
```python
# After lyrics processing
if not args.non_interactive:
instrumental_selection = run_instrumental_review(...)
```
After:
```python
# Instrumental selection comes from controller.results.instrumental_selection
instrumental_selection = controller.results.instrumental_selection
```
**3.2. Pass instrumental options to controller**
The controller needs instrumental options to display in review. These come from the separated audio stems:
```python
controller = LyricsTranscriber(
# ... existing params ...
)
# Before calling process(), set up instrumental options
# These are typically discovered during audio separation
```
### Phase 4: Delete Standalone InstrumentalReviewServer
**4.1. Delete files**
- Delete `karaoke_gen/instrumental_review/server.py`
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
**4.2. Clean up imports**
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
### Phase 5: Update Frontend for Local Mode
**5.1. Verify local mode detection works**
File: `frontend/lib/api.ts`
The frontend already has local mode detection based on hostname and port. Verify it works:
```typescript
const isLocalMode = () => {
if (typeof window === 'undefined') return false;
const hostname = window.location.hostname;
const port = parseInt(window.location.port, 10);
return hostname === 'localhost' && port >= 8000 && port <= 8770;
};
```
**5.2. Verify correction-data endpoint parsing**
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
### Phase 6: Testing
**6.1. Manual testing checklist**
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
- [ ] Combined UI shows both lyrics editor and instrumental selector
- [ ] Selecting instrumental and completing review works
- [ ] No standalone instrumental review page appears
- [ ] Cloud web app still works (no regression)
**6.2. Update unit tests**
- Remove tests for `InstrumentalReviewServer`
- Add tests for `ReviewServer` with instrumental options
- Add tests for `controller._run_human_review()`
## Files to Modify
| File | Action | Description |
|------|--------|-------------|
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
| Missing instrumental options in local mode | Fail clearly if options not provided |
| Review server port conflicts | Keep existing port allocation logic |
## Open Questions
1. **Where does backing vocals analysis run for local CLI?**
- Currently runs in `render_video_worker.py` for cloud
- For local, need to run it after audio separation, before review
- May need to call analysis function from controller
2. **How are instrumental stems exposed to local review?**
- Need to serve audio files from ReviewServer
- Or use file:// URLs (may have CORS issues)
- Existing approach: ReviewServer already serves static files
## Success Criteria
1. `SKIP_CORRECTION=true` no longer skips human review
2. Single combined review UI appears (no separate instrumental step)
3. All platforms (CLI, cloud CLI, web) use same review flow
4. No dead code remaining for standalone instrumental review
-317
View File
@@ -1,317 +0,0 @@
# Plan: White Label B2B Karaoke Portals
**Created:** 2026-01-08
**Branch:** feat/sess-20260108-0358-white-label-portal
**Status:** Draft
## Overview
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
**First customer: Vocal Star**
- Subdomain: `vocalstar.nomadkaraoke.com`
- Workflow: Upload original + instrumental audio (no audio search)
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
- Distribution: Download only (no Dropbox/GDrive/YouTube)
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
## Requirements
### Functional Requirements
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
- [ ] Per-tenant frontend branding (logo, colors, metadata)
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
- [ ] Per-tenant default theme (always use Vocal Star theme)
- [ ] Per-tenant allowed email domains for magic link auth
- [ ] Per-tenant fixed API token support (for automation)
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
- [ ] Users scoped to tenant
### Non-Functional Requirements
- [ ] Minimal code duplication - single codebase serves all tenants
- [ ] Easy to add new tenants via config, not code changes
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
- [ ] No performance impact on existing single-tenant flow
## Technical Approach
### Architecture Decision: Config-Driven Multi-Tenancy
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
2. **Frontend detects subdomain** and fetches tenant config from API
3. **Backend middleware** extracts tenant from request and applies config
4. **Firestore collections** get `tenant_id` field for isolation
This approach:
- Requires no database migrations for new tenants
- Configs can be managed via simple JSON files
- Easy to test locally with query param override (`?tenant=vocalstar`)
- Can evolve to admin UI later if needed
### Tenant Config Schema
```json
{
"id": "vocalstar",
"name": "Vocal Star",
"subdomain": "vocalstar.nomadkaraoke.com",
"branding": {
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
"primary_color": "#ffff00",
"secondary_color": "#006CF9",
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
"site_title": "Vocal Star Karaoke Generator"
},
"features": {
"audio_search": false,
"file_upload": true,
"youtube_upload": false,
"dropbox_upload": false,
"gdrive_upload": false,
"theme_selection": false
},
"defaults": {
"theme_id": "vocalstar",
"distribution_mode": "download_only"
},
"auth": {
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
"fixed_tokens": ["<will be in Secret Manager>"]
}
}
```
### Key Implementation Details
#### 1. Subdomain Detection (Frontend)
- Next.js middleware checks `request.headers.host`
- If subdomain matches tenant, store tenant ID in cookie/context
- Fetch `/api/tenant/config` to get branding/features
- Apply CSS variables dynamically for colors
- Swap logo component based on config
#### 2. Tenant Middleware (Backend)
- FastAPI middleware extracts tenant from:
- `X-Tenant-ID` header (set by frontend)
- Or subdomain from `Host` header
- Or query param `?tenant=` (dev only)
- Attaches `tenant_id` to request state
- Tenant config loaded from GCS (cached 5 min like themes)
#### 3. Auth Enhancement
- `AuthService.validate_token_full()` already returns `user_email`
- Add tenant validation: check if user's email domain is in tenant's allowed list
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
- Magic links include tenant context for proper redirect
#### 4. Data Isolation
- Add `tenant_id: Optional[str]` to User and Job models
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
- Query filters: `where('tenant_id', '==', tenant_id or None)`
- No schema migration needed - new field, optional
#### 5. Theme Integration
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
- Tenant config specifies `default_theme_id: "vocalstar"`
- When `theme_selection: false`, UI hides theme picker and uses default
## Implementation Steps
### Phase 1: Backend Tenant Foundation (Day 1-2)
1. [ ] **Create tenant model and config loader**
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
- Create `backend/services/tenant_service.py` with GCS config loading and caching
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
2. [ ] **Add tenant middleware**
- Create `backend/api/middleware/tenant.py`
- Extract tenant from subdomain or header
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
- Add `get_tenant()` dependency for routes
3. [ ] **Update data models**
- Add `tenant_id: Optional[str] = None` to `User` model
- Add `tenant_id: Optional[str] = None` to `Job` model
- Update Firestore indexes in Pulumi for tenant-scoped queries
4. [ ] **Update auth for tenant awareness**
- Modify `AuthService` to validate email domain against tenant config
- Add tenant context to magic link generation
- Support per-tenant fixed tokens in `auth_tokens` collection
### Phase 2: Upload Vocal Star Theme (Day 2)
5. [ ] **Prepare Vocal Star theme for GCS**
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
- Upload background images to `themes/vocalstar/assets/`
- Upload Oswald font to `themes/vocalstar/assets/`
- Upload `style_params.json` to `themes/vocalstar/`
- Create preview image and upload to `themes/vocalstar/preview.png`
- Add Vocal Star to `themes/_metadata.json`
6. [ ] **Create Vocal Star tenant config**
- Create `tenants/vocalstar/config.json` in GCS
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
- Upload favicon to `tenants/vocalstar/favicon.ico`
### Phase 3: Frontend Tenant Support (Day 3-4)
7. [ ] **Add tenant context provider**
- Create `frontend/lib/tenant.ts` with tenant config types
- Create `frontend/hooks/useTenant.ts` for tenant state
- Create `TenantProvider` component that fetches config on mount
- Detect subdomain in `_app.tsx` or middleware
8. [ ] **Dynamic branding**
- Create `TenantLogo` component that renders tenant or default logo
- Update `globals.css` to support CSS variable overrides
- Add `applyTenantColors()` function to set CSS variables from config
- Update page metadata (title, favicon) from tenant config
9. [ ] **Feature-flagged UI**
- Wrap audio search in `{tenant.features.audio_search && ...}`
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
- Wrap distribution options based on tenant features
- Show simplified form for Vocal Star (just file upload fields)
10. [ ] **Update magic link flow**
- Pass tenant context when requesting magic link
- Magic link redirect goes to tenant subdomain
- Verify endpoint extracts tenant from link context
### Phase 4: Backend Feature Enforcement (Day 4-5)
11. [ ] **Enforce tenant features in API**
- Audio search endpoint: return 403 if `audio_search: false`
- File upload: check `file_upload: true`
- Distribution settings: ignore/error if disabled features requested
- Theme selection: use default if `theme_selection: false`
12. [ ] **Job creation with tenant context**
- `JobCreate` handler sets `tenant_id` from request state
- Distribution config respects tenant defaults
- Theme defaults to tenant's default theme
13. [ ] **Query scoping**
- Update `JobManager.get_jobs()` to filter by tenant_id
- Update `UserService` queries to include tenant_id
- Admin endpoints bypass tenant filter (see all tenants)
### Phase 5: Infrastructure & Deployment (Day 5-6)
14. [ ] **DNS setup**
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
- Configure Cloudflare Pages to accept wildcard subdomain
- Add SSL certificate for subdomain
15. [ ] **Backend CORS update**
- Add `vocalstar.nomadkaraoke.com` to allowed origins
- Or use pattern matching for `*.nomadkaraoke.com`
16. [ ] **Pulumi updates**
- Add Firestore indexes for tenant_id queries
- Add any new secrets (tenant fixed tokens)
### Phase 6: Testing & Documentation (Day 6-7)
17. [ ] **Testing**
- Unit tests for TenantService config loading
- Unit tests for tenant middleware
- E2E test for Vocal Star flow (upload audio, generate, download)
- Test magic link works for @vocal-star.com email
18. [ ] **Documentation**
- Update CLAUDE.md with tenant info
- Create `docs/WHITE-LABEL.md` with tenant setup guide
- Document how to add new tenants
## Files to Create/Modify
| File | Action | Description |
|------|--------|-------------|
| `backend/models/tenant.py` | Create | Tenant config models |
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
| `backend/models/user.py` | Modify | Add `tenant_id` field |
| `backend/models/job.py` | Modify | Add `tenant_id` field |
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
| `backend/main.py` | Modify | Add tenant middleware |
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
## Testing Strategy
### Unit Tests
- `TenantService`: Config loading, caching, validation
- `TenantMiddleware`: Subdomain extraction, header extraction
- `AuthService`: Tenant email domain validation
- `JobManager`: Tenant-scoped queries
### Integration Tests (Emulator)
- Create job with tenant context
- Query jobs filtered by tenant
- Auth with tenant-allowed email domain
- Auth rejection for wrong email domain
### E2E Tests
- Vocal Star happy path: Upload audio files, review lyrics, download video
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
- Feature hiding: Verify audio search not shown on Vocal Star portal
## Open Questions
- [x] **Q: Should tenant configs be in GCS or Firestore?**
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
- [x] **Q: How to handle localhost development for specific tenant?**
A: Query param override `?tenant=vocalstar` for local testing
- [x] **Q: Should admin portal be tenant-aware?**
A: No - admins see all tenants. Admin routes bypass tenant filter.
- [x] **Q: Do we need tenant-specific email sender identity?**
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
## Rollback Plan
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
1. Revert frontend to not check tenant
2. Revert backend middleware to not extract tenant
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
4. No schema migration to reverse
For DNS: Simply remove CNAME record if needed.
## Success Criteria
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
- [ ] Portal shows Vocal Star branding (logo, colors)
- [ ] Audio search is hidden; only file upload available
- [ ] Theme is locked to Vocal Star style
- [ ] Distribution options hidden; download only
- [ ] Magic link works for @vocal-star.com emails
- [ ] Fixed API token works for automation
- [ ] Generated videos use Vocal Star theme
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
-58
View File
@@ -1,58 +0,0 @@
#!/usr/bin/env bash
# Nomad Karaoke pre-commit hook (karaoke-gen variant)
# 1. Gitleaks secret scan (fail fast)
# 2. i18n auto-translation when en.json changes
# Enable: git config core.hooksPath .githooks
set -eu
# ---------------------------------------------------------------------------
# 1. Gitleaks secret scan
# ---------------------------------------------------------------------------
if command -v gitleaks >/dev/null 2>&1; then
if ! gitleaks protect --staged --redact --no-banner; then
printf '\n\033[31m✗ gitleaks detected a potential secret in staged changes.\033[0m\n' >&2
printf ' • If real: remove it, rotate the credential, then re-commit.\n' >&2
printf ' • If false positive: add an allowlist entry to .gitleaks.toml.\n' >&2
printf ' • Emergency override (not recommended): git commit --no-verify\n' >&2
exit 1
fi
else
printf '\033[33mwarn:\033[0m gitleaks not installed — skipping secret scan\n' >&2
printf ' install: brew install gitleaks\n' >&2
fi
# ---------------------------------------------------------------------------
# 2. i18n auto-translation
# ---------------------------------------------------------------------------
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
if [ -z "$STAGED_EN" ]; then
exit 0
fi
echo "i18n: English translations changed — running translation pipeline..."
for en_file in $STAGED_EN; do
dir=$(dirname "$en_file")
if [[ "$dir" == frontend/messages ]]; then
script="frontend/scripts/translate.py"
elif [[ "$dir" == backend/translations ]]; then
script="frontend/scripts/translate.py"
else
echo " Warning: Unknown messages dir $dir, skipping"
continue
fi
echo " Translating $dir..."
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
git add "$dir"/*.json
echo " Done: $dir"
else
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
fi
done
echo "i18n: Translation complete."
@@ -1,115 +0,0 @@
# Build Encoding Worker GCE Image
#
# Rebuilds the custom Packer image when provisioning changes.
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
#
# Triggers:
# - Push to main with changes to infrastructure/packer/**
# - Manual dispatch (workflow_dispatch)
#
# After image build, the encoding-worker VM will use the new image
# on next recreation (e.g., via pulumi up or manual restart).
name: Build Encoding Worker Image
on:
push:
branches: [main]
paths:
- 'infrastructure/packer/**'
workflow_dispatch:
inputs:
python_version:
description: 'Python version to install'
required: false
default: '3.13.1'
concurrency:
group: build-encoding-worker-image
cancel-in-progress: false # Don't cancel in-progress builds
jobs:
build-image:
name: Build Packer Image
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # Required for Workload Identity Federation
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Authenticate to Google Cloud (Workload Identity)
uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
- name: Set up Cloud SDK
uses: google-github-actions/setup-gcloud@v2
with:
version: "latest"
- name: Install Packer
run: |
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y packer
- name: Initialize Packer
working-directory: infrastructure/packer
run: packer init encoding-worker.pkr.hcl
- name: Validate Packer Template
working-directory: infrastructure/packer
run: packer validate encoding-worker.pkr.hcl
- name: Build Image
working-directory: infrastructure/packer
env:
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
run: |
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
packer build \
-var "python_version=${PYTHON_VERSION}" \
encoding-worker.pkr.hcl
- name: Get Latest Image Info
id: image-info
run: |
IMAGE_NAME=$(gcloud compute images list \
--project=nomadkaraoke \
--filter="family:encoding-worker" \
--sort-by="~creationTimestamp" \
--limit=1 \
--format="value(name)")
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
echo "Built image: ${IMAGE_NAME}"
- name: Summary
run: |
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
notify-on-failure:
name: Notify on Failure
runs-on: ubuntu-latest
needs: build-image
if: failure()
steps:
- name: Log Failure
run: |
echo "::error::Encoding worker image build failed. Check the build-image job logs."
+119 -762
View File
File diff suppressed because it is too large. Load diff
-149
View File
@@ -1,149 +0,0 @@
name: DR backup freshness monitor
# Off-platform monitor: runs in GitHub (not GCP) so it survives a full GCP
# compromise. If the nightly backup pipeline dies silently, this catches it.
#
# Required secrets:
# AWS_BACKUP_READONLY_ACCESS_KEY_ID
# AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
# DR_MONITOR_DISCORD_WEBHOOK
#
# Setup: see docs/DISASTER-RECOVERY.md § "Backup freshness monitor".
on:
schedule:
# Daily at 14:00 UTC (~9h after the 1am ET backup window).
- cron: "0 14 * * *"
workflow_dispatch: {}
permissions:
contents: read
jobs:
check-freshness:
runs-on: ubuntu-latest
timeout-minutes: 5
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BACKUP_READONLY_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BACKUP_READONLY_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-1
BUCKET: nomadkaraoke-backup
DISCORD_WEBHOOK: ${{ secrets.DR_MONITOR_DISCORD_WEBHOOK }}
steps:
- name: Verify required secrets are set
run: |
missing=0
for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY DISCORD_WEBHOOK; do
if [ -z "${!var}" ]; then
echo "::error::Required secret $var is empty"
missing=1
fi
done
[ "$missing" = "0" ] || exit 1
- name: Install boto3
run: pip install --quiet boto3
- name: Check freshness of each tracked prefix
id: check
shell: bash
run: |
python3 - <<'PY'
import os, sys, datetime, boto3
BUCKET = os.environ["BUCKET"]
# (label, prefix, max_age_hours)
TARGETS = [
("Firestore", "firestore/", 36),
("GCS job files", "gcs/job-files/", 36),
("Encrypted secrets", "secrets/", 36),
("BigQuery weekly", "bigquery/daily-refresh/", 192),
]
s3 = boto3.client("s3")
now = datetime.datetime.now(datetime.timezone.utc)
stale, ok = [], []
# Paginated scan — JMESPath --query on `aws s3api list-objects-v2`
# operates per-page, which silently breaks "newest across all pages"
# queries on prefixes with >1000 objects (e.g. firestore/ has ~22k).
paginator = s3.get_paginator("list_objects_v2")
for label, prefix, max_hours in TARGETS:
latest = None
for page in paginator.paginate(Bucket=BUCKET, Prefix=prefix):
for obj in page.get("Contents", []):
if latest is None or obj["LastModified"] > latest:
latest = obj["LastModified"]
if latest is None:
stale.append(f"{label}: NO OBJECTS in s3://{BUCKET}/{prefix}")
continue
age = (now - latest).total_seconds()
age_h = int(age // 3600)
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
(stale if age > max_hours * 3600 else ok).append(line)
summary = ["## DR backup freshness", ""]
if stale:
summary.append("### Stale")
summary.extend(f"- {s}" for s in stale)
summary.append("")
summary.append("### Fresh")
summary.extend(f"- {o}" for o in ok)
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
f.write("\n".join(summary) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
f.write(f"ok_count={len(ok)}\n")
f.write(f"stale_count={len(stale)}\n")
if stale:
f.write("STALE_LIST<<EOF\n")
f.write("\n".join(stale) + "\n")
f.write("EOF\n")
for o in ok:
print("OK:", o)
if stale:
for s in stale:
print("STALE:", s, file=sys.stderr)
sys.exit(1)
PY
- name: Alert Discord on stale backup
if: failure() && steps.check.outputs.stale_count != ''
env:
STALE: ${{ steps.check.outputs.STALE_LIST }}
run: |
payload=$(jq -n \
--arg title "🚨 DR backup is stale" \
--arg desc "$STALE" \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15158332,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
- name: Alert Discord on workflow failure (catch-all)
if: failure() && steps.check.outputs.stale_count == ''
run: |
payload=$(jq -n \
--arg title "⚠️ DR freshness monitor failed to run" \
--arg desc "Could not list S3 backup bucket — credentials, network, or bucket policy may be broken." \
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
'{
embeds: [{
title: $title,
description: $desc,
color: 15844367,
fields: [{name: "Workflow run", value: $url}]
}]
}')
curl -fsS -X POST -H "Content-Type: application/json" \
-d "$payload" "$DISCORD_WEBHOOK"
-337
View File
@@ -1,337 +0,0 @@
name: E2E Daily Test (Payment + Happy Path)
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
on:
schedule:
- cron: '0 6 * * *'
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
skip_payment:
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
required: false
default: 'false'
type: boolean
use_test_token:
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
# =========================================================================
# Stage 1: Purchase credits via real Stripe Checkout
# =========================================================================
e2e-credit-purchase:
name: "Stage 1: Credit Purchase"
runs-on: ubuntu-latest
timeout-minutes: 15
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
outputs:
session_token: ${{ steps.extract_token.outputs.token }}
result: ${{ steps.test_result.outputs.result }}
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Run Credit Purchase Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
echo "Time: $(date)"
npx playwright test credit-purchase-real.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=600000 \
2>&1 | tee test-output-payment.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "CREDIT PURCHASE TEST PASSED" test-output-payment.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Extract session token
id: extract_token
if: steps.test_result.outputs.result == 'success'
working-directory: frontend
run: |
if [ -f test-results/e2e-session-token.txt ]; then
TOKEN=$(cat test-results/e2e-session-token.txt)
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
echo "Session token extracted for Stage 2"
else
echo "No session token file found"
echo "token=" >> $GITHUB_OUTPUT
fi
- name: Upload artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-payment-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/test-output-payment.log
retention-days: 30
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Stage 2: Happy Path (uses credit from Stage 1)
# =========================================================================
e2e-happy-path:
name: "Stage 2: Happy Path"
runs-on: ubuntu-latest
timeout-minutes: 90
needs: e2e-credit-purchase
if: |
always() && (
needs.e2e-credit-purchase.result == 'success' ||
github.event.inputs.skip_payment == 'true' ||
github.event.inputs.use_test_token == 'true'
)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Ensure testmail.app domain is allowlisted
run: |
curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json" \
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
- name: Refresh E2E session token (when use_test_token enabled)
id: refresh_token
if: github.event.inputs.use_test_token == 'true'
run: |
RESPONSE=$(curl -sf -X POST \
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
-H "Content-Type: application/json")
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Run Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "=== Stage 2: Happy Path ==="
echo "Time: $(date)"
TOKEN_SOURCE="none"
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
TOKEN_SOURCE="use_test_token (impersonation)"
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
TOKEN_SOURCE="Stage 1 (credit purchase)"
fi
echo "Token source: $TOKEN_SOURCE"
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output-happy-path.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output-happy-path.log && ! grep -q "failed" test-output-happy-path.log; then
echo "result=success" >> $GITHUB_OUTPUT
else
echo "result=failure" >> $GITHUB_OUTPUT
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output-happy-path.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
- name: Discord notification (failure)
if: steps.test_result.outputs.result == 'failure'
run: |
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
# =========================================================================
# Final: Send notifications
# =========================================================================
notify:
name: "Notify"
runs-on: ubuntu-latest
needs: [e2e-credit-purchase, e2e-happy-path]
if: always()
steps:
- name: Discord notification (success)
if: needs.e2e-happy-path.result == 'success'
run: |
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
curl -s -H "Content-Type: application/json" \
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
- name: Send email notification
if: github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Daily Test Results
======================
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+173
View File
@@ -0,0 +1,173 @@
name: E2E Happy Path Test
# Comprehensive end-to-end test that validates the full karaoke generation journey
# using ONLY browser interactions (no API shortcuts)
#
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
on:
# Daily run at 6 AM UTC
schedule:
- cron: '0 6 * * *'
# Manual trigger with optional parameters
workflow_dispatch:
inputs:
debug_mode:
description: 'Enable debug mode (slower, more screenshots)'
required: false
default: 'false'
type: boolean
browser:
description: 'Browser to test with'
required: false
default: 'chromium'
type: choice
options:
- chromium
- firefox
- webkit
use_test_token:
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
required: false
default: 'false'
type: boolean
permissions:
contents: read
jobs:
e2e-happy-path:
name: "Full Happy Path (Real User)"
runs-on: ubuntu-latest
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: frontend/package-lock.json
- name: Install dependencies
working-directory: frontend
run: npm ci --legacy-peer-deps
- name: Install Playwright browsers
working-directory: frontend
run: |
BROWSER=${{ inputs.browser || 'chromium' }}
npx playwright install --with-deps $BROWSER
- name: Create test-results directory
working-directory: frontend
run: mkdir -p test-results
- name: Run E2E Happy Path Test
working-directory: frontend
env:
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
run: |
echo "Starting E2E Happy Path Test"
echo "Browser: ${{ inputs.browser || 'chromium' }}"
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
echo "Time: $(date)"
echo ""
# Run the real user happy path test
npx playwright test happy-path-real-user.spec.ts \
--config=playwright.production.config.ts \
--project=${{ inputs.browser || 'chromium' }} \
--reporter=list \
--timeout=3600000 \
2>&1 | tee test-output.log
continue-on-error: true
- name: Check test result
id: test_result
working-directory: frontend
run: |
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
echo "result=success" >> $GITHUB_OUTPUT
echo "Test PASSED"
else
echo "result=failure" >> $GITHUB_OUTPUT
echo "Test FAILED - see artifacts for details"
fi
- name: Upload test artifacts
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-results-${{ github.run_number }}
path: |
frontend/test-results/
frontend/playwright-report/
frontend/test-output.log
retention-days: 30
- name: Upload video recordings
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-happy-path-videos-${{ github.run_number }}
path: frontend/test-results/**/*.webm
retention-days: 30
if-no-files-found: ignore
- name: Create summary
if: always()
run: |
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
if [ -f frontend/test-output.log ]; then
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
fi
- name: Send email notification
if: always() && github.event_name == 'schedule'
uses: dawidd6/action-send-mail@v3
with:
server_address: smtp.sendgrid.net
server_port: 587
username: apikey
password: ${{ secrets.SENDGRID_API_KEY }}
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
to: andrew@nomadkaraoke.com
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
body: |
E2E Happy Path Test Results
===========================
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
Run: #${{ github.run_number }}
Trigger: ${{ github.event_name }}
Time: ${{ github.event.repository.updated_at }}
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
This is an automated notification from the daily E2E happy path test.
- name: Fail if test failed
if: steps.test_result.outputs.result == 'failure'
run: exit 1
-10
View File
@@ -1,10 +0,0 @@
name: Security
on:
pull_request:
push:
branches: [main]
jobs:
gitleaks:
uses: nomadkaraoke/.github/.github/workflows/gitleaks.yml@main
+2 -6
View File
@@ -3,7 +3,6 @@
.vscode
*.csv
*.mp3
ABBA*
# Andrew functional adds
/audio/
@@ -13,9 +12,6 @@ tests/outputs
/input
/output/
# Local karaoke-gen CLI test output dirs (created during manual validation)
/test-output-*/
# Cloud Infra
.emulator-logs
@@ -62,7 +58,8 @@ frontend/.eslintcache
# Storybook
frontend/storybook-static/
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
# PWA files
frontend/**/public/sw.js
frontend/**/public/workbox-*.js
frontend/**/public/swe-worker-*.js
@@ -235,4 +232,3 @@ flacfetch
# Test output directories
Rick Astley*/
.superpowers/
-60
View File
@@ -1,60 +0,0 @@
# Nomad Karaoke shared gitleaks config.
# Extends the default ruleset with one custom rule for Discord webhooks
# and an allowlist covering known false-positive paths.
title = "Nomad Karaoke"
[extend]
useDefault = true
# ---------------------------------------------------------------------------
# Custom rules
# ---------------------------------------------------------------------------
[[rules]]
id = "discord-webhook-url"
description = "Discord webhook URL (token embedded in URL)"
regex = '''https://(?:discord|discordapp)\.com/api/webhooks/\d{17,20}/[A-Za-z0-9_-]{60,}'''
tags = ["discord", "webhook", "secret"]
# ---------------------------------------------------------------------------
# Global allowlist
# ---------------------------------------------------------------------------
[allowlist]
description = "Paths and patterns that are known false positives"
paths = [
'''(?i)\.(jpg|jpeg|png|gif|bmp|svg|webp|pdf|ico|mp3|mp4|wav|flac|m4a)$''',
'''(^|/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|poetry\.lock|Podfile\.lock|Cargo\.lock|go\.sum|composer\.lock|Gemfile\.lock|uv\.lock)$''',
'''(^|/)node_modules/''',
'''(^|/)vendor/''',
'''(^|/)wp-content/''',
'''(^|/)\.dart_tool/''',
'''(^|/)\.venv/''',
'''(^|/)\.next/''',
'''(^|/)build/''',
'''(^|/)dist/''',
'''(^|/)\.flutter-plugins''',
'''(^|/)test[-_]?fixtures?/''',
'''(^|/)tests?/.*conftest\.py$''',
'''(^|/)tests?/.*test_[^/]+\.py$''',
'''(^|/)tests?/.*_test\.py$''',
'''(^|/).*\.(test|spec)\.(js|mjs|cjs|ts|tsx|jsx)$''',
'''(^|/).*\.test\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
'''(^|/).*\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
]
regexes = [
# Explicit placeholder patterns
'''YOUR_[A-Z_]{2,}''',
'''EXAMPLE_[A-Z_]{2,}''',
'''PLACEHOLDER_[A-Z_]{2,}''',
'''CHANGE[_-]?ME''',
'''<[A-Z_]{2,}>''',
'''\$\{[A-Z_]{2,}\}''',
# Firebase / GCP client API keys — public by design (identify the project,
# access is gated by Firebase Security Rules).
'''AIzaSy[A-Za-z0-9_-]{33}''',
]
-1
View File
@@ -1 +0,0 @@
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
-1
View File
@@ -1 +0,0 @@
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
+14 -75
View File
@@ -16,19 +16,14 @@
| Current status | `docs/README.md` |
| Architecture | `docs/ARCHITECTURE.md` |
| Dev setup & testing | `docs/DEVELOPMENT.md` |
| **Testing & code quality** | `docs/TESTING.md` |
| API reference | `docs/API.md` |
| Past learnings | `docs/LESSONS-LEARNED.md` |
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
## Tech Stack
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
- **Frontend**: Next.js on Cloudflare Pages
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
## Essential Rules
@@ -39,78 +34,24 @@
- Create PR with summary, changes, testing info
- Merge only after CI passes
### Testing (Required)
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
- Which test types are needed (unit, integration, E2E)
- Where tests should be placed
- What mocking approach to use
- Coverage expectations
- **Production E2E tests for user-facing features** - see below
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
**Before committing**, run all tests:
### Testing (Required Before Commit)
```bash
# Backend (10 min timeout, limit output)
make test 2>&1 | tail -n 500
# Frontend
cd frontend && npm run test:all 2>&1 | tail -n 200
```
This single command:
- Installs dependencies automatically (backend + frontend)
- Runs all backend tests (unit, integration, emulator)
- Runs all frontend tests (Jest unit + Playwright E2E)
- Takes ~5-10 minutes
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
### Testing in Production (for Agents)
When asked to "test it yourself in prod" or verify a fix in production:
```bash
# Get admin token
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
# Option 1: Use the debug template directly
cd frontend && node e2e/helpers/debug-prod-template.mjs
# Option 2: Copy template for customization (gitignored)
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
# Edit the script, then run it
node test-my-issue.local.mjs
# Option 3: Run existing production E2E tests
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
```
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
### Version Bumping
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
- Skip for docs-only changes
### Infrastructure
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
- Never modify GCP resources directly via console or `gcloud` CLI
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
- All GCP changes via Pulumi in `infrastructure/`
- `gcloud` CLI for reading/debugging only
- Stop and notify user on auth issues
### Internationalization (i18n)
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
- Components use `useTranslations('namespace')` from next-intl
- Pages are under `frontend/src/app/[locale]/` — locale-aware routing
- Admin pages (under `/admin/`, `/app/`) use DefaultIntlProvider (English-only, excluded from i18n)
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
## API Authentication (for Agents)
When you need to call backend APIs programmatically:
@@ -144,16 +85,14 @@ EOF
### GCE Encoding Worker
```bash
# Check health (includes wheel_version)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
# Restart service (pick up new wheel after a deploy)
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
# SSH to restart service (clears in-memory job queue)
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="sudo systemctl restart encoding-worker"
```
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
# Check health
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
--command="curl -s http://localhost:8080/health"
```
### What Doesn't Work
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
+15 -73
View File
@@ -1,77 +1,34 @@
# Makefile for karaoke-gen project
# Run `make help` to see available commands
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
# Per-stage timeout in seconds (default: 10 minutes)
# Prevents hangs from stale dev servers, broken emulators, etc.
# Override with: make test TEST_TIMEOUT=300
TEST_TIMEOUT ?= 600
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
# Default target
help:
@echo "Available commands:"
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
@echo " make test-backend - Run backend tests only (unit + emulator)"
@echo " make test-frontend - Run frontend unit tests (Jest)"
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
@echo " make test-e2e - Run emulator tests with auto-start/stop"
@echo " make install - Install all dependencies (backend + frontend)"
@echo " make build-frontend - Build frontend and copy to Python package"
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
@echo " make test - Run all tests (unit + backend + emulator-based)"
@echo " make test-unit - Run unit tests only (karaoke_gen)"
@echo " make test-backend - Run backend unit tests only"
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
@echo " make test-all - Alias for 'make test'"
@echo " make lint - Run linter checks"
@echo " make emulators-start - Start GCP emulators for local development"
@echo " make emulators-stop - Stop GCP emulators"
@echo ""
@echo "Before committing, run: make test"
# Install dependencies (only if needed)
install-backend:
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
echo "=== Installing backend dependencies ==="; \
poetry install; \
fi
install-frontend:
@if [ ! -d "frontend/node_modules" ]; then \
echo "=== Installing frontend dependencies ==="; \
cd frontend && npm install; \
fi
install: install-backend install-frontend
# Build frontend and copy to Python package
build-frontend: install-frontend
@echo "=== Building frontend ==="
cd frontend && npm run build
@echo "=== Copying build to Python package ==="
rm -rf karaoke_gen/nextjs_frontend/out
mkdir -p karaoke_gen/nextjs_frontend/out
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
# Build frontend + install package in editable mode for local CLI testing
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
dev-install: build-frontend
@echo "=== Installing package in editable mode ==="
pip install -e .
@echo ""
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
# Run unit tests for karaoke_gen package
test-unit: install-backend
test-unit:
@echo "=== Running karaoke_gen unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
# Run backend unit tests (excludes emulator tests)
test-backend-unit: install-backend
# Run backend unit tests
test-backend:
@echo "=== Running backend unit tests ==="
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
# Run E2E integration tests with emulators
test-e2e: install-backend
test-e2e:
@echo "=== Running E2E integration tests with emulators ==="
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
@@ -80,28 +37,13 @@ test-e2e: install-backend
export GCS_BUCKET_NAME=test-bucket && \
export ADMIN_TOKENS=test-admin-token && \
export ENVIRONMENT=test && \
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
poetry run pytest backend/tests/emulator/ -v; \
TEST_RESULT=$$?; \
./scripts/stop-emulators.sh; \
exit $$TEST_RESULT
# Run all backend tests (unit + emulator)
test-backend: test-unit test-backend-unit test-e2e
@echo ""
@echo "✅ Backend tests passed!"
# Run frontend unit tests (Jest only - E2E requires manual setup)
test-frontend: install-frontend
@echo "=== Running frontend unit tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
# Run frontend E2E tests (requires dev server - run manually, not in make test)
test-frontend-e2e: install-frontend
@echo "=== Running frontend E2E tests ==="
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
# Run ALL tests (backend + frontend) - use this before committing!
test: test-backend test-frontend
# Run all tests (use this before committing!)
test: test-unit test-backend test-e2e
@echo ""
@echo "✅ All tests passed! Ready to commit."
+20 -5
View File
@@ -159,9 +159,6 @@ karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
# Generate CDG and TXT packages
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
# Skip video encoding (CDG/TXT only, faster)
karaoke-gen --no-video --enable_cdg "Artist" "Title"
# YouTube upload
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
@@ -406,8 +403,26 @@ gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
### Deploy Cloud Run
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
See `.github/workflows/ci.yml` for the full deployment workflow.
```bash
# Build and deploy
gcloud builds submit --config=cloudbuild.yaml
# Get outputs from Pulumi
SA_EMAIL=$(pulumi stack output service_account_email)
BUCKET_NAME=$(pulumi stack output bucket_name)
# Deploy Cloud Run service
gcloud run deploy karaoke-backend \
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--service-account $SA_EMAIL \
--memory 2Gi \
--cpu 2 \
--timeout 600 \
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
```
### Point CLI to Your Backend
-2
View File
@@ -29,8 +29,6 @@ tmp/
# Google Cloud
.gcloud/
*.json
# Allow i18n translation files
!translations/*.json
# Testing
.pytest_cache/
+4 -11
View File
@@ -22,19 +22,12 @@ WORKDIR /app
COPY backend /app/backend
# Copy and reinstall karaoke_gen from local source
# This ensures new modules are included without rebuilding base image
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
# This ensures new modules (like style_loader.py) are included without rebuilding base image
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY pyproject.toml README.md LICENSE /app/
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
# CPU images: normal install resolves all deps from pyproject.toml.
RUN if [ -n "$MODEL_DIR" ]; then \
pip install --no-cache-dir --no-deps -e .; \
else \
pip install --no-cache-dir -e .; \
fi
RUN pip install --no-cache-dir -e .
# Set environment variables
ENV PYTHONUNBUFFERED=1
+7 -27
View File
@@ -1,15 +1,8 @@
# Base image with all Python dependencies pre-installed
# This image is rebuilt only when poetry.lock or this Dockerfile changes
#
# This image is rebuilt only when poetry.lock changes
#
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. pyproject.toml + package stubs (changes with dep updates)
# 3. pip install (the slow part - cached unless pyproject.toml changes)
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
# 5. spaCy model download (cached unless pip install layer changes)
FROM python:3.11-slim
@@ -23,7 +16,6 @@ RUN apt-get update && apt-get install -y \
build-essential \
curl \
xz-utils \
chromium \
&& rm -rf /var/lib/apt/lists/*
# Install comprehensive font support for international lyrics and symbols
@@ -58,26 +50,14 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
# Copy dependency files and source code needed for pip install
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds, so partial
# dependency updates only download changed packages instead of everything
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Install all Python dependencies (the slow part - cached in base image)
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -e .
# Download spaCy language model for lyrics transcription
# This must be done at build time, not runtime (Cloud Run network restrictions)
-136
View File
@@ -1,136 +0,0 @@
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
# This image is rebuilt only when deps or models change (infrequently).
#
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
#
# Layer caching strategy:
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
# 2. CUDA library path configuration
# 3. pyproject.toml + package stubs (changes with dep updates)
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
# 6. spaCy model download (cached unless pip install layer changes)
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
# Prevent interactive prompts during package installation
ENV DEBIAN_FRONTEND=noninteractive
# Install Python 3.11 from deadsnakes PPA and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes/ppa \
&& apt-get update && apt-get install -y --no-install-recommends \
# Python 3.11
python3.11 \
python3.11-dev \
python3.11-venv \
# Audio libraries
libsndfile1 \
libsox-dev \
sox \
# Build tools
build-essential \
# Utilities
curl \
xz-utils \
&& rm -rf /var/lib/apt/lists/* \
&& python3.11 --version
# Install comprehensive font support for international lyrics and symbols
# Noto fonts provide full Unicode coverage including:
# - Latin, Greek, Cyrillic (core)
# - CJK: Chinese, Japanese, Korean (~150MB)
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
# - Color emoji (limited support in libass, but available)
# - Musical symbols (♪ ♫ etc.)
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
RUN apt-get update && apt-get install -y --no-install-recommends \
fonts-noto-core \
fonts-noto-cjk \
fonts-noto-extra \
fonts-noto-color-emoji \
fontconfig \
&& fc-cache -f \
&& rm -rf /var/lib/apt/lists/*
# Install FFmpeg from Ubuntu package
# The audio worker only uses FFmpeg for format conversion and padding, not
# encoding-intensive tasks, so the Debian package is sufficient. The static
# build from johnvansickle.com is used for the encoding worker instead.
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
# Set Python 3.11 as default and install pip
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
# Environment configuration for CUDA and Python
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
PATH=/usr/local/cuda/bin:$PATH \
PYTHONUNBUFFERED=1 \
MODEL_DIR=/models
# Set working directory
WORKDIR /app
# Copy ONLY dependency specification files first (changes less often than source code)
COPY pyproject.toml README.md LICENSE /app/
# Create minimal package stubs so pip can resolve the editable install
# pip install -e . needs package directories to exist but doesn't validate content
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
# Install all karaoke-gen Python dependencies (the slow part)
# BuildKit cache mount preserves downloaded wheels across builds
RUN --mount=type=cache,target=/root/.cache/pip \
pip install --upgrade pip && \
pip install -e .
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
# This is separate from pip install -e . to keep layer caching granular
#
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
RUN --mount=type=cache,target=/root/.cache/pip \
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
pip install "audio-separator[gpu]"
# Now copy actual source code (AFTER deps installed, so code changes don't
# invalidate the expensive pip install layer in registry cache)
COPY karaoke_gen /app/karaoke_gen
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
COPY backend /app/backend
# Download spaCy language model for lyrics transcription
# Must be done at build time, not runtime (Cloud Run network restrictions)
RUN python -m spacy download en_core_web_sm
# Create models directory and pre-download ensemble preset models
# Baking models into the image eliminates cold-start model download latency.
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
RUN mkdir -p /models
COPY backend/scripts/download_models.py /tmp/download_models.py
RUN python /tmp/download_models.py /models && \
rm /tmp/download_models.py && \
ls -lh /models/
# Label with build info
ARG BUILD_DATE
ARG BUILD_ID
ARG BASE_CONTENT_HASH
LABEL org.opencontainers.image.created="${BUILD_DATE}"
LABEL org.opencontainers.image.revision="${BUILD_ID}"
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
LABEL base.content.hash="${BASE_CONTENT_HASH}"
+114 -14
View File
@@ -113,19 +113,6 @@ async def require_auth(
detail=f"Authentication failed: {auth_result.message}"
)
# If token has a tenant_id, override the hostname-based tenant detection.
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
# We must also load tenant_config so downstream handlers (e.g. job creation)
# that rely on tenant_config for setting tenant_id on jobs work correctly.
if auth_result.tenant_id:
request.state.tenant_id = auth_result.tenant_id
if not getattr(request.state, "tenant_config", None):
from backend.services.tenant_service import get_tenant_service
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
if tenant_config and tenant_config.is_active:
request.state.tenant_config = tenant_config
# Log successful auth with request_id for correlation with request audit
logger.info(
"auth_success",
@@ -135,7 +122,6 @@ async def require_auth(
"user_type": auth_result.user_type.value if auth_result.user_type else None,
"is_admin": auth_result.is_admin,
"remaining_uses": auth_result.remaining_uses,
"tenant_id": auth_result.tenant_id,
"audit_type": "auth_event",
}
)
@@ -355,3 +341,117 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
require_review_auth = require_review_auth_factory()
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
"""
Factory to create an instrumental review authentication dependency.
Accepts either:
1. Full user authentication (admin/user token) - also validates job ownership
2. Job-specific instrumental token (only valid for the specific job)
Args:
job_id_param: Name of the path parameter containing the job ID
Returns:
Dependency function that validates instrumental review access
"""
async def require_instrumental_auth(
request: Request,
job_id: str = Path(...),
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
auth_service: AuthService = Depends(get_auth_service),
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
token: Optional[str] = Query(None, alias="token", description="Full access token")
) -> Tuple[str, str]:
"""
Validate instrumental review access for a job.
Returns:
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
Raises:
HTTPException: 401 if authentication fails
HTTPException: 403 if user doesn't own the job (for full auth)
"""
# Import here to avoid circular dependency
from backend.services.job_manager import JobManager
job_manager = JobManager()
# Try full authentication first
full_token = None
if credentials:
full_token = credentials.credentials
elif token:
full_token = token
if full_token:
auth_result = auth_service.validate_token_full(full_token)
if auth_result.is_valid:
# For full auth, also verify job ownership
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Check ownership: admin can access all, users only their own jobs
if not auth_result.is_admin:
if auth_result.user_email and job.user_email:
if auth_result.user_email.lower() != job.user_email.lower():
logger.warning(
f"Instrumental access denied: user {auth_result.user_email} "
f"tried to access job {job_id} owned by {job.user_email}"
)
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
elif job.user_email:
# Token auth without email trying to access a job with owner
raise HTTPException(
status_code=403,
detail="You don't have permission to access this job's instrumental review"
)
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
return job_id, "full"
# Try instrumental token
if instrumental_token:
job = job_manager.get_job(job_id)
if not job:
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
# Validate instrumental token matches
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
# Check expiry if set
if job.instrumental_token_expires_at:
now = datetime.now(UTC)
# Handle timezone-naive datetimes from Firestore
expiry = job.instrumental_token_expires_at
if expiry.tzinfo is None:
expiry = expiry.replace(tzinfo=UTC)
if now > expiry:
logger.warning(f"Instrumental token expired for job {job_id}")
raise HTTPException(
status_code=401,
detail="Instrumental review token has expired. Please request a new review link."
)
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
return job_id, "instrumental_token"
else:
logger.warning(f"Invalid instrumental token for job {job_id}")
# No valid authentication
raise HTTPException(
status_code=401,
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
)
return require_instrumental_auth
# Default instance for most instrumental review endpoints
require_instrumental_auth = require_instrumental_auth_factory()
File diff suppressed because it is too large. Load diff
File diff suppressed because it is too large. Load diff
+35 -41
View File
@@ -9,7 +9,7 @@ Provides endpoints for:
import logging
from typing import Optional
from datetime import datetime
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
from fastapi import APIRouter, HTTPException, BackgroundTasks
from pydantic import BaseModel
from backend.services.credential_manager import (
@@ -18,7 +18,6 @@ from backend.services.credential_manager import (
CredentialCheckResult,
)
from backend.config import get_settings
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/auth", tags=["auth"])
@@ -126,16 +125,15 @@ async def get_credentials_status():
@router.get("/status/{service}", response_model=CredentialStatusResponse)
async def get_service_credential_status(service: str, request: Request):
async def get_service_credential_status(service: str):
"""
Get the status of a specific service's OAuth credentials.
Args:
service: One of "youtube", "gdrive", "dropbox"
"""
locale = get_locale_from_request(request)
manager = get_credential_manager()
if service == "youtube":
result = manager.check_youtube_credentials()
elif service == "gdrive":
@@ -143,7 +141,7 @@ async def get_service_credential_status(service: str, request: Request):
elif service == "dropbox":
result = manager.check_dropbox_credentials()
else:
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
return CredentialStatusResponse(
service=result.service,
@@ -155,43 +153,42 @@ async def get_service_credential_status(service: str, request: Request):
@router.post("/validate", response_model=CredentialValidationResponse)
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
async def validate_credentials(request: CredentialValidationRequest):
"""
Validate that credentials are available for requested services.
Use this before submitting a job to ensure all required
credentials are valid.
"""
locale = get_locale_from_request(http_request)
manager = get_credential_manager()
invalid_services = []
if request.youtube:
result = manager.check_youtube_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("youtube")
if request.gdrive:
result = manager.check_gdrive_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("gdrive")
if request.dropbox:
result = manager.check_dropbox_credentials()
if result.status != CredentialStatus.VALID:
invalid_services.append("dropbox")
if invalid_services:
return CredentialValidationResponse(
valid=False,
invalid_services=invalid_services,
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
)
return CredentialValidationResponse(
valid=True,
invalid_services=[],
message=t(locale, "auth.validCredentials")
message="All requested credentials are valid"
)
@@ -200,41 +197,40 @@ async def validate_credentials(request: CredentialValidationRequest, http_reques
# ===========================================================================
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start YouTube device authorization flow.
This initiates a device auth flow that allows authorization
from any device. The user must visit the verification URL
and enter the user code.
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/youtube/device/{device_code} to
check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_youtube_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start YouTube device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -263,36 +259,35 @@ async def poll_youtube_device_auth(device_code: str):
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
"""
Start Google Drive device authorization flow.
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
unless explicitly provided in the request body.
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
"""
locale = get_locale_from_request(http_request) if http_request else "en"
manager = get_credential_manager()
try:
device_info = manager.start_gdrive_device_auth(
client_id=request.client_id if request else None,
client_secret=request.client_secret if request else None
)
return DeviceAuthStartResponse(
device_code=device_info.device_code,
user_code=device_info.user_code,
verification_url=device_info.verification_url,
expires_in=device_info.expires_in,
interval=device_info.interval,
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
)
except Exception as e:
logger.error(f"Failed to start Google Drive device auth: {e}")
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
raise HTTPException(status_code=500, detail=str(e))
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
@@ -316,20 +311,19 @@ async def poll_gdrive_device_auth(device_code: str):
# ===========================================================================
@router.post("/test-alert")
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
async def test_credential_alert(background_tasks: BackgroundTasks):
"""
Test the credential alert mechanism.
Sends a test alert to Discord if configured.
"""
locale = get_locale_from_request(request)
settings = get_settings()
discord_url = settings.get_secret("discord-alert-webhook")
if not discord_url:
raise HTTPException(
status_code=400,
detail=t(locale, "auth.discordAlertNotConfigured")
detail="Discord alert webhook not configured"
)
manager = get_credential_manager()
-163
View File
@@ -1,163 +0,0 @@
"""
Catalog API routes for song/artist lookup and community version detection.
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
and checks karaokenerds.com for existing community karaoke versions.
"""
import logging
import time
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth
from backend.services.auth_service import AuthResult
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/catalog", tags=["catalog"])
# Simple per-user rate limiting: 20 requests/minute
# { user_email: [timestamp, timestamp, ...] }
_rate_limit_windows: dict[str, list[float]] = {}
RATE_LIMIT_MAX_REQUESTS = 20
RATE_LIMIT_WINDOW_SECONDS = 60
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
"""Check per-user rate limit. Raises 429 if exceeded."""
now = time.monotonic()
window_start = now - RATE_LIMIT_WINDOW_SECONDS
# Get or create window for this user
if user_email not in _rate_limit_windows:
_rate_limit_windows[user_email] = []
# Remove expired entries
timestamps = _rate_limit_windows[user_email]
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
raise HTTPException(
status_code=429,
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
)
_rate_limit_windows[user_email].append(now)
# --- Response Models ---
class ArtistResult(BaseModel):
"""Artist search result from MusicBrainz/Spotify catalog."""
name: str
mbid: Optional[str] = None
disambiguation: Optional[str] = None
artist_type: Optional[str] = None
spotify_id: Optional[str] = None
popularity: Optional[int] = None
genres: Optional[list[str]] = None
tags: Optional[list[str]] = None
class TrackResult(BaseModel):
"""Track search result from Spotify catalog."""
track_name: str
artist_name: str
track_id: Optional[str] = None
artist_id: Optional[str] = None
popularity: Optional[int] = None
duration_ms: Optional[int] = None
explicit: Optional[bool] = None
class CommunityTrack(BaseModel):
"""A community-approved karaoke track from karaokenerds."""
brand_name: str
brand_code: str
youtube_url: str
is_community: bool = True
class CommunityCheckSong(BaseModel):
"""A song with community karaoke tracks."""
title: str
artist: str
community_tracks: list[CommunityTrack]
class CommunityCheckRequest(BaseModel):
"""Request body for community version check."""
artist: str = Field(..., min_length=1)
title: str = Field(..., min_length=1)
class CommunityCheckResponse(BaseModel):
"""Response for community version check."""
has_community: bool
songs: list[CommunityCheckSong]
best_youtube_url: Optional[str] = None
# --- Routes ---
@router.get("/artists", response_model=list[ArtistResult])
async def search_artists(
request: Request,
q: str = Query(..., min_length=2, description="Artist name search query"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_artists
results = await search_artists(q, limit)
return results
@router.get("/tracks", response_model=list[TrackResult])
async def search_tracks(
request: Request,
q: str = Query(..., min_length=2, description="Track name search query"),
artist: Optional[str] = Query(None, description="Filter by artist name"),
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
auth_result: AuthResult = Depends(require_auth),
):
"""
Search for tracks by name, optionally filtered by artist.
Returns canonical Spotify track data for autocomplete in job creation forms.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.catalog_proxy_service import search_tracks
results = await search_tracks(q, artist, limit)
return results
@router.post("/community-check", response_model=CommunityCheckResponse)
async def check_community_versions(
request: Request,
body: CommunityCheckRequest,
auth_result: AuthResult = Depends(require_auth),
):
"""
Check if a song already has community-approved karaoke versions on YouTube
via karaokenerds.com. Non-blocking — just informs the user.
"""
locale = get_locale_from_request(request)
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
from backend.services.karaokenerds_service import check_community_versions
result = await check_community_versions(body.artist, body.title)
return result
-92
View File
@@ -1,92 +0,0 @@
"""POST /api/client-errors — ingest frontend crash reports.
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
via the existing error-monitor pipeline — see
``backend/services/error_monitor``.
"""
from __future__ import annotations
import logging
import time
from typing import Optional
from fastapi import APIRouter, Request, HTTPException
from pydantic import BaseModel, Field, field_validator
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
from backend.services.error_monitor.frontend_ingestion import (
FrontendErrorReport,
RateLimiter,
build_pattern_data,
)
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
_limiter = RateLimiter(max_per_minute=60)
_adapter_singleton: ErrorPatternsAdapter | None = None
def _get_adapter() -> ErrorPatternsAdapter:
global _adapter_singleton
if _adapter_singleton is None:
_adapter_singleton = ErrorPatternsAdapter()
return _adapter_singleton
class ClientErrorPayload(BaseModel):
message: str = Field(..., min_length=1, max_length=4000)
stack: Optional[str] = Field(None, max_length=1_000_000)
url: str = Field("", max_length=2048)
user_agent: str = Field("", max_length=1024)
release: str = Field("", max_length=64)
user_email: Optional[str] = Field(None, max_length=320)
viewport: Optional[dict] = None
locale: str = Field("en", max_length=10)
source: str = Field("unknown", max_length=64)
# optional free-form extra for debugging. Capped server-side.
extra: Optional[dict] = None
@field_validator("message", "url", "user_agent", "locale", "source")
@classmethod
def strip(cls, v: str) -> str:
return (v or "").strip()
class ClientErrorResponse(BaseModel):
pattern_id: str
is_new: bool
@router.post("", response_model=ClientErrorResponse, status_code=202)
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
client_ip = request.client.host if request.client else "unknown"
if not _limiter.allow(client_ip, time.monotonic()):
raise HTTPException(status_code=429, detail="too many reports")
report = FrontendErrorReport(
message=payload.message,
stack=payload.stack,
url=payload.url,
user_agent=payload.user_agent,
release=payload.release,
user_email=payload.user_email,
viewport=payload.viewport,
locale=payload.locale,
extra=payload.extra,
)
pattern_data = build_pattern_data(report)
try:
result = _get_adapter().upsert_pattern(pattern_data)
except Exception: # pragma: no cover - Firestore transient errors
logger.exception("Failed to upsert frontend error pattern")
raise HTTPException(status_code=503, detail="storage unavailable")
logger.info(
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
result.pattern_id,
result.is_new,
payload.source,
)
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
-65
View File
@@ -1,65 +0,0 @@
"""
Encoding worker lifecycle endpoints.
Provides warmup and heartbeat endpoints for the blue-green
encoding worker VMs. Called by the frontend to ensure the
primary VM is running before encoding requests.
"""
import logging
from fastapi import APIRouter, Depends
from backend.api.dependencies import require_admin
from backend.services.encoding_worker_manager import EncodingWorkerManager
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
# Lazy-loaded singleton
_manager = None
def get_worker_manager():
global _manager
if _manager is None:
from google.cloud import compute_v1, firestore
from backend.config import get_settings
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
_manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
return _manager
@router.post("/warmup")
async def warmup_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Start the primary encoding worker VM if it's stopped."""
try:
result = manager.ensure_primary_running()
if result["started"]:
logger.info(f"Started encoding worker VM: {result['vm_name']}")
return result
except Exception as e:
logger.error(f"Failed to warm up encoding worker: {e}")
return {"started": False, "error": str(e)}
@router.post("/heartbeat")
async def heartbeat_encoding_worker(
_admin=Depends(require_admin),
manager=Depends(get_worker_manager),
):
"""Update activity timestamp to prevent idle shutdown."""
try:
manager.update_activity()
return {"status": "ok"}
except Exception as e:
logger.error(f"Failed to update encoding worker heartbeat: {e}")
return {"status": "error", "error": str(e)}
File diff suppressed because it is too large. Load diff
+3 -370
View File
@@ -5,22 +5,14 @@ import os
import logging
import subprocess
import platform
from fastapi import APIRouter, Depends
from typing import Dict, Any, Optional, Tuple
from fastapi import APIRouter
from typing import Dict, Any, Optional
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
from backend.api.dependencies import optional_auth
from backend.services.auth_service import UserType
from backend.version import VERSION
from backend.services.flacfetch_client import get_flacfetch_client
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.services.encoding_service import get_encoding_service
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
from backend.services.job_health_service import check_job_consistency_detailed
from backend.services.job_manager import JobManager
from backend.models.job import JobStatus
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -235,7 +227,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
"active_jobs": health.get("active_jobs", 0),
"queue_length": health.get("queue_length", 0),
"ffmpeg_version": health.get("ffmpeg_version"),
"wheel_version": health.get("wheel_version"),
}
except Exception as e:
return {
@@ -246,239 +237,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
}
@router.get("/health/encoding-worker")
async def encoding_worker_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check encoding worker status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_encoding_worker_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("wheel_version"),
"active_jobs": status.get("active_jobs", 0),
"queue_length": status.get("queue_length", 0),
}
@router.get("/health/flacfetch")
async def flacfetch_health() -> Dict[str, Any]:
"""
Lightweight endpoint to check flacfetch service status.
Returns minimal info for frontend footer display.
No authentication required.
"""
status = await check_flacfetch_service_status()
# Return simplified response for frontend
if not status.get("configured"):
return {
"available": False,
"status": "not_configured",
}
if not status.get("available"):
return {
"available": False,
"status": "offline",
"error": status.get("error"),
}
return {
"available": True,
"status": "ok",
"version": status.get("version"),
}
def check_audio_separator_status() -> Dict[str, Any]:
"""Check audio separator availability without calling the remote GPU service."""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
@router.get("/health/audio-separator")
async def audio_separator_health() -> Dict[str, Any]:
"""
Return the installed audio-separator package version.
Does NOT call the remote GPU service — that would cold-start an expensive
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
installed package version, which matches what the audio worker uses.
No authentication required.
"""
from backend.config import get_settings
settings = get_settings()
if not settings.audio_separator_api_url:
return {
"available": False,
"status": "not_configured",
}
try:
from importlib.metadata import version
pkg_version = version("audio-separator")
except Exception:
pkg_version = None
return {
"available": True,
"status": "ok",
"version": pkg_version,
}
def _get_encoding_worker_manager():
"""Get the EncodingWorkerManager instance. Separated for testability."""
try:
encoding_service = get_encoding_service()
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
return encoding_service._worker_manager
except Exception:
pass
return None
@router.get("/health/system-status")
async def system_status(
auth: Optional[Tuple] = Depends(optional_auth),
) -> Dict[str, Any]:
"""
Aggregated system status for all services.
Returns basic info for everyone. Admin users get additional
blue-green deployment details for the encoder.
"""
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
# Fetch all health statuses concurrently
import asyncio
encoding_task = check_encoding_worker_status()
flacfetch_task = check_flacfetch_service_status()
encoding_status, flacfetch_status = await asyncio.gather(
encoding_task, flacfetch_task
)
separator_status = check_audio_separator_status()
# Build frontend service info (from env vars baked at build time — returned as-is)
frontend_svc = {
"status": "ok",
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
}
# Build backend service info
backend_svc = {
"status": "ok",
"version": VERSION,
"deployed_at": STARTUP_TIME,
}
if COMMIT_SHA:
backend_svc["commit_sha"] = COMMIT_SHA
if PR_NUMBER:
backend_svc["pr_number"] = PR_NUMBER
if PR_TITLE:
backend_svc["pr_title"] = PR_TITLE
# Build encoder service info
encoder_svc = {
"status": "ok" if encoding_status.get("available") else "offline",
"version": encoding_status.get("wheel_version"),
"active_jobs": encoding_status.get("active_jobs", 0),
}
# Build flacfetch service info
flacfetch_svc = {
"status": "ok" if flacfetch_status.get("available") else "offline",
"version": flacfetch_status.get("version"),
}
# Build separator service info
separator_svc = {
"status": "ok" if separator_status.get("available") else "offline",
"version": separator_status.get("version"),
}
# Admin-only: blue-green encoder details from Firestore
if is_admin:
manager = _get_encoding_worker_manager()
if manager:
try:
config = manager.get_config()
encoder_svc["admin_details"] = {
"primary_vm": config.primary_vm,
"primary_ip": config.primary_ip,
"primary_version": config.primary_version,
"primary_deployed_at": config.primary_deployed_at,
"secondary_vm": config.secondary_vm,
"secondary_ip": config.secondary_ip,
"secondary_version": config.secondary_version,
"secondary_deployed_at": config.secondary_deployed_at,
"last_swap_at": config.last_swap_at,
"deploy_in_progress": config.deploy_in_progress,
"active_jobs": encoding_status.get("active_jobs", 0),
"queue_length": encoding_status.get("queue_length", 0),
}
except Exception as e:
logger.warning(f"Failed to get blue-green config: {e}")
# Add error details for offline services
if not flacfetch_status.get("available"):
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
if not separator_status.get("available"):
separator_svc["admin_details"] = {"error": separator_status.get("error")}
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
return {
"services": {
"frontend": frontend_svc,
"backend": backend_svc,
"encoder": encoder_svc,
"flacfetch": flacfetch_svc,
"separator": separator_svc,
}
}
@router.get("/health/detailed")
async def detailed_health_check() -> Dict[str, Any]:
"""
@@ -542,131 +300,6 @@ async def detailed_health_check() -> Dict[str, Any]:
}
@router.get("/health/preload-status")
async def preload_status() -> Dict[str, Any]:
"""
Check status of preloaded resources for performance optimization.
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
were successfully preloaded at container startup. If any show as
not preloaded, check Cloud Run startup logs for errors.
Expected state after successful deployment:
- spacy.preloaded: true
- nltk.preloaded: true
- langfuse.preloaded: true (if configured) or configured: false
"""
# SpaCy status
spacy_model = get_preloaded_model("en_core_web_sm")
spacy_status = {
"preloaded": is_model_preloaded("en_core_web_sm"),
"model": "en_core_web_sm",
}
if spacy_model:
spacy_status["vocab_size"] = len(spacy_model.vocab)
# NLTK status
cmudict = get_preloaded_cmudict()
nltk_status = {
"preloaded": is_cmudict_preloaded(),
"resource": "cmudict",
}
if cmudict:
nltk_status["entries"] = len(cmudict)
# Langfuse status
langfuse_handler = get_preloaded_langfuse_handler()
langfuse_status = {
"configured": is_langfuse_configured(),
"preloaded": is_langfuse_preloaded(),
}
if langfuse_handler:
langfuse_status["handler_type"] = type(langfuse_handler).__name__
# Overall status
all_preloaded = (
spacy_status["preloaded"]
and nltk_status["preloaded"]
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
)
return {
"status": "ok" if all_preloaded else "degraded",
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
"spacy": spacy_status,
"nltk": nltk_status,
"langfuse": langfuse_status,
"performance_impact": {
"spacy_preload": "Saves ~60s on first lyrics correction",
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
}
}
@router.get("/health/job-consistency")
async def job_consistency_check(
status: Optional[str] = None,
limit: int = 50
) -> Dict[str, Any]:
"""
Check job consistency across the system.
This endpoint identifies jobs with state inconsistencies that may indicate
bugs in the state machine or trigger logic. Use this to find stuck jobs
or jobs in invalid states.
Args:
status: Optional filter by job status (e.g., "pending", "downloading")
limit: Maximum number of jobs to check (default 50)
Returns:
Dictionary with:
- total_checked: Number of jobs checked
- inconsistent_count: Number of jobs with issues
- inconsistent_jobs: List of job details with their issues
- summary: Summary of issue types found
"""
job_manager = JobManager()
# Query jobs to check
query_params = {}
if status:
try:
job_status = JobStatus(status)
query_params['status'] = job_status
except ValueError:
return {
"error": f"Invalid status: {status}",
"valid_statuses": [s.value for s in JobStatus]
}
# Get jobs from Firestore
jobs = job_manager.list_jobs(limit=limit, **query_params)
# Check each job for consistency
inconsistent_jobs = []
issue_summary = {}
for job in jobs:
result = check_job_consistency_detailed(job)
if not result['is_healthy']:
inconsistent_jobs.append(result)
# Count issue types
for issue in result['issues']:
issue_type = issue.split(':')[0] if ':' in issue else issue
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
return {
"total_checked": len(jobs),
"inconsistent_count": len(inconsistent_jobs),
"inconsistent_jobs": inconsistent_jobs,
"summary": issue_summary,
"status_filter": status,
"limit": limit,
}
@router.get("/readiness")
async def readiness_check() -> Dict[str, str]:
"""Readiness check endpoint for Cloud Run."""
+3 -515
View File
@@ -25,7 +25,7 @@ from backend.workers.screens_worker import generate_screens
from backend.workers.video_worker import generate_video
from backend.workers.render_video_worker import process_render_video
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult, UserType
from backend.services.auth_service import UserType
from backend.services.job_manager import JobManager
from backend.services.tracing import (
extract_trace_context,
@@ -76,18 +76,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
message=f"Job {job_id} not found"
)
# Reject workers for jobs in terminal states (failed, cancelled, complete)
# This prevents Cloud Tasks retries from triggering workers on dead jobs
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
if job_status in terminal_statuses:
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
return WorkerResponse(
status="skipped",
job_id=job_id,
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
)
# Check worker-specific progress in state_data
progress_key = f"{worker_name}_progress"
worker_progress = job.state_data.get(progress_key, {})
@@ -114,31 +102,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
return None
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
async def sync_disposable_domains_endpoint(
request: Request,
auth_result: AuthResult = Depends(require_admin),
):
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
from backend.services.email_validation_service import EmailValidationService
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
EmailValidationService._blocklist_cache = None
return WorkerResponse(
status="completed",
job_id="sync-disposable-domains",
message=f"Synced {result['external_count']} domains"
)
@router.post("/workers/audio", response_model=WorkerResponse)
async def trigger_audio_worker(
request: WorkerRequest,
@@ -325,7 +288,7 @@ async def trigger_render_video_worker(
Idempotency: If worker is already running or complete, returns early.
Output: with_vocals.mkv in GCS
Next state: INSTRUMENTAL_SELECTED (instrumental is now selected during review)
Next state: AWAITING_INSTRUMENTAL_SELECTION
"""
job_id = request.job_id
@@ -353,490 +316,15 @@ async def trigger_render_video_worker(
)
@router.post("/jobs/{job_id}/check-idle-reminder")
async def check_idle_reminder(
job_id: str,
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Check if a job needs an idle reminder email.
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
a job enters AWAITING_REVIEW state.
If the job is still in the blocking state and no reminder has been sent yet,
sends a reminder email to the user.
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
"""
from backend.models.job import JobStatus
from backend.services.job_notification_service import get_job_notification_service
# Extract trace context from incoming request
trace_context = extract_trace_context(dict(http_request.headers))
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
add_span_attribute("job_id", job_id)
add_span_attribute("operation", "idle_reminder_check")
job_manager = JobManager()
job = job_manager.get_job(job_id)
if not job:
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
add_span_event("job_not_found")
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
# Check if job is still in a blocking state
# Note: AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs only
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
if job.status not in [s.value for s in blocking_states]:
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
add_span_event("not_blocking", {"current_status": job.status})
return {
"status": "skipped",
"job_id": job_id,
"message": f"Job not in blocking state (current: {job.status})"
}
# Normalize state_data to prevent None errors
state_data = job.state_data or {}
# Check if reminder was already sent (idempotency)
if state_data.get('reminder_sent'):
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
add_span_event("already_sent")
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
if getattr(job, 'made_for_you', False):
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
add_span_event("made_for_you_skip")
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
# Check if user has an email
if not job.user_email:
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
add_span_event("no_email")
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
# Determine action type
action_type = state_data.get('blocking_action_type')
if not action_type:
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
# Send the reminder email
try:
notification_service = get_job_notification_service()
success = await notification_service.send_action_reminder_email(
job_id=job.job_id,
user_email=job.user_email,
action_type=action_type,
user_name=None, # Could fetch from user service if needed
artist=job.artist,
title=job.title,
audio_hash=job.audio_hash,
review_token=job.review_token,
instrumental_token=job.instrumental_token,
)
if success:
# Mark reminder as sent (prevents duplicate sends)
job_manager.firestore.update_job(job_id, {
'state_data': {**state_data, 'reminder_sent': True}
})
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
add_span_event("reminder_sent", {"action_type": action_type})
return {
"status": "sent",
"job_id": job_id,
"message": f"Sent {action_type} reminder to {job.user_email}"
}
else:
logger.error(f"[job:{job_id}] Failed to send reminder email")
add_span_event("send_failed")
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
except Exception as e:
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "job_id": job_id, "message": str(e)}
@router.post("/youtube-queue/process")
async def process_youtube_upload_queue(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process queued YouTube uploads.
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
due to YouTube API quota exhaustion. Can also be triggered manually
from the admin dashboard.
Processes queued uploads one at a time, stopping if quota is exhausted.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("YOUTUBE_QUEUE_PROCESS starting")
add_span_attribute("operation", "youtube_queue_process")
# Run in background so the HTTP response returns quickly
# (Cloud Scheduler has a 30min timeout but we don't want to block)
async def _process():
try:
result = await process_queue()
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("queue_processing_started")
return {
"status": "started",
"message": "YouTube upload queue processing started in background"
}
@router.post("/process-stale-reviews")
async def process_stale_reviews_endpoint(
http_request: Request,
background_tasks: BackgroundTasks,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Process stale review jobs — send reminders and expire old ones.
Called by Cloud Scheduler (hourly). Can also be triggered manually
from the admin dashboard.
- Jobs in review for >= 24h get a reminder email
- Jobs in review for >= 48h are auto-cancelled with credit refund
"""
from backend.workers.stale_review_processor import process_stale_reviews
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("STALE_REVIEW_PROCESS starting")
add_span_attribute("operation", "stale_review_process")
async def _process():
try:
result = await process_stale_reviews()
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
except Exception as e:
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
background_tasks.add_task(_process)
add_span_event("stale_review_processing_started")
return {
"status": "started",
"message": "Stale review processing started in background"
}
@router.post("/trigger-gdrive-validation")
async def trigger_gdrive_validation_endpoint(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Trigger GDrive validation via the Cloud Function.
Called by a Cloud Tasks delayed task after a job completes and uploads
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
finishes before the validator checks for sequence gaps.
"""
from backend.services.gdrive_validator_client import trigger_gdrive_validation
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
add_span_attribute("operation", "gdrive_validation_trigger")
try:
result = trigger_gdrive_validation()
if result is None:
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
status = result.get("status", "unknown")
add_span_event("validation_complete", {"result_status": status})
return {"status": status, "message": "GDrive validation completed", "result": result}
except Exception as e:
logger.exception(f"GDrive validation trigger failed: {e}")
add_span_event("error", {"error": str(e)})
return {"status": "error", "message": str(e)}
@router.post("/recover-stuck-jobs")
async def recover_stuck_jobs(
http_request: Request,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Detect and recover jobs stuck in downloading_audio status.
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
Finds jobs stuck in DOWNLOADING_AUDIO for >10 minutes and fails them
so they can be retried.
"""
from backend.services.job_health_service import check_job_consistency
trace_context = extract_trace_context(dict(http_request.headers))
logger.info("RECOVER_STUCK_JOBS starting")
add_span_attribute("operation", "recover_stuck_jobs")
job_manager = JobManager()
# Query jobs in downloading_audio status
from google.cloud.firestore_v1 import FieldFilter
jobs_ref = job_manager.firestore.db.collection("jobs")
stuck_query = jobs_ref.where(
filter=FieldFilter("status", "==", "downloading_audio")
).stream()
recovered = []
for doc in stuck_query:
job_data = doc.to_dict()
job_id = job_data.get("job_id", doc.id)
# Use the health service to check if actually stuck
job = job_manager.get_job(job_id)
if not job:
continue
issues = check_job_consistency(job)
stuck_issues = [i for i in issues if "downloading_audio_stuck" in i]
if stuck_issues:
logger.warning(f"[job:{job_id}] Recovering stuck download: {stuck_issues[0]}")
job_manager.fail_job(
job_id,
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
)
recovered.append(job_id)
logger.info(f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} jobs")
add_span_event("recovery_complete", {"recovered_count": len(recovered)})
return {
"status": "success",
"recovered_jobs": recovered,
"recovered_count": len(recovered),
}
@router.get("/health")
async def internal_health(
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Internal health check endpoint.
Used to verify the internal API is responsive.
Requires admin authentication.
"""
return {"status": "healthy", "service": "karaoke-backend-internal"}
# =============================================================================
# Test Webhook Endpoint (for E2E testing)
# =============================================================================
class TestWebhookRequest(BaseModel):
"""
Request to simulate a Stripe webhook event for E2E testing.
This allows E2E tests to trigger payment flow logic without requiring
actual Stripe checkout sessions or valid webhook signatures.
"""
event_type: str # e.g., "checkout.session.completed"
session_id: str # Must start with "e2e-test-" prefix
customer_email: str
metadata: dict # order_type, package_id, credits, artist, title, etc.
class TestWebhookResponse(BaseModel):
"""Response from test webhook processing."""
status: str # "processed", "already_processed", "error"
job_id: Optional[str] = None # For made-for-you orders
credits_added: Optional[int] = None # For credit purchases
new_balance: Optional[int] = None # For credit purchases
message: str
@router.post("/test-webhook", response_model=TestWebhookResponse)
async def test_webhook(
request: TestWebhookRequest,
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
):
"""
Test endpoint that simulates Stripe webhook events for E2E testing.
SECURITY:
- Protected by admin authentication (X-Admin-Token header)
- Session IDs must start with "e2e-test-" prefix to prevent collision
with real Stripe sessions
- Only for E2E testing - bypasses Stripe signature verification
This endpoint reuses the same handler logic as the real webhook endpoint,
ensuring E2E tests validate actual business logic.
Supported event types:
- checkout.session.completed: Handles credit purchases and made-for-you orders
For credit purchases, metadata must include:
- package_id: e.g., "1_credit"
- credits: e.g., "1"
- user_email: Email of user to credit
For made-for-you orders, metadata must include:
- order_type: "made_for_you"
- customer_email: Customer email for delivery
- artist: Song artist
- title: Song title
- source_type: "search" or "youtube"
- youtube_url: (optional) If source_type is "youtube"
- notes: (optional) Customer notes
"""
from backend.services.user_service import get_user_service
from backend.services.email_service import get_email_service
from backend.services.stripe_service import get_stripe_service
from backend.api.routes.users import _handle_made_for_you_order
# Validate session_id prefix for safety
if not request.session_id.startswith("e2e-test-"):
logger.warning(f"Test webhook rejected: session_id '{request.session_id}' missing required prefix")
raise HTTPException(
status_code=400,
detail="Session ID must start with 'e2e-test-' prefix for test webhooks"
)
logger.info(f"TEST_WEBHOOK event_type={request.event_type} session_id={request.session_id}")
add_span_attribute("event_type", request.event_type)
add_span_attribute("session_id", request.session_id)
add_span_attribute("is_test_webhook", True)
user_service = get_user_service()
email_service = get_email_service()
stripe_service = get_stripe_service()
if request.event_type == "checkout.session.completed":
session_id = request.session_id
metadata = request.metadata
# Idempotency check: Skip if this session was already processed
if user_service.is_stripe_session_processed(session_id):
logger.info(f"Test webhook: session {session_id} already processed")
return TestWebhookResponse(
status="already_processed",
message=f"Session {session_id} was already processed"
)
# Check if this is a made-for-you order
if metadata.get("order_type") == "made_for_you":
try:
# Call the same handler used by the real webhook
await _handle_made_for_you_order(
session_id=session_id,
metadata=metadata,
user_service=user_service,
email_service=email_service,
)
# Get the job ID from the most recent job for this customer
# The handler creates a job, so we need to find it
from google.cloud import firestore
from google.cloud.firestore_v1 import FieldFilter
db = user_service.db
# Look for the job by session_id pattern in state_data or by customer_email
# Since the job was just created, query by customer_email and made_for_you flag
customer_email = metadata.get("customer_email", "")
jobs_query = db.collection("jobs").where(
filter=FieldFilter("customer_email", "==", customer_email)
).where(
filter=FieldFilter("made_for_you", "==", True)
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(1)
jobs = list(jobs_query.stream())
job_id = jobs[0].to_dict().get("job_id") if jobs else None
logger.info(f"Test webhook: made-for-you order processed, job_id={job_id}")
return TestWebhookResponse(
status="processed",
job_id=job_id,
message=f"Made-for-you order created successfully"
)
except Exception as e:
logger.exception(f"Test webhook: error processing made-for-you order: {e}")
return TestWebhookResponse(
status="error",
message=f"Error processing made-for-you order: {str(e)}"
)
else:
# Handle regular credit purchase
# Build a synthetic session object that matches Stripe's format
synthetic_session = {
"id": session_id,
"customer_email": request.customer_email,
"metadata": metadata,
}
success, user_email, credits, msg = stripe_service.handle_checkout_completed(
synthetic_session
)
if not success:
logger.warning(f"Test webhook: credit purchase validation failed: {msg}")
return TestWebhookResponse(
status="error",
message=msg
)
if user_email and credits > 0:
# Add credits to user account
ok, new_balance, credit_msg = user_service.add_credits(
email=user_email,
amount=credits,
reason="stripe_purchase",
stripe_session_id=session_id,
)
if ok:
# Send confirmation email (same as real webhook)
email_service.send_credits_added(user_email, credits, new_balance)
logger.info(f"Test webhook: added {credits} credits to {user_email}, new balance: {new_balance}")
return TestWebhookResponse(
status="processed",
credits_added=credits,
new_balance=new_balance,
message=f"Added {credits} credits to {user_email}"
)
else:
logger.error(f"Test webhook: failed to add credits: {credit_msg}")
return TestWebhookResponse(
status="error",
message=f"Failed to add credits: {credit_msg}"
)
return TestWebhookResponse(
status="error",
message="Invalid credit purchase data"
)
else:
# Unsupported event type
logger.warning(f"Test webhook: unsupported event type '{request.event_type}'")
return TestWebhookResponse(
status="error",
message=f"Unsupported event type: {request.event_type}"
)
+528 -1412
View File
File diff suppressed because it is too large. Load diff
-263
View File
@@ -1,263 +0,0 @@
"""
Push Notification API routes.
Provides endpoints for managing Web Push notification subscriptions:
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
- POST /api/push/subscribe: Register a push subscription
- POST /api/push/unsubscribe: Remove a push subscription
- GET /api/push/subscriptions: List user's subscriptions
- POST /api/push/test: Send a test notification (admin only)
"""
import logging
from typing import Optional, Dict, List
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.config import get_settings
from backend.api.dependencies import require_auth, require_admin
from backend.services.auth_service import AuthResult
from backend.services.push_notification_service import get_push_notification_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/push", tags=["push"])
# Request/Response Models
class VapidPublicKeyResponse(BaseModel):
"""Response containing VAPID public key."""
enabled: bool
vapid_public_key: Optional[str] = None
class SubscribeRequest(BaseModel):
"""Request to subscribe to push notifications."""
endpoint: str
keys: Dict[str, str] # p256dh and auth
device_name: Optional[str] = None
tenant_id: Optional[str] = None # Tenant scope for notification isolation
class SubscribeResponse(BaseModel):
"""Response after subscribing."""
status: str
message: str
class UnsubscribeRequest(BaseModel):
"""Request to unsubscribe from push notifications."""
endpoint: str
class UnsubscribeResponse(BaseModel):
"""Response after unsubscribing."""
status: str
message: str
class SubscriptionInfo(BaseModel):
"""Information about a push subscription."""
endpoint: str
device_name: Optional[str] = None
created_at: Optional[str] = None
last_used_at: Optional[str] = None
class SubscriptionsListResponse(BaseModel):
"""Response containing user's subscriptions."""
subscriptions: List[SubscriptionInfo]
count: int
class TestNotificationRequest(BaseModel):
"""Request to send a test notification."""
title: Optional[str] = "Test Notification"
body: Optional[str] = "This is a test push notification from Karaoke Generator"
class TestNotificationResponse(BaseModel):
"""Response after sending test notification."""
status: str
sent_count: int
message: str
# Routes
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
async def get_vapid_public_key():
"""
Get the VAPID public key for push subscription.
This endpoint is public - no authentication required.
Returns the public key needed for client-side PushManager.subscribe().
"""
settings = get_settings()
push_service = get_push_notification_service()
if not settings.enable_push_notifications:
return VapidPublicKeyResponse(enabled=False)
public_key = push_service.get_public_key()
if not public_key:
return VapidPublicKeyResponse(enabled=False)
return VapidPublicKeyResponse(
enabled=True,
vapid_public_key=public_key
)
@router.post("/subscribe", response_model=SubscribeResponse)
async def subscribe_push(
subscribe_request: SubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
Register a push notification subscription for the current user.
Requires authentication. Users can have up to 5 subscriptions
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage. The tenant is resolved from:
1. Request body tenant_id field (explicit from frontend)
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
# Validate keys
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
# Resolve tenant_id: prefer explicit from body, fall back to middleware
tenant_id = subscribe_request.tenant_id
if tenant_id is None:
tenant_id = getattr(http_request.state, "tenant_id", None)
success = await push_service.add_subscription(
user_email=auth_result.user_email,
endpoint=subscribe_request.endpoint,
keys=subscribe_request.keys,
device_name=subscribe_request.device_name,
tenant_id=tenant_id or None,
)
if not success:
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
return SubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRegistered")
)
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
async def unsubscribe_push(
request: UnsubscribeRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
Remove a push notification subscription.
Requires authentication. Users can only remove their own subscriptions.
"""
locale = get_locale_from_request(http_request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
success = await push_service.remove_subscription(
user_email=auth_result.user_email,
endpoint=request.endpoint
)
if not success:
# Don't error if subscription wasn't found - might already be removed
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionNotFound")
)
return UnsubscribeResponse(
status="success",
message=t(locale, "push.subscriptionRemoved")
)
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
async def list_subscriptions(
request: Request,
auth_result: AuthResult = Depends(require_auth)
):
"""
List all push notification subscriptions for the current user.
Requires authentication.
"""
locale = get_locale_from_request(request)
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
return SubscriptionsListResponse(
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
count=len(subscriptions)
)
@router.post("/test", response_model=TestNotificationResponse)
async def send_test_notification(
request: TestNotificationRequest,
http_request: Request,
auth_result: AuthResult = Depends(require_admin)
):
"""
Send a test push notification to the current user's devices.
Admin only. Useful for testing push notification setup.
"""
locale = get_locale_from_request(http_request)
settings = get_settings()
if not settings.enable_push_notifications:
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
if not auth_result.user_email:
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
push_service = get_push_notification_service()
sent_count = await push_service.send_push(
user_email=auth_result.user_email,
title=request.title or "Test Notification",
body=request.body or "This is a test push notification",
url="/app/",
tag="test"
)
return TestNotificationResponse(
status="success",
sent_count=sent_count,
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
)
-346
View File
@@ -1,346 +0,0 @@
"""
Admin API routes for blocklist and YouTube queue management.
Handles:
- Blocklist management (disposable domains, blocked emails, blocked IPs)
- YouTube upload queue management
"""
import logging
from datetime import datetime
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from backend.api.dependencies import require_admin
from backend.services.auth_service import AuthResult
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
# =============================================================================
# Response Models
# =============================================================================
class YouTubeQueueEntry(BaseModel):
"""A queued YouTube upload entry."""
job_id: str
status: str
reason: Optional[str] = None
user_email: Optional[str] = None
artist: Optional[str] = None
title: Optional[str] = None
brand_code: Optional[str] = None
queued_at: Optional[datetime] = None
attempts: int = 0
max_attempts: int = 5
last_error: Optional[str] = None
youtube_url: Optional[str] = None
notification_sent: bool = False
class YouTubeQueueListResponse(BaseModel):
"""List of YouTube upload queue entries."""
entries: List[YouTubeQueueEntry]
stats: dict
class BlocklistsResponse(BaseModel):
"""All blocklist data with domain source separation."""
external_domains: List[str]
manual_domains: List[str]
allowlisted_domains: List[str]
blocked_emails: List[str]
blocked_ips: List[str]
last_sync_at: Optional[datetime] = None
last_sync_count: Optional[int] = None
updated_at: Optional[datetime] = None
updated_by: Optional[str] = None
class DomainRequest(BaseModel):
"""Request to add/remove a domain."""
domain: str
class EmailRequest(BaseModel):
"""Request to add/remove an email."""
email: str
class IPRequest(BaseModel):
"""Request to add/remove an IP."""
ip_address: str
class SuccessResponse(BaseModel):
"""Generic success response."""
success: bool
message: str
# =============================================================================
# Blocklist Management Endpoints
# =============================================================================
@router.get("/blocklists", response_model=BlocklistsResponse)
async def get_blocklists(
auth_result: AuthResult = Depends(require_admin),
):
"""Get all blocklist data with domain source separation."""
email_validation = get_email_validation_service()
raw_data = email_validation.get_blocklist_raw_data()
return BlocklistsResponse(**raw_data)
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
async def add_disposable_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_disposable_domain(domain, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.domainAddedToBl", domain=domain)
)
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
async def remove_disposable_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the disposable domains blocklist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.domainRemovedFromBl", domain=domain)
)
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
async def add_allowlisted_domain(
request: DomainRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add a domain to the allowlist (overrides external blocklist)."""
email_validation = get_email_validation_service()
domain = request.domain.lower().strip()
if not domain or "." not in domain:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
async def remove_allowlisted_domain(
domain: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove a domain from the allowlist."""
email_validation = get_email_validation_service()
domain = domain.lower().strip()
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
@router.post("/blocklists/sync", response_model=SuccessResponse)
async def trigger_sync(
auth_result: AuthResult = Depends(require_admin),
):
"""Manually trigger a sync of the external disposable domain blocklist."""
import asyncio
from backend.services.disposable_domain_sync_service import (
fetch_external_blocklist, sync_disposable_domains
)
from backend.services.firestore_service import get_firestore_client
domains = await fetch_external_blocklist()
db = get_firestore_client()
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
# Invalidate cache
EmailValidationService._blocklist_cache = None
return SuccessResponse(
success=True,
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
)
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
async def add_blocked_email(
request: EmailRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an email to the blocked emails list."""
email_validation = get_email_validation_service()
email = request.email.lower().strip()
if not email or "@" not in email:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
email_validation.add_blocked_email(email, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.emailAddedToBl", email=email)
)
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
async def remove_blocked_email(
email: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an email from the blocked emails list."""
email_validation = get_email_validation_service()
email = email.lower().strip()
if not email_validation.remove_blocked_email(email, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.emailRemovedFromBl", email=email)
)
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
async def add_blocked_ip(
request: IPRequest,
auth_result: AuthResult = Depends(require_admin),
):
"""Add an IP address to the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = request.ip_address.strip()
if not ip_address:
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
return SuccessResponse(
success=True,
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
)
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
async def remove_blocked_ip(
ip_address: str,
auth_result: AuthResult = Depends(require_admin),
):
"""Remove an IP address from the blocked IPs list."""
email_validation = get_email_validation_service()
ip_address = ip_address.strip()
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
return SuccessResponse(
success=True,
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
)
# =============================================================================
# YouTube Upload Queue Management Endpoints
# =============================================================================
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
async def get_youtube_queue(
auth_result: AuthResult = Depends(require_admin),
):
"""
Get YouTube upload queue entries and stats.
Returns all queue entries (any status) and aggregate statistics.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
entries = queue_service.get_all_queue_entries(limit=50)
stats = queue_service.get_queue_stats()
return YouTubeQueueListResponse(
entries=[YouTubeQueueEntry(**entry) for entry in entries],
stats=stats,
)
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
async def retry_youtube_upload(
job_id: str,
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually retry a failed YouTube upload.
Resets the entry back to 'queued' status with 0 attempts.
"""
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
queue_service = get_youtube_upload_queue_service()
if not queue_service.retry_upload(job_id):
raise HTTPException(
status_code=404,
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
)
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
)
@router.post("/youtube-queue/process", response_model=SuccessResponse)
async def trigger_youtube_queue_processing(
auth_result: AuthResult = Depends(require_admin),
):
"""
Manually trigger YouTube upload queue processing.
Same as the scheduled processor but triggered from the admin dashboard.
Runs in the background and returns immediately.
"""
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
import asyncio
async def _process():
try:
result = await process_youtube_upload_queue()
logger.info(f"Manual YouTube queue processing complete: {result}")
except Exception as e:
logger.exception(f"Manual YouTube queue processing failed: {e}")
# Fire and forget
try:
loop = asyncio.get_running_loop()
loop.create_task(_process())
except RuntimeError:
pass
return SuccessResponse(
success=True,
message=t("en", "admin.youtubeQueueProcessingStarted")
)
-374
View File
@@ -1,374 +0,0 @@
"""
Referral API routes.
Handles:
- Public referral interstitial (click tracking)
- Authenticated referral dashboard
- Stripe Connect onboarding
- Admin vanity link creation and link management
"""
import logging
from typing import Optional
from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import Response
from pydantic import BaseModel, Field
from backend.api.dependencies import require_auth, require_admin
from backend.models.referral import (
CreateReferralLinkRequest,
UpdateReferralLinkRequest,
ReferralInterstitialResponse,
)
from backend.services.flyer_service import get_flyer_service, FlyerError
from backend.services.referral_service import get_referral_service
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/referrals", tags=["referrals"])
class GenerateFlyerRequest(BaseModel):
theme: str = Field(..., pattern="^(light|dark)$")
qr_data_url: str = Field(..., max_length=500_000)
# ============================================================================
# Public endpoints
# ============================================================================
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
async def get_referral_interstitial(code: str, no_track: bool = False):
"""
Public endpoint: get interstitial data for a referral link.
Increments click count unless no_track=true. Returns valid=False if code not found.
"""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
return ReferralInterstitialResponse(valid=False)
# Track the click (skip when fetching for display purposes)
if not no_track:
try:
service.increment_clicks(code)
except Exception:
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
return ReferralInterstitialResponse(
valid=True,
referral_code=link.code,
referrer_display_name=link.display_name,
discount_percent=link.discount_percent,
discount_duration_days=link.discount_duration_days,
custom_message=link.custom_message,
)
# ============================================================================
# Authenticated user endpoints
# ============================================================================
@router.get("/me")
async def get_my_dashboard(auth=Depends(require_auth)):
"""Get referral dashboard data for the authenticated user."""
service = get_referral_service()
return service.get_dashboard_data(auth.user_email)
@router.put("/me")
async def update_my_link(
updates: UpdateReferralLinkRequest,
auth=Depends(require_auth),
):
"""Update display_name and custom_message on the user's own referral link."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
update_fields = {}
if updates.display_name is not None:
update_fields["display_name"] = updates.display_name
if updates.custom_message is not None:
update_fields["custom_message"] = updates.custom_message
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(link.code, **update_fields)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "message": message}
@router.post("/me/connect")
async def start_stripe_connect(auth=Depends(require_auth)):
"""Start Stripe Connect onboarding for the authenticated user."""
# Check if user already has a Connect account
from backend.services.user_service import get_user_service
user_service = get_user_service()
user = user_service.get_user(auth.user_email)
if user and user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
service = get_referral_service()
account_id, onboarding_url = service.create_connect_account(auth.user_email)
if not account_id:
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
return {"account_id": account_id, "onboarding_url": onboarding_url}
@router.post("/me/connect/dashboard-link")
async def get_connect_dashboard_link(auth=Depends(require_auth)):
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_login_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
return {"url": url}
@router.post("/me/connect/update-link")
async def get_connect_update_link(auth=Depends(require_auth)):
"""Get a link to update Stripe Connect account info (re-onboarding)."""
from backend.services.user_service import get_user_service
user = get_user_service().get_user(auth.user_email)
if not user or not user.stripe_connect_account_id:
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
service = get_referral_service()
url = service.create_connect_update_link(user.stripe_connect_account_id)
if not url:
raise HTTPException(status_code=500, detail="Failed to create update link")
return {"url": url}
class VanityRequest(BaseModel):
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
@router.post("/me/vanity-request")
async def request_vanity_url(
request: VanityRequest,
auth=Depends(require_auth),
):
"""Request a custom vanity referral URL. Sends email to admin for review."""
from backend.services.email_service import get_email_service
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
# Check if code is already taken
existing = service.get_link_by_code(request.desired_code)
if existing:
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
email_service = get_email_service()
email_service.provider.send_email(
to_email="andrew@nomadkaraoke.com",
subject=f"Vanity referral URL request: {request.desired_code}",
html_content=f"""
<h2>Vanity Referral URL Request</h2>
<p><strong>User:</strong> {auth.user_email}</p>
<p><strong>Current code:</strong> {link.code}</p>
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
""",
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
)
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
return {"ok": True, "message": "Request sent"}
@router.post("/me/flyer")
async def generate_flyer(
request: GenerateFlyerRequest,
auth=Depends(require_auth),
):
"""Generate a personalized referral flyer PDF."""
service = get_referral_service()
link = service.get_or_create_link(auth.user_email)
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
},
)
# ============================================================================
# Admin endpoints
# ============================================================================
@router.post("/admin/vanity")
async def create_vanity_link(
request: CreateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: create a vanity referral link."""
if not request.vanity_code:
raise HTTPException(status_code=400, detail="vanity_code is required")
service = get_referral_service()
success, link, message = service.create_vanity_link(
code=request.vanity_code,
owner_email=request.owner_email,
display_name=request.display_name,
custom_message=request.custom_message,
discount_percent=request.discount_percent,
kickback_percent=request.kickback_percent,
discount_duration_days=request.discount_duration_days,
earning_duration_days=request.earning_duration_days,
)
if not success:
raise HTTPException(status_code=400, detail=message)
return {"ok": True, "code": link.code, "message": message}
@router.get("/admin/links")
async def list_links(
limit: int = Query(50, ge=1, le=200),
offset: int = Query(0, ge=0),
auth=Depends(require_admin),
):
"""Admin: list all referral links."""
service = get_referral_service()
links = service.list_links(limit=limit, offset=offset)
return {
"links": [link.model_dump(mode="json") for link in links],
"count": len(links),
}
@router.post("/admin/apply-discount")
async def admin_apply_discount(
request: Request,
auth=Depends(require_admin),
):
"""Admin: apply a referral discount to any user, even without a referral link."""
from backend.services.user_service import get_user_service
body = await request.json()
user_email = body.get("email", "").lower()
discount_percent = body.get("discount_percent", 10)
duration_days = body.get("duration_days", 30)
referral_code = body.get("referral_code", "admin-grant")
if not user_email:
raise HTTPException(status_code=400, detail="Email is required")
user_service = get_user_service()
user = user_service.get_user(user_email)
if not user:
raise HTTPException(status_code=404, detail="User not found")
from datetime import datetime, timedelta
now = datetime.utcnow()
user_service.update_user(
user_email,
referred_by_code=referral_code,
referred_at=now,
referral_discount_expires_at=now + timedelta(days=duration_days),
)
# Create/ensure the Stripe coupon exists for this discount percent
service = get_referral_service()
service.get_or_create_stripe_coupon(discount_percent)
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
return {
"ok": True,
"message": f"Applied {discount_percent}% discount for {duration_days} days",
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
}
@router.put("/admin/links/{code}")
async def update_link(
code: str,
updates: UpdateReferralLinkRequest,
auth=Depends(require_admin),
):
"""Admin: update any referral link's settings."""
service = get_referral_service()
update_fields = {
k: v for k, v in updates.model_dump().items() if v is not None
}
if not update_fields:
return {"ok": True, "message": "No changes"}
success, message = service.update_link(code, **update_fields)
if not success:
raise HTTPException(status_code=404, detail=message)
return {"ok": True, "message": message}
@router.post("/admin/links/{code}/flyer")
async def admin_generate_flyer(
code: str,
request: GenerateFlyerRequest,
auth=Depends(require_admin),
):
"""Admin: generate a personalized referral flyer PDF for any code."""
service = get_referral_service()
link = service.get_link_by_code(code)
if not link:
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
try:
flyer_service = get_flyer_service()
pdf_bytes = flyer_service.generate_pdf(
theme=request.theme,
referral_code=link.code,
discount_percent=link.discount_percent,
qr_data_url=request.qr_data_url,
)
except FlyerError as e:
raise HTTPException(status_code=400, detail=str(e))
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
},
)
+161 -1580
View File
File diff suppressed because it is too large. Load diff
-121
View File
@@ -1,121 +0,0 @@
"""
Tenant API routes for white-label portal configuration.
These endpoints are public (no auth required) since the frontend needs
to fetch tenant branding before the user logs in.
"""
import logging
from typing import Optional
from fastapi import APIRouter, Query, Request
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
from backend.services.tenant_service import get_tenant_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
@router.get("/config", response_model=TenantConfigResponse)
async def get_tenant_config(
request: Request,
tenant: Optional[str] = Query(
None,
description="Tenant ID override (for development). If not provided, detected from subdomain.",
),
):
"""
Get tenant configuration for the current portal.
This endpoint detects the tenant from:
1. Query parameter `tenant` (for development/testing)
2. X-Tenant-ID header (set by frontend)
3. Host header subdomain
Returns the public tenant configuration (branding, features, defaults)
or indicates this is the default Nomad Karaoke portal.
"""
tenant_service = get_tenant_service()
# Priority 1: Query parameter (for dev/testing)
tenant_id = tenant
# Priority 2: X-Tenant-ID header
if not tenant_id:
tenant_id = request.headers.get("X-Tenant-ID")
# Priority 3: Detect from Host header
if not tenant_id:
host = request.headers.get("Host", "")
# Check if this is a tenant subdomain
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
if host and "nomadkaraoke.com" in host.lower():
parts = host.lower().split(".")
# Skip known non-tenant subdomains
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
potential_tenant = parts[0]
if tenant_service.tenant_exists(potential_tenant):
tenant_id = potential_tenant
# If no tenant detected, return default config
if not tenant_id:
logger.debug("No tenant detected, returning default config")
return TenantConfigResponse(tenant=None, is_default=True)
# Load tenant config
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
logger.warning(f"Tenant not found: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
logger.info(f"Returning config for tenant: {tenant_id}")
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
async def get_tenant_config_by_id(tenant_id: str):
"""
Get tenant configuration by explicit tenant ID.
This is useful for admin tools or debugging.
"""
tenant_service = get_tenant_service()
public_config = tenant_service.get_public_config(tenant_id)
if not public_config:
return TenantConfigResponse(tenant=None, is_default=True)
# Check if tenant is active
if not public_config.is_active:
logger.warning(f"Tenant is inactive: {tenant_id}")
return TenantConfigResponse(tenant=None, is_default=True)
return TenantConfigResponse(tenant=public_config, is_default=False)
@router.get("/asset/{tenant_id}/{asset_name}")
async def get_tenant_asset(tenant_id: str, asset_name: str):
"""
Get a signed URL for a tenant asset (logo, favicon, etc.).
This redirects to the signed GCS URL for the asset.
"""
from fastapi.responses import RedirectResponse
tenant_service = get_tenant_service()
url = tenant_service.get_asset_url(tenant_id, asset_name)
if not url:
from fastapi import HTTPException
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
return RedirectResponse(url=url)
+13 -18
View File
@@ -9,11 +9,10 @@ before job creation.
import logging
from typing import Optional
from fastapi import APIRouter, HTTPException, Request
from fastapi import APIRouter, HTTPException
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
from backend.services.theme_service import get_theme_service
from backend.i18n import t, get_locale_from_request
logger = logging.getLogger(__name__)
@@ -21,7 +20,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
@router.get("", response_model=ThemesListResponse)
async def list_themes(request: Request) -> ThemesListResponse:
async def list_themes() -> ThemesListResponse:
"""
List all available themes.
@@ -30,7 +29,6 @@ async def list_themes(request: Request) -> ThemesListResponse:
This endpoint is public and does not require authentication.
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
themes = theme_service.list_themes()
@@ -39,12 +37,12 @@ async def list_themes(request: Request) -> ThemesListResponse:
logger.error(f"Error listing themes: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themesLoadError")
detail="Failed to load themes"
)
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
async def get_theme(theme_id: str) -> ThemeDetailResponse:
"""
Get detailed information about a specific theme.
@@ -60,7 +58,6 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
theme = theme_service.get_theme(theme_id)
@@ -68,7 +65,7 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
if theme is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
return ThemeDetailResponse(theme=theme)
@@ -78,12 +75,12 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
logger.error(f"Error getting theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.themeLoadError")
detail="Failed to load theme"
)
@router.get("/{theme_id}/preview")
async def get_theme_preview(theme_id: str, request: Request) -> dict:
async def get_theme_preview(theme_id: str) -> dict:
"""
Get a signed URL for the theme's preview image.
@@ -99,21 +96,20 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
Raises:
404: Theme not found or no preview available
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
theme = theme_service.get_theme(theme_id)
if theme is None or theme.preview_url is None:
raise HTTPException(
status_code=404,
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
detail=f"No preview available for theme: {theme_id}"
)
return {"preview_url": theme.preview_url}
@@ -123,12 +119,12 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
logger.error(f"Error getting preview for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.previewLoadError")
detail="Failed to load theme preview"
)
@router.get("/{theme_id}/youtube-description")
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
async def get_theme_youtube_description(theme_id: str) -> dict:
"""
Get the YouTube description template for a theme.
@@ -145,14 +141,13 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
Raises:
404: Theme not found
"""
locale = get_locale_from_request(request)
try:
theme_service = get_theme_service()
if not theme_service.theme_exists(theme_id):
raise HTTPException(
status_code=404,
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
detail=f"Theme not found: {theme_id}"
)
description = theme_service.get_youtube_description(theme_id)
@@ -163,5 +158,5 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
raise HTTPException(
status_code=500,
detail=t(locale, "themes.youtubeDescriptionLoadError")
detail="Failed to load YouTube description"
)
+368 -1181
View File
File diff suppressed because it is too large. Load diff
-35
View File
@@ -1,35 +0,0 @@
# Cloud Build config for GPU base image
#
# Builds the CUDA + PyTorch + models base image used by the audio worker.
# This image is large (~10GB) so builds take 15-30 minutes.
#
# Trigger manually:
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
#
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
steps:
- name: 'gcr.io/cloud-builders/docker'
args:
- 'build'
- '-f'
- 'backend/Dockerfile.gpu-base'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- '-t'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
- '--build-arg'
- 'BUILD_DATE=$_BUILD_DATE'
- '--build-arg'
- 'BUILD_ID=$BUILD_ID'
- '.'
images:
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
timeout: '3600s'
options:
machineType: 'E2_HIGHCPU_32'
diskSizeGb: 100
+2 -63
View File
@@ -32,10 +32,6 @@ class Settings(BaseSettings):
# Authentication
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
scheduler_service_account: str = os.getenv(
"SCHEDULER_SERVICE_ACCOUNT",
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
)
# Application
environment: str = os.getenv("ENVIRONMENT", "development")
@@ -45,15 +41,7 @@ class Settings(BaseSettings):
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
# Lyrics Correction Settings
# Skip all auto-correction (both agentic AI and heuristic handlers).
# When true, raw transcription goes directly to human review without any automatic fixes.
# This is the default because auto-correction currently creates more work for reviewers.
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
# Agentic AI Correction (for lyrics correction via LLM)
# Only used when skip_correction=false
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
@@ -61,12 +49,6 @@ class Settings(BaseSettings):
# use uncorrected transcription - human review will fix any issues.
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
# AI Credit Evaluation (anti-abuse gating for free credits)
# When enabled, welcome and feedback credit grants are evaluated by Gemini
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
# Cloud Tasks (for scalable worker coordination)
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
# When disabled (default), workers are triggered via direct HTTP (for development)
@@ -86,16 +68,7 @@ class Settings(BaseSettings):
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
# GCE Preview Encoding (for faster preview video generation)
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
# Karaoke-Decide catalog API (for song/artist autocomplete)
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
# Storage paths
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
@@ -110,13 +83,7 @@ class Settings(BaseSettings):
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
# Flacfetch client retry settings
# Retries help handle transient network failures when calling the flacfetch service
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
# Default distribution settings (can be overridden per-request)
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
@@ -128,41 +95,13 @@ class Settings(BaseSettings):
# Default values for web service jobs (YouTube/Dropbox distribution)
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
# Private (non-published) track distribution settings
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
# YouTube Data API v3 Quota Configuration
# Daily quota limit (YouTube grants 10,000 units/day by default)
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
# Safety margin to reserve (avoid hitting exact limit)
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
# E2E test bypass key for rate limiting (set via secret in production)
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
default_youtube_description: str = os.getenv(
"DEFAULT_YOUTUBE_DESCRIPTION",
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
"AI-powered vocal separation and synchronized lyrics.\n\n"
"#karaoke #music #singing #instrumental #lyrics"
)
# Default CDG/TXT generation settings
# When True, CDG and TXT packages are generated by default (when a theme is set)
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
# Push Notifications Configuration
# When enabled, users can subscribe to push notifications for job status updates
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
# Maximum number of push subscriptions per user (oldest removed when exceeded)
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
# VAPID subject (email or URL for push service to contact)
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
# Secret Manager cache
_secret_cache: Dict[str, str] = {}
-94
View File
@@ -1,94 +0,0 @@
"""
Custom exceptions for the karaoke generation backend.
These exceptions are used for structured error handling across the application,
particularly for validation errors and credit checks.
"""
class EmailValidationError(Exception):
"""
Raised when email validation fails.
Attributes:
message: Human-readable error message
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
"""
def __init__(self, message: str, reason: str = "invalid"):
self.message = message
self.reason = reason
super().__init__(message)
class IPBlockedError(Exception):
"""
Raised when a request comes from a blocked IP address.
Attributes:
message: Human-readable error message
ip_address: The blocked IP address (may be partially masked)
"""
def __init__(self, message: str, ip_address: str = ""):
self.message = message
self.ip_address = ip_address
super().__init__(message)
class InsufficientCreditsError(Exception):
"""
Raised when a user attempts to create a job without sufficient credits.
Attributes:
message: Human-readable error message
credits_available: Number of credits the user currently has
credits_required: Number of credits required (always 1 for now)
"""
def __init__(
self,
message: str,
credits_available: int = 0,
credits_required: int = 1
):
self.message = message
self.credits_available = credits_available
self.credits_required = credits_required
super().__init__(message)
class InvalidStateTransitionError(Exception):
"""
Raised when an invalid job state transition is attempted.
This exception is raised by JobManager.transition_to_state() when the
requested transition is not allowed by the state machine defined in
STATE_TRANSITIONS.
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
or CANCELLED.
Attributes:
message: Human-readable error message
job_id: The job ID that failed to transition
from_status: Current status of the job
to_status: Attempted target status
valid_transitions: List of valid transitions from current status
"""
def __init__(
self,
message: str,
job_id: str = "",
from_status: str = "",
to_status: str = "",
valid_transitions: list = None
):
self.message = message
self.job_id = job_id
self.from_status = from_status
self.to_status = to_status
self.valid_transitions = valid_transitions or []
super().__init__(message)
-99
View File
@@ -1,99 +0,0 @@
"""
Lightweight i18n utility for backend translations.
Usage:
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
# In a route handler:
locale = get_locale_from_request(request)
message = t(locale, "auth.userNotFound")
message = t(locale, "jobs.notFound", job_id="abc123")
"""
import json
from functools import lru_cache
from pathlib import Path
from typing import Any
from starlette.requests import Request
SUPPORTED_LOCALES = ("en", "es", "de")
DEFAULT_LOCALE = "en"
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
def _load_translations(locale: str) -> dict[str, Any]:
"""Load and cache translation file for a locale."""
path = TRANSLATIONS_DIR / f"{locale}.json"
if not path.exists():
if locale != DEFAULT_LOCALE:
return _load_translations(DEFAULT_LOCALE)
return {}
with open(path, encoding="utf-8") as f:
result: dict[str, Any] = json.load(f)
return result
def t(locale: str, key: str, **kwargs: Any) -> str:
"""Get a translated string by dotted key path.
Args:
locale: Language code (en, es, de)
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
**kwargs: Interpolation variables (e.g., job_id="abc123")
Returns:
Translated string with variables interpolated, or the key itself if not found.
"""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
translations = _load_translations(locale)
# Walk the dotted path
value: Any = translations
for part in key.split("."):
if isinstance(value, dict) and part in value:
value = value[part]
else:
# Key not found — fall back to English
if locale != DEFAULT_LOCALE:
return t(DEFAULT_LOCALE, key, **kwargs)
return key # Return the key itself as last resort
if not isinstance(value, str):
return key
result: str = value
# Interpolate variables
if kwargs:
try:
result = result.format(**kwargs)
except KeyError:
pass # Missing variable — return template as-is
return result
def get_locale_from_request(request: Request) -> str:
"""Extract locale from Accept-Language header.
Returns the first supported locale found, or DEFAULT_LOCALE.
"""
accept_lang = request.headers.get("accept-language", "")
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
for part in accept_lang.split(","):
lang = part.split(";")[0].strip().split("-")[0].lower()
if lang in SUPPORTED_LOCALES:
return lang
return DEFAULT_LOCALE
def get_locale_prefix(locale: str) -> str:
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
if locale not in SUPPORTED_LOCALES:
locale = DEFAULT_LOCALE
return f"/{locale}"
+6 -80
View File
@@ -7,15 +7,10 @@ from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from backend.config import settings
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker, client_errors
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
from backend.services.structured_logging import setup_structured_logging
from backend.services.spacy_preloader import preload_spacy_model
from backend.services.nltk_preloader import preload_all_nltk_resources
from backend.services.langfuse_preloader import preload_langfuse_handler
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware
from backend.workers.registry import worker_registry
from backend.version import VERSION
@@ -72,28 +67,7 @@ async def lifespan(app: FastAPI):
logger.info(f"Environment: {settings.environment}")
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
logger.info(f"Tracing enabled: {tracing_enabled}")
# Preload NLP models and resources to avoid cold start delays
# See docs/archive/2026-01-08-performance-investigation.md for background
# 1. SpaCy model (60+ second delay without preload)
try:
preload_spacy_model("en_core_web_sm")
except Exception as e:
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
# 2. NLTK cmudict (50-100+ second delay without preload)
try:
preload_all_nltk_resources()
except Exception as e:
logger.warning(f"NLTK preload failed (will load lazily): {e}")
# 3. Langfuse callback handler (200+ second delay without preload)
try:
preload_langfuse_handler()
except Exception as e:
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
# Validate OAuth credentials (non-blocking)
try:
await validate_credentials_on_startup()
@@ -101,23 +75,8 @@ async def lifespan(app: FastAPI):
logger.error(f"Credential validation failed: {e}")
yield
# Shutdown - wait for any active workers to complete before terminating
# This prevents Cloud Run from killing workers mid-processing
logger.info("Shutdown requested, checking for active workers...")
if worker_registry.has_active_workers():
active = worker_registry.get_active_workers()
logger.info(f"Active workers found: {active}")
logger.info("Waiting for workers to complete (timeout: 600s)...")
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
if not completed:
logger.error(
"Shutdown timeout - some workers may not have completed cleanly. "
f"Remaining workers: {worker_registry.get_active_workers()}"
)
else:
logger.info("No active workers, proceeding with shutdown")
# Shutdown
logger.info("Shutting down karaoke generation backend")
@@ -136,8 +95,8 @@ if tracing_enabled:
# Configure CORS
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=False,
allow_origins=["*"], # Configure this properly for production
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@@ -145,50 +104,17 @@ app.add_middleware(
# Add audit logging middleware (captures all requests with request_id for correlation)
app.add_middleware(AuditLoggingMiddleware)
# Add tenant detection middleware (extracts tenant from subdomain/headers)
app.add_middleware(TenantMiddleware)
# Include routers
app.include_router(health.router, prefix="/api")
app.include_router(jobs.router, prefix="/api")
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
app.include_router(auth.router, prefix="/api") # OAuth credential management
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
app.include_router(themes.router, prefix="/api") # Theme selection for styles
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
from backend.api.routes import referrals
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
app.include_router(push.router, prefix="/api") # Push notification subscription management
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
# Exception handlers
from fastapi import Request
from fastapi.responses import JSONResponse
from backend.exceptions import InsufficientCreditsError
from backend.i18n import t, get_locale_from_request
@app.exception_handler(InsufficientCreditsError)
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
"""Handle insufficient credits errors with 402 Payment Required status."""
locale = get_locale_from_request(request)
return JSONResponse(
status_code=402,
content={
"detail": t(locale, "errors.insufficientCredits"),
"credits_available": exc.credits_available,
"credits_required": exc.credits_required,
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
},
)
@app.get("/")
+1 -7
View File
@@ -1,11 +1,5 @@
"""Middleware package for FastAPI application."""
from backend.middleware.audit_logging import AuditLoggingMiddleware
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
__all__ = [
"AuditLoggingMiddleware",
"TenantMiddleware",
"get_tenant_from_request",
"get_tenant_config_from_request",
]
__all__ = ["AuditLoggingMiddleware"]
-192
View File
@@ -1,192 +0,0 @@
"""
Tenant detection middleware for white-label B2B portals.
This middleware extracts the tenant ID from incoming requests and attaches
the tenant configuration to the request state for use by downstream handlers.
Tenant detection priority:
1. X-Tenant-ID header (explicitly set by frontend)
2. `tenant` query parameter (for development/testing)
3. Host header subdomain detection (production flow)
The middleware is non-blocking - if no tenant is detected, the request
proceeds as a default Nomad Karaoke request (tenant_id = None).
"""
import logging
import os
from typing import Optional
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from backend.services.tenant_service import get_tenant_service
from backend.models.tenant import TenantConfig
logger = logging.getLogger(__name__)
# Only allow query param tenant override in non-production environments
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
os.environ.get("ENVIRONMENT", "").lower() == "production"
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
class TenantMiddleware(BaseHTTPMiddleware):
"""
Middleware that detects tenant from request and attaches config to state.
After this middleware runs, routes can access:
- request.state.tenant_id: str or None
- request.state.tenant_config: TenantConfig or None
Usage in route handlers:
```python
@router.get("/something")
async def handler(request: Request):
tenant_id = getattr(request.state, "tenant_id", None)
tenant_config = getattr(request.state, "tenant_config", None)
```
"""
async def dispatch(self, request: Request, call_next) -> Response:
# Extract tenant ID from request
tenant_id = self._extract_tenant_id(request)
# Load tenant config if tenant detected
tenant_config: Optional[TenantConfig] = None
if tenant_id:
tenant_service = get_tenant_service()
tenant_config = tenant_service.get_tenant_config(tenant_id)
if tenant_config and not tenant_config.is_active:
# Tenant exists but is inactive - treat as default
logger.warning(f"Inactive tenant requested: {tenant_id}")
tenant_id = None
tenant_config = None
# Attach to request state
request.state.tenant_id = tenant_id
request.state.tenant_config = tenant_config
# Log tenant detection for debugging
if tenant_id:
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
# Process request
response = await call_next(request)
# Add tenant ID to response headers for debugging
if tenant_id:
response.headers["X-Tenant-ID"] = tenant_id
return response
def _extract_tenant_id(self, request: Request) -> Optional[str]:
"""
Extract tenant ID from request using priority-based detection.
Returns:
Tenant ID if detected, None for default Nomad Karaoke
"""
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
header_tenant = request.headers.get("X-Tenant-ID")
if header_tenant:
return header_tenant.lower().strip()
# Priority 2: Query parameter (for development/testing only)
# Disabled in production to prevent tenant spoofing
if not IS_PRODUCTION:
query_tenant = request.query_params.get("tenant")
if query_tenant:
return query_tenant.lower().strip()
# Priority 3: Host header subdomain detection
host = request.headers.get("Host", "")
return self._extract_tenant_from_host(host)
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
"""
Extract tenant ID from Host header subdomain.
Patterns handled:
- vocalstar.nomadkaraoke.com -> vocalstar
- vocalstar.gen.nomadkaraoke.com -> vocalstar
- localhost:3000 -> None (local dev)
- gen.nomadkaraoke.com -> None (main app)
- api.nomadkaraoke.com -> None (API)
Returns:
Tenant ID if subdomain matches tenant, None otherwise
"""
if not host:
return None
# Normalize
host_lower = host.lower()
# Remove port if present
if ":" in host_lower:
host_lower = host_lower.split(":")[0]
# Check if this is a nomadkaraoke.com domain
if "nomadkaraoke.com" not in host_lower:
return None
# Split into parts
parts = host_lower.split(".")
# Need at least 3 parts: subdomain.nomadkaraoke.com
if len(parts) < 3:
return None
# First part is the potential tenant ID
potential_tenant = parts[0]
# Skip known non-tenant subdomains
if potential_tenant in NON_TENANT_SUBDOMAINS:
return None
# Verify tenant exists in GCS
tenant_service = get_tenant_service()
if tenant_service.tenant_exists(potential_tenant):
return potential_tenant
return None
def get_tenant_from_request(request: Request) -> Optional[str]:
"""
Helper function to get tenant ID from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_from_request
@router.get("/something")
async def handler(request: Request):
tenant_id = get_tenant_from_request(request)
```
"""
return getattr(request.state, "tenant_id", None)
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
"""
Helper function to get tenant config from request state.
Usage in route handlers:
```python
from backend.middleware.tenant import get_tenant_config_from_request
@router.get("/something")
async def handler(request: Request):
config = get_tenant_config_from_request(request)
if config and not config.features.audio_search:
raise HTTPException(403, "Audio search not available")
```
"""
return getattr(request.state, "tenant_config", None)
-110
View File
@@ -1,110 +0,0 @@
"""
Audio edit session model for persisting input audio editing progress.
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class AudioEditSessionSummary:
"""Compact summary of operations in an audio edit session, for list views."""
total_operations: int = 0
operations_breakdown: Dict[str, int] = field(default_factory=dict)
duration_change_seconds: float = 0.0
net_duration_seconds: float = 0.0
def to_dict(self) -> Dict[str, Any]:
return {
"total_operations": self.total_operations,
"operations_breakdown": self.operations_breakdown,
"duration_change_seconds": self.duration_change_seconds,
"net_duration_seconds": self.net_duration_seconds,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
return cls(
total_operations=data.get("total_operations", 0),
operations_breakdown=data.get("operations_breakdown", {}),
duration_change_seconds=data.get("duration_change_seconds", 0.0),
net_duration_seconds=data.get("net_duration_seconds", 0.0),
)
@dataclass
class AudioEditSession:
"""
A saved snapshot of an audio editing session.
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "manual" | "submit"
audio_duration_seconds: Optional[float] = None
original_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
edit_data_gcs_path: str = ""
# Hash of the edit data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"original_duration_seconds": self.original_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"edit_data_gcs_path": self.edit_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
original_duration_seconds=data.get("original_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
+56 -179
View File
@@ -9,70 +9,58 @@ from datetime import datetime
from enum import Enum
from pydantic import BaseModel, Field, validator
from karaoke_gen.utils import normalize_text
class JobStatus(str, Enum):
"""
Job status enumeration - Complete state machine.
The workflow has 8 main stages with up to 3 human interaction points:
The workflow has 8 main stages with 3 human interaction points:
1. Input & Setup (may include audio source selection)
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
2. Parallel Processing (audio + lyrics)
3. Title/End Screen Generation + Backing Vocals Analysis
3. Title/End Screen Generation
4. Countdown Padding Synchronization
5. Combined Human Review (BLOCKING) - lyrics review + instrumental selection
6. Video Finalization
7. Distribution
Note: As of 2026-01, lyrics review and instrumental selection are combined
into a single human review step. The user selects their instrumental
during the lyrics review phase, before the video is rendered.
5. Human Review (BLOCKING)
6. Instrumental Selection (BLOCKING)
7. Video Finalization
8. Distribution
"""
# Initial states
PENDING = "pending" # Job created, queued for processing
# Audio search states (for artist+title search mode)
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
DOWNLOADING = "downloading" # Downloading from URL or processing upload
# Optional: Audio editing (before processing begins)
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
# Stage 2a: Audio separation (parallel track 1)
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
# Stage 2b: Lyrics processing (parallel track 2)
TRANSCRIBING = "transcribing" # AudioShake API transcription
CORRECTING = "correcting" # Automatic lyrics correction
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
# Stage 3: Title/End screens + Backing Vocals Analysis
# Stage 3: Title/End screens
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
# Stage 4: Countdown padding (automatic)
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
# Stage 5: Combined Human Review (BLOCKING) - lyrics + instrumental selection
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - combined review needed
IN_REVIEW = "in_review" # User is actively reviewing
REVIEW_COMPLETE = "review_complete" # User submitted lyrics + instrumental selection
# Stage 5: Human review (BLOCKING)
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
IN_REVIEW = "in_review" # User is actively reviewing lyrics
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
# Stage 5.5: Render video with corrected lyrics (post-review)
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
# Stage 6: Instrumental already selected (during combined review)
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # LEGACY - no longer used
INSTRUMENTAL_SELECTED = "instrumental_selected" # Instrumental was selected during combined review
# Stage 6: Instrumental selection (BLOCKING)
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
# Stage 7: Video generation and finalization
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
@@ -99,83 +87,71 @@ class JobStatus(str, Enum):
# Valid state transitions
STATE_TRANSITIONS = {
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
# Audio search flow (for artist+title search mode)
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
# Can also enter optional audio edit phase if requires_audio_edit flag is set
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.FAILED],
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Audio separation flow
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Lyrics flow
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
# Post-parallel processing (screens + backing vocals analysis)
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Post-parallel processing
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
# Combined human review flow (lyrics + instrumental selection)
# Human review flow
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
# Video rendering (post-review) - instrumental was already selected during combined review
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
# Video rendering (post-review)
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
# Instrumental selection flow
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
# Video generation flow
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
# Distribution flow
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
# PREP_COMPLETE allows continuation from combined review
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
# PREP_COMPLETE allows finalise-only continuation
JobStatus.COMPLETE: [],
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
JobStatus.FAILED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
],
JobStatus.CANCELLED: [
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
JobStatus.REVIEW_COMPLETE, # Retry from render stage
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
JobStatus.AWAITING_REVIEW, # Retry from combined review
],
# Legacy states (for backward compatibility)
JobStatus.QUEUED: [JobStatus.PENDING],
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
@@ -191,7 +167,6 @@ class TimelineEvent(BaseModel):
timestamp: str
progress: Optional[int] = None
message: Optional[str] = None
metadata: Optional[Dict[str, Any]] = None
class LogEntry(BaseModel):
@@ -221,14 +196,7 @@ class Job(BaseModel):
title: Optional[str] = None
filename: Optional[str] = None # Original uploaded filename
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
# Audio source tracking (for retry capability)
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
download_url: Optional[str] = None # Direct download URL if available - for audio_search
# User preferences
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
@@ -237,9 +205,6 @@ class Job(BaseModel):
webhook_url: Optional[str] = None # Webhook for notifications
user_email: Optional[str] = None # Email for notifications
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
tenant_id: str = "" # Tenant ID for white-label portal scoping
# Theme configuration (pre-made themes from GCS)
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
@@ -309,31 +274,7 @@ class Job(BaseModel):
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Creation-time decision params (for observability — what did the user actually choose?)
creation_params: Dict[str, Any] = Field(default_factory=dict)
# Anti-abuse: IP address at job creation time (for cross-account correlation)
creation_ip: Optional[str] = None
# Output deletion tracking (for admin cleanup without deleting job)
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
credit_refunded: bool = False
# Edit tracking (how many times a completed track has been re-edited)
edit_count: int = 0
# Processing state
track_output_dir: Optional[str] = None # Local output directory (temp)
audio_hash: Optional[str] = None # Hash for deduplication
@@ -424,43 +365,6 @@ class Job(BaseModel):
}
"""
# Processing metadata (immutable provenance data written by workers during processing)
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
Immutable processing provenance data. Written once by each worker during processing.
Never modified after initial write. Used for analytics, debugging, and data export.
Structure:
{
"transcription": {
"provider": "audioshake",
"audioshake_task_id": "...",
"audioshake_asset_id": "...",
"language_detected": "en",
"word_count": 358,
"segment_count": 42,
},
"correction": {
"handlers_applied": [...],
"corrections_made": 12,
"correction_ratio": 0.034,
"reference_sources_found": ["genius", "spotify"],
"agentic_routing": "rule-based",
"duration_seconds": 33.2,
},
"separation": {
"provider": "modal",
"clean_model": "...",
"duration_seconds": 420.5,
},
"timing": {
"lyrics_worker_seconds": 125.3,
"audio_worker_seconds": 420.5,
...
},
}
"""
# Request metadata (captured at job creation for tracking and filtering)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -568,15 +472,7 @@ class JobCreate(BaseModel):
prep_only: bool = False # Stop after review, don't run finalisation
finalise_only: bool = False # Skip prep, run only finalisation
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
# Made-for-you order tracking
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
customer_notes: Optional[str] = None # Notes provided by customer with their order
# Private (non-published) track mode
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
# Request metadata (set by API endpoint from request headers)
request_metadata: Dict[str, Any] = Field(default_factory=dict)
"""
@@ -588,33 +484,14 @@ class JobCreate(BaseModel):
- server_version: Current server version
- custom_headers: All X-* headers
"""
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
tenant_id: str = "" # Tenant ID for job scoping
@validator('url')
def validate_url(cls, v):
"""Validate URL is not empty."""
@validator('url', 'artist', 'title')
def validate_inputs(cls, v):
"""Validate string inputs are not empty."""
if v is not None and isinstance(v, str) and not v.strip():
raise ValueError("Field cannot be empty string")
return v.strip() if isinstance(v, str) else v
@validator('artist', 'title')
def normalize_artist_title(cls, v):
"""Normalize artist/title text to standardize Unicode characters.
This ensures consistent data storage by converting:
- Curly quotes -> straight quotes
- Various dashes -> hyphen
- Unusual whitespace -> regular space
"""
if v is not None and isinstance(v, str):
if not v.strip():
raise ValueError("Field cannot be empty string")
# normalize_text handles stripping and Unicode normalization
return normalize_text(v)
return v
class JobResponse(BaseModel):
"""Job response model."""
-178
View File
@@ -1,178 +0,0 @@
"""
Referral system models.
Supports:
- Referral link creation and management
- Earnings tracking from referred purchases
- Payout processing via Stripe Connect
- API request/response shapes for referral endpoints
"""
from datetime import datetime
from typing import Optional, List
from pydantic import BaseModel, Field
# ============================================================================
# Firestore Document Models
# ============================================================================
class ReferralLinkStats(BaseModel):
"""Aggregated statistics for a referral link."""
clicks: int = 0
signups: int = 0
purchases: int = 0
total_earned_cents: int = 0
class ReferralLink(BaseModel):
"""
Referral link stored in Firestore.
Each user can have one referral link. Vanity links use a custom code
(e.g., "karaoke-king") while standard links use a generated code.
"""
code: str # Unique referral code (URL slug)
owner_email: str # Email of the referrer
display_name: Optional[str] = None # Public name shown on interstitial
custom_message: Optional[str] = None # Optional message on interstitial
# Reward configuration
discount_percent: int = 10 # Discount for referred user
kickback_percent: int = 20 # Earnings for referrer
discount_duration_days: int = 30 # How long the discount lasts
earning_duration_days: int = 365 # How long referrer earns from this link
# Stripe
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
# Flags
is_vanity: bool = False # Custom vanity code vs generated
enabled: bool = True # Can be disabled without deleting
# Timestamps
created_at: datetime = Field(default_factory=datetime.utcnow)
updated_at: datetime = Field(default_factory=datetime.utcnow)
# Aggregated stats
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
class ReferralEarning(BaseModel):
"""
Record of a single earning from a referred purchase.
Created when a referred user completes a Stripe checkout.
"""
id: str
referrer_email: str
referred_email: str
referral_code: str
stripe_session_id: str
purchase_amount_cents: int
earning_amount_cents: int
status: str = "pending" # pending, paid, cancelled
created_at: datetime = Field(default_factory=datetime.utcnow)
paid_at: Optional[datetime] = None
payout_id: Optional[str] = None
class ReferralPayout(BaseModel):
"""
Record of a payout to a referrer via Stripe Connect.
Groups multiple earnings into a single transfer.
"""
id: str
referrer_email: str
stripe_transfer_id: str
amount_cents: int
earnings_included: List[str] # List of ReferralEarning IDs
status: str = "processing" # processing, completed, failed
created_at: datetime = Field(default_factory=datetime.utcnow)
# ============================================================================
# API Request Models
# ============================================================================
class CreateReferralLinkRequest(BaseModel):
"""Admin request to create a vanity referral link."""
vanity_code: Optional[str] = None
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int = 10
kickback_percent: int = 20
discount_duration_days: int = 30
earning_duration_days: int = 365
class UpdateReferralLinkRequest(BaseModel):
"""Request to update a referral link (user: name/message only, admin: all fields)."""
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: Optional[int] = None
kickback_percent: Optional[int] = None
discount_duration_days: Optional[int] = None
earning_duration_days: Optional[int] = None
enabled: Optional[bool] = None
# ============================================================================
# API Response Models
# ============================================================================
class ReferralLinkResponse(BaseModel):
"""Public referral link information."""
code: str
owner_email: str
display_name: Optional[str] = None
custom_message: Optional[str] = None
discount_percent: int
kickback_percent: int
is_vanity: bool
enabled: bool
stats: ReferralLinkStats
created_at: datetime
class ReferralEarningResponse(BaseModel):
"""Public earning information (for referrer dashboard)."""
id: str
referred_email: str
referral_code: str
purchase_amount_cents: int
earning_amount_cents: int
status: str
created_at: datetime
paid_at: Optional[datetime] = None
class ReferralPayoutResponse(BaseModel):
"""Public payout information (for referrer dashboard)."""
id: str
amount_cents: int
earnings_count: int
status: str
created_at: datetime
class ReferralDashboardResponse(BaseModel):
"""Full referral dashboard data for the referrer."""
referral_link: ReferralLinkResponse
recent_earnings: List[ReferralEarningResponse]
recent_payouts: List[ReferralPayoutResponse]
total_earned_cents: int
total_paid_cents: int
pending_balance_cents: int
stripe_connect_onboarded: bool
class ReferralInterstitialResponse(BaseModel):
"""Data for the referral landing/interstitial page."""
valid: bool = True
referral_code: str = ""
referrer_display_name: Optional[str] = None
discount_percent: int = 0
discount_duration_days: int = 0
custom_message: Optional[str] = None
+6 -49
View File
@@ -10,15 +10,14 @@ class URLSubmissionRequest(BaseModel):
url: HttpUrl
artist: Optional[str] = None # Auto-detected if not provided
title: Optional[str] = None # Auto-detected if not provided
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
is_private: Optional[bool] = None # Private (non-published) track mode
class UploadSubmissionRequest(BaseModel):
@@ -29,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
# Optional preferences
enable_cdg: bool = False # Requires style config
enable_txt: bool = False # Requires style config
enable_youtube_upload: Optional[bool] = None # None = use server default
enable_youtube_upload: bool = False
youtube_description: Optional[str] = None
webhook_url: Optional[str] = None
user_email: Optional[str] = None
@@ -38,13 +37,12 @@ class UploadSubmissionRequest(BaseModel):
class CorrectionsSubmission(BaseModel):
"""
Request to submit corrected lyrics after human review.
This is the critical human-in-the-loop interaction point.
The corrections data comes from the lyrics-transcriber review interface.
"""
corrections: Dict[str, Any] # Full corrections JSON from frontend
user_notes: Optional[str] = None # Optional notes from reviewer
is_duet: Optional[bool] = None # When present, persists duet flag to state_data
@validator('corrections')
def validate_corrections_format(cls, v):
@@ -59,13 +57,13 @@ class CorrectionsSubmission(BaseModel):
class InstrumentalSelection(BaseModel):
"""
Request to select instrumental audio option.
This is the second critical human-in-the-loop interaction point.
User chooses between clean instrumental, instrumental with backing vocals,
or a custom instrumental (created via create-custom-instrumental endpoint).
"""
selection: str # "clean", "with_backing", or "custom"
@validator('selection')
def validate_selection(cls, v):
"""Validate selection is a valid option."""
@@ -75,26 +73,6 @@ class InstrumentalSelection(BaseModel):
return v
class CompleteReviewRequest(BaseModel):
"""
Request to complete the review with optional instrumental selection.
For the combined review flow, this includes the instrumental selection.
For legacy flows, the instrumental_selection can be omitted.
"""
instrumental_selection: Optional[str] = None # "clean", "with_backing", or "custom"
@validator('instrumental_selection')
def validate_instrumental_selection(cls, v):
"""Validate instrumental selection if provided."""
if v is None:
return v
valid_options = ['clean', 'with_backing', 'custom']
if v not in valid_options:
raise ValueError(f"instrumental_selection must be one of: {valid_options}")
return v
class MuteRegionRequest(BaseModel):
"""A region to mute in the backing vocals."""
start_seconds: float
@@ -143,24 +121,3 @@ class RetryJobRequest(BaseModel):
"""Request to retry a failed job."""
from_stage: Optional[str] = None # Optional: restart from specific stage
class ChangeVisibilityRequest(BaseModel):
"""Request to change job visibility (public/private)."""
target_visibility: str # "public" or "private"
@validator('target_visibility')
def validate_target(cls, v):
if v not in ('public', 'private'):
raise ValueError("target_visibility must be 'public' or 'private'")
return v
class ChangeVisibilityResponse(BaseModel):
"""Response from change visibility endpoint."""
status: str # "success", "processing", "error"
job_id: str
message: str
previous_visibility: str # "public" or "private"
new_visibility: str # "public" or "private"
reprocessing_required: bool
-107
View File
@@ -1,107 +0,0 @@
"""
Review session model for persisting lyrics review progress.
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
"""
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Optional, Dict, Any, List
import uuid
@dataclass
class ReviewSessionSummary:
"""Compact summary of edits in a review session, for list views."""
total_segments: int = 0
total_words: int = 0
corrections_made: int = 0
changed_words: List[Dict[str, Any]] = field(default_factory=list)
def to_dict(self) -> Dict[str, Any]:
return {
"total_segments": self.total_segments,
"total_words": self.total_words,
"corrections_made": self.corrections_made,
"changed_words": self.changed_words,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
return cls(
total_segments=data.get("total_segments", 0),
total_words=data.get("total_words", 0),
corrections_made=data.get("corrections_made", 0),
changed_words=data.get("changed_words", []),
)
@dataclass
class ReviewSession:
"""
A saved snapshot of a lyrics review editing session.
Stored at: jobs/{job_id}/review_sessions/{session_id}
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
"""
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
job_id: str = ""
user_email: str = ""
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
edit_count: int = 0
trigger: str = "auto" # "auto" | "preview" | "manual"
audio_duration_seconds: Optional[float] = None
artist: Optional[str] = None
title: Optional[str] = None
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
correction_data_gcs_path: str = ""
# Hash of the correction data JSON, used to skip duplicate saves
data_hash: str = ""
def to_dict(self) -> Dict[str, Any]:
return {
"session_id": self.session_id,
"job_id": self.job_id,
"user_email": self.user_email,
"created_at": self.created_at,
"updated_at": self.updated_at,
"edit_count": self.edit_count,
"trigger": self.trigger,
"audio_duration_seconds": self.audio_duration_seconds,
"artist": self.artist,
"title": self.title,
"summary": self.summary.to_dict(),
"correction_data_gcs_path": self.correction_data_gcs_path,
"data_hash": self.data_hash,
}
@classmethod
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
created_at = data.get("created_at")
if isinstance(created_at, str):
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
updated_at = data.get("updated_at")
if isinstance(updated_at, str):
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
summary_data = data.get("summary", {})
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
return cls(
session_id=data.get("session_id", str(uuid.uuid4())),
job_id=data.get("job_id", ""),
user_email=data.get("user_email", ""),
created_at=created_at or datetime.now(timezone.utc),
updated_at=updated_at or datetime.now(timezone.utc),
edit_count=data.get("edit_count", 0),
trigger=data.get("trigger", "auto"),
audio_duration_seconds=data.get("audio_duration_seconds"),
artist=data.get("artist"),
title=data.get("title"),
summary=summary,
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
data_hash=data.get("data_hash", ""),
)
-218
View File
@@ -1,218 +0,0 @@
"""
Tenant data models for white-label B2B portals.
Tenants are B2B customers who get their own branded portal with custom
configuration. Each tenant has:
- Custom branding (logo, colors, site title)
- Feature flags (enable/disable audio search, distribution, etc.)
- Default settings (theme, distribution mode)
- Auth configuration (allowed email domains, fixed tokens)
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
"""
from datetime import datetime
from typing import List, Optional
from pydantic import BaseModel, Field
class TenantBranding(BaseModel):
"""Branding configuration for a tenant's portal."""
logo_url: Optional[str] = Field(
None, description="URL to tenant's logo image (PNG with transparency preferred)"
)
logo_height: int = Field(40, description="Logo height in pixels for header display")
primary_color: str = Field(
"#ff5bb8", description="Primary brand color (hex format)"
)
secondary_color: str = Field(
"#8b5cf6", description="Secondary brand color (hex format)"
)
accent_color: Optional[str] = Field(
None, description="Accent color for highlights (hex format)"
)
background_color: Optional[str] = Field(
None, description="Custom background color (defaults to dark theme)"
)
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
site_title: str = Field(
"Karaoke Generator", description="Browser tab title"
)
tagline: Optional[str] = Field(
None, description="Optional tagline shown below logo"
)
class TenantFeatures(BaseModel):
"""Feature flags controlling what's available in the tenant's portal."""
# Audio input methods
audio_search: bool = Field(
True, description="Enable audio search (flacfetch integration)"
)
file_upload: bool = Field(True, description="Enable direct file upload")
youtube_url: bool = Field(True, description="Enable YouTube URL input")
# Distribution options
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
# Customization options
theme_selection: bool = Field(
True, description="Allow user to select theme (false = use default)"
)
color_overrides: bool = Field(
True, description="Allow user to customize colors"
)
# Output formats
enable_cdg: bool = Field(True, description="Generate CDG format output")
enable_4k: bool = Field(True, description="Generate 4K video output")
# Advanced features
admin_access: bool = Field(
False, description="Allow access to admin dashboard"
)
class TenantDefaults(BaseModel):
"""Default settings applied to all jobs for this tenant."""
theme_id: Optional[str] = Field(
None, description="Default theme ID (required if theme_selection=false)"
)
locked_theme: Optional[str] = Field(
None,
description="If set, users cannot change theme - always uses this theme ID"
)
distribution_mode: str = Field(
"all",
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
)
brand_prefix: Optional[str] = Field(
None, description="Prefix for output filenames (e.g., 'VSTAR')"
)
dropbox_path: Optional[str] = Field(
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
)
gdrive_folder_id: Optional[str] = Field(
None, description="Tenant-specific Google Drive folder ID for output uploads"
)
youtube_description_template: Optional[str] = Field(
None, description="Default YouTube description template"
)
class TenantAuth(BaseModel):
"""Authentication configuration for a tenant."""
allowed_email_domains: List[str] = Field(
default_factory=list,
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
)
require_email_domain: bool = Field(
True,
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
)
fixed_token_ids: List[str] = Field(
default_factory=list,
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
)
sender_email: Optional[str] = Field(
None,
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
)
class TenantConfig(BaseModel):
"""Complete configuration for a white-label tenant."""
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
subdomain: str = Field(
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
)
is_active: bool = Field(True, description="Whether tenant portal is active")
branding: TenantBranding = Field(default_factory=TenantBranding)
features: TenantFeatures = Field(default_factory=TenantFeatures)
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
auth: TenantAuth = Field(default_factory=TenantAuth)
created_at: Optional[datetime] = None
updated_at: Optional[datetime] = None
def get_frontend_url(self) -> str:
"""Get the frontend URL for this tenant's portal."""
return f"https://{self.subdomain}"
def get_sender_email(self) -> str:
"""Get the email sender address for this tenant."""
if self.auth.sender_email:
return self.auth.sender_email
# Default pattern: {tenant_id}@nomadkaraoke.com
return f"{self.id}@nomadkaraoke.com"
def is_email_allowed(self, email: str) -> bool:
"""Check if an email address is allowed for this tenant."""
if not self.auth.allowed_email_domains:
# No domain restrictions
return True
email_lower = email.lower()
for domain in self.auth.allowed_email_domains:
if email_lower.endswith(f"@{domain.lower()}"):
return True
return not self.auth.require_email_domain
class TenantPublicConfig(BaseModel):
"""
Public tenant configuration returned to frontend.
This excludes sensitive auth details like token IDs.
"""
id: str
name: str
subdomain: str
is_active: bool
branding: TenantBranding
features: TenantFeatures
defaults: TenantDefaults
# Auth info limited to what frontend needs
allowed_email_domains: List[str] = Field(default_factory=list)
@classmethod
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
"""Create public config from full config."""
return cls(
id=config.id,
name=config.name,
subdomain=config.subdomain,
is_active=config.is_active,
branding=config.branding,
features=config.features,
defaults=TenantDefaults(
theme_id=config.defaults.theme_id,
locked_theme=config.defaults.locked_theme,
distribution_mode=config.defaults.distribution_mode,
# Don't expose brand_prefix or youtube_description_template
),
allowed_email_domains=config.auth.allowed_email_domains,
)
class TenantConfigResponse(BaseModel):
"""Response from GET /api/tenant/config endpoint."""
tenant: Optional[TenantPublicConfig] = Field(
None, description="Tenant config if found, null for default Nomad Karaoke"
)
is_default: bool = Field(
True, description="True if using default Nomad Karaoke config"
)
+52 -109
View File
@@ -6,10 +6,11 @@ Supports:
- Credit system for karaoke generation
- Role-based access control (user/admin)
- Stripe integration for payments
- Beta tester program with feedback collection
"""
from datetime import datetime, timezone
from datetime import datetime
from enum import Enum
from typing import Optional, List, Dict
from typing import Optional, List
from pydantic import BaseModel, Field
@@ -19,6 +20,14 @@ class UserRole(str, Enum):
ADMIN = "admin"
class BetaTesterStatus(str, Enum):
"""Status of beta tester participation."""
ACTIVE = "active" # Enrolled, free credits available
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
COMPLETED = "completed" # Feedback submitted
EXPIRED = "expired" # 24hr deadline passed without feedback
class CreditTransaction(BaseModel):
"""Record of a credit transaction."""
id: str
@@ -30,42 +39,17 @@ class CreditTransaction(BaseModel):
created_by: Optional[str] = None # Admin email if granted by admin
class PushSubscription(BaseModel):
"""
Web Push subscription for a user's device.
Stores the push subscription endpoint and encryption keys needed
to send push notifications to the user's browser/device.
Subscriptions are scoped by tenant_id to prevent cross-tenant
notification leakage (e.g., a consumer job notification appearing
on a tenant portal's service worker).
"""
endpoint: str # Push service endpoint URL
keys: Dict[str, str] # p256dh and auth keys for encryption
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
last_used_at: Optional[datetime] = None # Last time a notification was sent
class User(BaseModel):
"""
User model stored in Firestore.
Users are identified by email address. Authentication is via magic links
(no passwords). Credits are consumed when creating karaoke jobs.
For multi-tenant white-label portals, users are scoped to a tenant.
Users with tenant_id=None are default Nomad Karaoke users.
"""
email: str # Primary identifier
role: UserRole = UserRole.USER
credits: int = 0
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Stripe integration
stripe_customer_id: Optional[str] = None
@@ -85,35 +69,16 @@ class User(BaseModel):
total_jobs_created: int = 0
total_jobs_completed: int = 0
# Payment tracking (cached from stripe_payments, in cents)
total_spent: int = 0
# Optional profile fields for future use
display_name: Optional[str] = None
# i18n locale preference (set from Accept-Language on login, used for emails/push)
locale: Optional[str] = None
# Push notification subscriptions (Web Push API)
# Users can subscribe from multiple devices/browsers
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
# Feedback-for-credits program
has_submitted_feedback: bool = False
# Anti-abuse: IP address and device fingerprint used during signup
signup_ip: Optional[str] = None
device_fingerprint: Optional[str] = None
# Flag to prevent duplicate welcome credit grants (idempotency)
welcome_credits_granted: bool = False
# Referral system
referral_code: Optional[str] = None # User's own referral code
referred_by_code: Optional[str] = None # Code used when this user signed up
referred_at: Optional[datetime] = None # When the referral was applied
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
# Beta tester program
is_beta_tester: bool = False
beta_tester_status: Optional[BetaTesterStatus] = None
beta_enrolled_at: Optional[datetime] = None
beta_promise_text: Optional[str] = None # User's promise statement
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
beta_feedback_email_sent: bool = False
class MagicLinkToken(BaseModel):
@@ -122,9 +87,6 @@ class MagicLinkToken(BaseModel):
Tokens are short-lived (15 minutes) and single-use.
Stored in Firestore with automatic TTL.
For multi-tenant portals, the tenant_id determines which portal
the user will be redirected to after verification.
"""
token: str # Secure random token
email: str
@@ -134,18 +96,6 @@ class MagicLinkToken(BaseModel):
used_at: Optional[datetime] = None
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Referral attribution (travels with magic link for cross-device support)
referral_code: Optional[str] = None
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
credit_eval_reasoning: Optional[str] = None
credit_eval_error: Optional[str] = None
class Session(BaseModel):
@@ -154,8 +104,6 @@ class Session(BaseModel):
Sessions are created after successful magic link verification.
Sessions expire after 7 days of inactivity or 30 days absolute.
For multi-tenant portals, sessions are scoped to a tenant.
"""
token: str # Secure random session token
user_email: str
@@ -164,20 +112,14 @@ class Session(BaseModel):
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
ip_address: Optional[str] = None
user_agent: Optional[str] = None
device_fingerprint: Optional[str] = None
is_active: bool = True
# Multi-tenant support (None = default Nomad Karaoke)
tenant_id: Optional[str] = None
# Pydantic models for API requests/responses
class SendMagicLinkRequest(BaseModel):
"""Request to send a magic link email."""
email: str
device_fingerprint: Optional[str] = None
referral_code: Optional[str] = None # From referral interstitial
class SendMagicLinkResponse(BaseModel):
@@ -197,9 +139,6 @@ class VerifyMagicLinkResponse(BaseModel):
session_token: str
user: "UserPublic"
message: str
tenant_subdomain: Optional[str] = None
credits_granted: int = 0
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
class UserPublic(BaseModel):
@@ -210,16 +149,6 @@ class UserPublic(BaseModel):
display_name: Optional[str] = None
total_jobs_created: int = 0
total_jobs_completed: int = 0
tenant_id: Optional[str] = None
feedback_eligible: bool = False
total_spent: int = 0
referral_code: Optional[str] = None
has_active_referral_discount: bool = False
referral_discount_percent: Optional[int] = None
referral_discount_expires_at: Optional[str] = None # ISO string
referred_by_code: Optional[str] = None
created_at: Optional[str] = None
last_login_at: Optional[str] = None
class AddCreditsRequest(BaseModel):
@@ -245,18 +174,18 @@ class UserListResponse(BaseModel):
# ============================================================================
# User Feedback-for-Credits Models
# Beta Tester Program Models
# ============================================================================
class UserFeedback(BaseModel):
class BetaTesterFeedback(BaseModel):
"""
Feedback from any user (not just beta testers).
Feedback from a beta tester after using the service.
Stored in Firestore 'user_feedback' collection.
Users earn 1 free credit for submitting detailed feedback.
Stored in Firestore 'beta_feedback' collection.
"""
id: str
id: str # Unique feedback ID
user_email: str
job_id: Optional[str] = None # The job they're providing feedback on
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
@@ -275,11 +204,33 @@ class UserFeedback(BaseModel):
# Metadata
created_at: datetime = Field(default_factory=datetime.utcnow)
submitted_via: str = "web"
submitted_via: str = "web" # 'web' or 'email'
class UserFeedbackRequest(BaseModel):
"""Request to submit user feedback for credits."""
class BetaTesterEnrollRequest(BaseModel):
"""Request to enroll as a beta tester."""
email: str
promise_text: str = Field(
min_length=10,
description="User's promise to provide feedback (min 10 chars)"
)
accept_corrections_work: bool = Field(
description="User accepts they may need to review/correct lyrics"
)
class BetaTesterEnrollResponse(BaseModel):
"""Response after enrolling as beta tester."""
status: str
message: str
credits_granted: int
session_token: Optional[str] = None # If new user, provide session
class BetaFeedbackRequest(BaseModel):
"""Request to submit beta tester feedback."""
job_id: Optional[str] = None
# Ratings (1-5 scale)
overall_rating: int = Field(ge=1, le=5)
ease_of_use_rating: int = Field(ge=1, le=5)
@@ -296,16 +247,8 @@ class UserFeedbackRequest(BaseModel):
would_use_again: bool = True
class UserFeedbackResponse(BaseModel):
"""Response after submitting user feedback."""
class BetaFeedbackResponse(BaseModel):
"""Response after submitting feedback."""
status: str
message: str
credits_granted: int = 0
class FeedbackEligibilityResponse(BaseModel):
"""Response for feedback eligibility check."""
eligible: bool
has_submitted: bool
jobs_completed: int
credits_reward: int = 1
bonus_credits: int = 0 # Bonus credits for great feedback
-48
View File
@@ -1,48 +0,0 @@
#!/usr/bin/env python3
"""
Download ensemble preset models at Docker build time.
Bakes models into the image so they're available immediately at runtime,
avoiding cold-start model download latency in Cloud Run Jobs.
Usage: python backend/scripts/download_models.py /models
"""
import json
import sys
import logging
import importlib.resources as resources
from audio_separator.separator import Separator
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Presets to bake into the GPU image
PRESETS_TO_DOWNLOAD = ["instrumental_clean", "karaoke"]
def download_preset_models(model_dir: str) -> None:
"""Download all models for the configured presets individually.
Ensemble presets resolve to multiple model filenames. We must download
each model individually via load_model(model_filename) — calling
load_model() with no args on an ensemble preset doesn't trigger downloads.
"""
with resources.open_text("audio_separator", "ensemble_presets.json") as f:
presets = json.load(f)["presets"]
models_to_download = set()
for preset_name in PRESETS_TO_DOWNLOAD:
models_to_download.update(presets[preset_name]["models"])
logger.info(f"Downloading {len(models_to_download)} models for presets: {PRESETS_TO_DOWNLOAD}")
for model in sorted(models_to_download):
logger.info(f" Downloading: {model}")
sep = Separator(model_file_dir=model_dir)
sep.load_model(model)
logger.info(f" Done: {model}")
if __name__ == "__main__":
model_dir = sys.argv[1] if len(sys.argv) > 1 else "/models"
download_preset_models(model_dir)
logger.info(f"All preset models downloaded to {model_dir}")
-265
View File
@@ -1,265 +0,0 @@
#!/usr/bin/env python3
"""
Generate a Mermaid state diagram from the job state machine.
This script reads the STATE_TRANSITIONS dictionary from models/job.py
and generates a Mermaid diagram that can be embedded in documentation.
Usage:
python scripts/generate_state_diagram.py > docs/state-machine.mmd
# Or to update ARCHITECTURE.md directly:
python scripts/generate_state_diagram.py --update-docs
Created as part of state machine robustness improvements (2026-02-02).
"""
import sys
import argparse
from pathlib import Path
# Add backend to path for imports
backend_dir = Path(__file__).parent.parent
sys.path.insert(0, str(backend_dir))
# Import the state machine data
# Note: Must be run from an environment where karaoke_gen is installed
try:
from models.job import STATE_TRANSITIONS, JobStatus
except ImportError:
# Fallback: define the state machine inline for documentation purposes
from enum import Enum
class JobStatus(str, Enum):
PENDING = "pending"
SEARCHING_AUDIO = "searching_audio"
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
DOWNLOADING_AUDIO = "downloading_audio"
DOWNLOADING = "downloading"
SEPARATING_STAGE1 = "separating_stage1"
SEPARATING_STAGE2 = "separating_stage2"
AUDIO_COMPLETE = "audio_complete"
TRANSCRIBING = "transcribing"
CORRECTING = "correcting"
LYRICS_COMPLETE = "lyrics_complete"
GENERATING_SCREENS = "generating_screens"
APPLYING_PADDING = "applying_padding"
AWAITING_REVIEW = "awaiting_review"
IN_REVIEW = "in_review"
REVIEW_COMPLETE = "review_complete"
RENDERING_VIDEO = "rendering_video"
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
INSTRUMENTAL_SELECTED = "instrumental_selected"
GENERATING_VIDEO = "generating_video"
ENCODING = "encoding"
PACKAGING = "packaging"
UPLOADING = "uploading"
NOTIFYING = "notifying"
COMPLETE = "complete"
PREP_COMPLETE = "prep_complete"
FAILED = "failed"
CANCELLED = "cancelled"
QUEUED = "queued"
PROCESSING = "processing"
ERROR = "error"
READY_FOR_FINALIZATION = "ready_for_finalization"
FINALIZING = "finalizing"
STATE_TRANSITIONS = {
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
}
def generate_mermaid() -> str:
"""
Generate a Mermaid state diagram from STATE_TRANSITIONS.
Returns:
Mermaid diagram as a string
"""
lines = ["stateDiagram-v2"]
lines.append(" direction TB")
lines.append("")
# Group states by category for better visualization
categories = {
"Initial": [JobStatus.PENDING],
"Audio Search": [
JobStatus.SEARCHING_AUDIO,
JobStatus.AWAITING_AUDIO_SELECTION,
JobStatus.DOWNLOADING_AUDIO,
],
"Processing": [
JobStatus.DOWNLOADING,
JobStatus.SEPARATING_STAGE1,
JobStatus.SEPARATING_STAGE2,
JobStatus.AUDIO_COMPLETE,
JobStatus.TRANSCRIBING,
JobStatus.CORRECTING,
JobStatus.LYRICS_COMPLETE,
],
"Screens": [
JobStatus.GENERATING_SCREENS,
JobStatus.APPLYING_PADDING,
],
"Review": [
JobStatus.AWAITING_REVIEW,
JobStatus.IN_REVIEW,
JobStatus.REVIEW_COMPLETE,
],
"Video": [
JobStatus.RENDERING_VIDEO,
JobStatus.INSTRUMENTAL_SELECTED,
JobStatus.GENERATING_VIDEO,
JobStatus.ENCODING,
JobStatus.PACKAGING,
],
"Distribution": [
JobStatus.UPLOADING,
JobStatus.NOTIFYING,
],
"Terminal": [
JobStatus.COMPLETE,
JobStatus.PREP_COMPLETE,
JobStatus.FAILED,
JobStatus.CANCELLED,
],
}
# Add state definitions with notes for key states
lines.append(" %% State definitions")
key_states = {
JobStatus.PENDING: "Job created",
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
JobStatus.COMPLETE: "✅ Success",
JobStatus.FAILED: "❌ Error",
}
for status, note in key_states.items():
lines.append(f" {status.value}: {status.value}")
lines.append(f" note right of {status.value}: {note}")
lines.append("")
lines.append(" %% Transitions")
# Generate transitions
for from_status, to_statuses in STATE_TRANSITIONS.items():
for to_status in to_statuses:
# Skip deprecated statuses
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
continue
lines.append(f" {from_status.value} --> {to_status.value}")
# Add start and end markers
lines.append("")
lines.append(" %% Start and end")
lines.append(" [*] --> pending")
lines.append(" complete --> [*]")
lines.append(" prep_complete --> [*]")
lines.append(" failed --> [*]")
lines.append(" cancelled --> [*]")
return "\n".join(lines)
def generate_markdown_section() -> str:
"""
Generate a markdown section with the state diagram.
Returns:
Markdown string with the diagram
"""
diagram = generate_mermaid()
return f"""## Job State Machine
The following diagram shows all valid state transitions for a karaoke generation job.
Key states:
- **PENDING**: Initial state when job is created
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
- **COMPLETE**: All processing finished successfully
```mermaid
{diagram}
```
### Parallel Processing
Jobs use parallel processing for audio separation and lyrics transcription:
1. **DOWNLOADING** triggers both:
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
2. When BOTH complete, the screens worker is triggered automatically.
### Human Review Points
Jobs wait for human input at these points:
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
### Error Handling
- Any state can transition to **FAILED** on unrecoverable errors
- **CANCELLED** is set when user cancels the job
"""
def main():
parser = argparse.ArgumentParser(description="Generate state machine diagram")
parser.add_argument(
"--update-docs",
action="store_true",
help="Update ARCHITECTURE.md with the diagram"
)
parser.add_argument(
"--markdown",
action="store_true",
help="Output markdown section instead of raw Mermaid"
)
args = parser.parse_args()
if args.update_docs:
# TODO: Implement auto-update of ARCHITECTURE.md
print("Auto-update of ARCHITECTURE.md not yet implemented")
print("Please copy the output below to the appropriate section:")
print()
print(generate_markdown_section())
elif args.markdown:
print(generate_markdown_section())
else:
print(generate_mermaid())
if __name__ == "__main__":
main()
+1 -73
View File
@@ -5,7 +5,6 @@ This service wraps the shared karaoke_gen.instrumental_review module
to provide GCS-integrated audio analysis capabilities.
"""
import json
import logging
import os
import tempfile
@@ -188,78 +187,7 @@ class AudioAnalysisService:
)
return amplitudes, duration
def cache_waveform_data(
self,
gcs_audio_path: str,
job_id: str,
cache_gcs_path: str,
num_points: int = 1000,
) -> tuple[list[float], float]:
"""
Generate waveform data and cache it as JSON in GCS.
Args:
gcs_audio_path: Path to the audio file in GCS
job_id: Job ID for logging
cache_gcs_path: GCS path to store the cached waveform JSON
num_points: Number of data points to generate
Returns:
Tuple of (amplitude_values, duration_seconds)
"""
logger.info(f"[{job_id}] Generating and caching waveform data: {gcs_audio_path}")
amplitudes, duration = self.get_waveform_data(
gcs_audio_path=gcs_audio_path,
job_id=job_id,
num_points=num_points,
)
cache_data = {
"amplitudes": list(amplitudes),
"duration_seconds": duration,
"num_points": num_points,
"source_gcs_path": gcs_audio_path,
}
self.storage_service.upload_json(cache_gcs_path, cache_data)
logger.info(f"[{job_id}] Waveform data cached at {cache_gcs_path}")
return amplitudes, duration
def load_cached_waveform(
self,
cache_gcs_path: str,
) -> Optional[tuple[list[float], float]]:
"""
Load cached waveform data from GCS.
Args:
cache_gcs_path: GCS path to the cached waveform JSON
Returns:
Tuple of (amplitude_values, duration_seconds) or None if not found
"""
try:
if not self.storage_service.file_exists(cache_gcs_path):
return None
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "waveform.json")
self.storage_service.download_file(cache_gcs_path, local_path)
with open(local_path) as f:
data = json.load(f)
amplitudes = data.get("amplitudes", [])
duration = data.get("duration_seconds", 0.0)
if not amplitudes:
return None
return amplitudes, duration
except Exception as e:
logger.warning(f"Failed to load cached waveform from {cache_gcs_path}: {e}")
return None
def generate_waveform_with_mutes(
self,
gcs_audio_path: str,
-180
View File
@@ -1,180 +0,0 @@
"""
Audio editing service — server-side FFmpeg operations for trim, cut, mute, join.
All operations are lossless (FLAC in, FLAC out). Playback URLs use OGG Opus
via the existing AudioTranscodingService.
"""
import logging
import os
import json
import subprocess
import tempfile
from dataclasses import dataclass, asdict
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
@dataclass
class AudioMetadata:
duration_seconds: float
sample_rate: int
channels: int
format: str
file_size_bytes: int
class AudioEditService:
"""Server-side audio editing via FFmpeg."""
def __init__(self, storage_service: StorageService | None = None):
self.storage = storage_service or StorageService()
def get_metadata(self, audio_path: str) -> AudioMetadata:
"""Get audio metadata from a local file using ffprobe."""
cmd = [
"ffprobe", "-v", "quiet", "-print_format", "json",
"-show_format", "-show_streams", audio_path,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if result.returncode != 0:
raise RuntimeError(f"ffprobe failed: {result.stderr}")
info = json.loads(result.stdout)
fmt = info.get("format", {})
streams = info.get("streams", [])
audio_stream = next((s for s in streams if s.get("codec_type") == "audio"), {})
return AudioMetadata(
duration_seconds=float(fmt.get("duration", 0)),
sample_rate=int(audio_stream.get("sample_rate", 44100)),
channels=int(audio_stream.get("channels", 2)),
format=fmt.get("format_name", "unknown"),
file_size_bytes=int(fmt.get("size", 0)),
)
def get_metadata_from_gcs(self, gcs_path: str) -> AudioMetadata:
"""Download from GCS and get metadata."""
with tempfile.TemporaryDirectory() as temp_dir:
local_path = os.path.join(temp_dir, "audio")
self.storage.download_file(gcs_path, local_path)
return self.get_metadata(local_path)
def trim_start(self, input_path: str, end_seconds: float, output_path: str) -> AudioMetadata:
"""Remove audio from 0 to end_seconds (skip the first N seconds)."""
self._run_ffmpeg([
"-ss", str(end_seconds),
"-i", input_path,
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def trim_end(self, input_path: str, start_seconds: float, output_path: str) -> AudioMetadata:
"""Keep audio from 0 to start_seconds (remove from start_seconds to end)."""
self._run_ffmpeg([
"-i", input_path,
"-t", str(start_seconds),
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def cut_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Remove a region from start to end, joining the remaining parts."""
self._run_ffmpeg([
"-i", input_path,
"-filter_complex",
f"[0]atrim=0:{start},asetpts=PTS-STARTPTS[a];"
f"[0]atrim={end},asetpts=PTS-STARTPTS[b];"
f"[a][b]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def mute_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
"""Silence a region (preserve duration)."""
self._run_ffmpeg([
"-i", input_path,
"-af", f"volume=enable='between(t,{start},{end})':volume=0",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def join_audio(self, input_path: str, other_path: str, position: str, output_path: str) -> AudioMetadata:
"""Join two audio files. position: 'start' (prepend other) or 'end' (append other)."""
if position == "start":
first, second = other_path, input_path
else:
first, second = input_path, other_path
self._run_ffmpeg([
"-i", first,
"-i", second,
"-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1[out]",
"-map", "[out]",
"-c:a", "flac",
output_path,
])
return self.get_metadata(output_path)
def apply_edit(
self,
input_gcs_path: str,
operation: str,
params: dict,
output_gcs_path: str,
job_id: str,
) -> tuple[AudioMetadata, str]:
"""
Apply an edit operation to an audio file in GCS.
Returns (metadata, output_gcs_path).
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.flac")
# Download current audio
self.storage.download_file(input_gcs_path, local_input)
# Apply the operation
if operation == "trim_start":
metadata = self.trim_start(local_input, params["end_seconds"], local_output)
elif operation == "trim_end":
metadata = self.trim_end(local_input, params["start_seconds"], local_output)
elif operation == "cut":
metadata = self.cut_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation == "mute":
metadata = self.mute_region(
local_input, params["start_seconds"], params["end_seconds"], local_output
)
elif operation in ("join_start", "join_end"):
# Download the upload file
upload_gcs_path = params["upload_gcs_path"]
local_other = os.path.join(temp_dir, "other.flac")
self.storage.download_file(upload_gcs_path, local_other)
position = "start" if operation == "join_start" else "end"
metadata = self.join_audio(local_input, local_other, position, local_output)
else:
raise ValueError(f"Unknown operation: {operation}")
# Upload edited audio
self.storage.upload_file(local_output, output_gcs_path)
logger.info(f"[{job_id}] Applied {operation}, uploaded to {output_gcs_path}")
return metadata, output_gcs_path
def _run_ffmpeg(self, args: list[str]) -> None:
"""Run an ffmpeg command with standard options."""
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"] + args
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if result.returncode != 0:
raise RuntimeError(f"ffmpeg failed (exit {result.returncode}): {result.stderr}")
+10 -15
View File
@@ -204,9 +204,7 @@ class AudioSearchService:
provider=item.get("provider", "Unknown"),
duration=item.get("duration_seconds"),
quality=item.get("quality", ""),
# source_id: use explicit source_id field first (YouTube video ID, Spotify track ID),
# fall back to info_hash for torrent sources (RED/OPS)
source_id=item.get("source_id") or item.get("info_hash"),
source_id=item.get("info_hash"),
index=item.get("index", 0),
seeders=item.get("seeders"),
target_file=item.get("target_file"),
@@ -612,13 +610,13 @@ class AudioSearchService:
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
"""
Async version of search for use in async routes.
Runs the sync search in a thread executor to avoid blocking
the event loop.
Note: Currently wraps sync search in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.search, artist, title)
async def download_async(
self,
result_index: int,
@@ -630,10 +628,10 @@ class AudioSearchService:
"""
Async version of download for use in async routes.
Runs the sync download in a thread executor to avoid blocking
the event loop.
Note: Currently wraps sync download in executor. Future optimization
could make this fully async.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
@@ -651,11 +649,8 @@ class AudioSearchService:
) -> AudioDownloadResult:
"""
Async version of download_by_id for use in async routes.
Runs the sync download_by_id in a thread executor to avoid blocking
the event loop.
"""
loop = asyncio.get_running_loop()
loop = asyncio.get_event_loop()
return await loop.run_in_executor(
None,
lambda: self.download_by_id(
@@ -1,181 +0,0 @@
"""
Audio transcoding service for review UI optimization.
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
Original FLAC files are never modified — transcoded versions are cached
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
Eager transcoding: called from screens_worker before AWAITING_REVIEW
so all files are cached before the user opens the review UI.
Falls back to original FLAC signed URLs if transcoded version is missing.
"""
import asyncio
import hashlib
import logging
import os
import subprocess
import tempfile
from pathlib import Path
from typing import Optional
from backend.services.storage_service import StorageService
logger = logging.getLogger(__name__)
class AudioTranscodingService:
"""
Transcodes audio files to OGG Opus and caches them in GCS.
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
"""
def __init__(self, storage_service: Optional[StorageService] = None):
self.storage = storage_service or StorageService()
def _get_cache_path(self, source_gcs_path: str) -> str:
"""
Derive the cache path for a transcoded file.
Examples:
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
"""
parts = Path(source_gcs_path).parts
# Find "jobs/{job_id}" prefix
try:
jobs_idx = list(parts).index("jobs")
job_id = parts[jobs_idx + 1]
except (ValueError, IndexError):
# Fallback: use a hash of the full path to avoid collisions
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
filename = Path(source_gcs_path).stem + ".ogg"
return f"jobs/{job_id}/review-audio/{filename}"
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
"""
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
Returns the cache GCS path on success.
Raises on failure.
"""
with tempfile.TemporaryDirectory() as temp_dir:
local_input = os.path.join(temp_dir, "input.flac")
local_output = os.path.join(temp_dir, "output.ogg")
# Download source
self.storage.download_file(source_gcs_path, local_input)
# Transcode: FLAC -> OGG Opus 128kbps
cmd = [
"ffmpeg", "-hide_banner", "-loglevel", "error",
"-y",
"-i", local_input,
"-c:a", "libopus",
"-b:a", "128k",
"-vn",
local_output,
]
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
if result.returncode != 0:
raise RuntimeError(
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
)
# Upload transcoded file
self.storage.upload_file(local_output, cache_path)
return cache_path
def transcode_if_needed(self, source_gcs_path: str) -> str:
"""
Return the GCS path of the transcoded OGG file, transcoding if not cached.
Returns the cache path (always an OGG file in review-audio/).
Raises if transcoding fails.
"""
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.debug(f"Cache hit: {cache_path}")
return cache_path
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
return self._transcode_and_upload(source_gcs_path, cache_path)
def get_review_audio_url(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""
Get a signed URL for the transcoded audio, falling back to original FLAC.
"""
try:
cache_path = self.transcode_if_needed(source_gcs_path)
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception as e:
# Source may be deleted but cache from earlier transcode persists
try:
cache_path = self._get_cache_path(source_gcs_path)
if self.storage.file_exists(cache_path):
logger.info(f"Source gone but cache exists, serving: {cache_path}")
return self.storage.generate_signed_url(cache_path, expiration_minutes)
except Exception:
pass # Fall through to FLAC fallback below
logger.warning(
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
)
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
async def get_review_audio_url_async(
self, source_gcs_path: str, expiration_minutes: int = 120
) -> str:
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
return await asyncio.to_thread(
self.get_review_audio_url, source_gcs_path, expiration_minutes
)
def prepare_review_audio_for_job(self, job) -> list[str]:
"""
Transcode all review audio files for a job (eager transcoding).
Called from screens_worker before AWAITING_REVIEW transition
and from the admin backfill endpoint.
Returns list of cache paths that were transcoded.
"""
job_id = job.job_id
transcoded = []
# Collect all audio paths to transcode
paths_to_transcode = []
# Main input audio
if job.input_media_gcs_path:
paths_to_transcode.append(("input", job.input_media_gcs_path))
# Stems
stems = job.file_urls.get("stems", {})
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
path = stems.get(stem_key)
if path:
paths_to_transcode.append((stem_key, path))
for label, source_path in paths_to_transcode:
try:
cache_path = self.transcode_if_needed(source_path)
transcoded.append(cache_path)
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
except Exception as e:
# Non-fatal: review UI will fall back to FLAC
logger.warning(
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
)
logger.info(
f"[{job_id}] Review audio preparation complete: "
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
)
return transcoded
+38 -92
View File
@@ -3,7 +3,6 @@ Authentication service for karaoke backend.
Provides token-based authentication with support for:
- Admin tokens (hardcoded in environment)
- Google OIDC tokens (from Cloud Scheduler / Cloud Tasks service accounts)
- Admin by email domain (@nomadkaraoke.com)
- User tokens (stored in Firestore auth_tokens collection)
- Session tokens (stored in Firestore sessions collection, from magic link auth)
@@ -20,9 +19,6 @@ from enum import Enum
from typing import Optional, Tuple, Dict, Any, List
from datetime import datetime
from google.auth.transport import requests as google_auth_requests
from google.oauth2 import id_token as google_id_token
from backend.services.firestore_service import FirestoreService
from backend.config import get_settings
from backend.models.user import UserRole
@@ -53,7 +49,6 @@ class AuthResult:
user_email: Optional[str] = None # Email if authenticated via session/API key
is_admin: bool = False # True if admin token or admin email domain
api_key_id: Optional[str] = None # API key ID if authenticated via API key
tenant_id: Optional[str] = None # Tenant scope from token or session
def __iter__(self):
"""Allow unpacking as tuple for backward compatibility."""
@@ -76,7 +71,6 @@ class AuthService:
self.firestore = FirestoreService()
self.settings = get_settings()
self._init_admin_tokens()
self._scheduler_service_account = self.settings.scheduler_service_account
def _init_admin_tokens(self):
"""Load admin tokens from environment variables."""
@@ -88,56 +82,6 @@ class AuthService:
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
else:
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
"""
Validate a Google OIDC token (from Cloud Scheduler or Cloud Tasks).
Returns AuthResult if token is a valid Google OIDC JWT from an allowed
service account, None if the token is not a JWT or validation fails.
"""
# Quick check: JWTs have 3 dot-separated parts
if token.count('.') != 2:
return None
try:
# Verify the token signature and claims with Google
# audience=None skips audience check — Cloud Scheduler sets audience
# to the endpoint URL, which varies per endpoint
claims = google_id_token.verify_oauth2_token(
token,
google_auth_requests.Request(),
audience=None,
)
token_email = claims.get("email", "")
# Only accept tokens from the configured service account
if token_email != self._scheduler_service_account:
logger.warning(
f"OIDC token from unexpected service account: {token_email} "
f"(expected {self._scheduler_service_account})"
)
return None
logger.info(f"OIDC token validated for service account: {token_email}")
return AuthResult(
is_valid=True,
user_type=UserType.ADMIN,
remaining_uses=-1,
message="OIDC service account access granted",
user_email=token_email,
is_admin=True,
)
except ValueError as e:
# verify_oauth2_token raises ValueError for invalid/expired tokens
logger.debug(f"OIDC token validation failed: {e}")
return None
except Exception as e:
# Network errors fetching Google certs, etc.
logger.warning(f"OIDC token verification error: {e}")
return None
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
"""
@@ -268,12 +212,6 @@ class AuthService:
is_admin=True
)
# Check for Google OIDC tokens (Cloud Scheduler / Cloud Tasks)
# Done before Firestore lookup to avoid wasting a read on JWT strings
oidc_result = self._validate_oidc_token(token)
if oidc_result is not None:
return oidc_result
# Check stored tokens in Firestore (auth_tokens collection)
token_data = self.firestore.get_token(token)
@@ -346,7 +284,6 @@ class AuthService:
# All auth_tokens must have an associated user_email for job ownership
token_user_email = token_data.get("user_email")
api_key_id = token_data.get("api_key_id")
token_tenant_id = token_data.get("tenant_id")
# Require user_email on all auth_tokens (no anonymous token auth)
if not token_user_email:
@@ -370,8 +307,7 @@ class AuthService:
message="Unlimited access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# LIMITED tokens: check usage count
@@ -384,8 +320,7 @@ class AuthService:
message="Limited token with unlimited uses",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
current_uses = token_data.get("usage_count", 0)
@@ -398,8 +333,7 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -409,8 +343,7 @@ class AuthService:
message=f"Limited token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# STRIPE tokens: check expiration and usage
@@ -424,8 +357,7 @@ class AuthService:
remaining_uses=0,
message="Token has expired",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
if max_uses > 0:
@@ -439,8 +371,7 @@ class AuthService:
remaining_uses=0,
message="Token usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -450,8 +381,7 @@ class AuthService:
message=f"Stripe token: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -461,8 +391,7 @@ class AuthService:
message="Stripe access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
# API_KEY tokens
@@ -477,8 +406,7 @@ class AuthService:
remaining_uses=0,
message="API key usage limit exceeded",
user_email=token_user_email,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
is_valid=True,
@@ -487,8 +415,7 @@ class AuthService:
message=f"API key valid: {remaining} uses remaining",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -498,8 +425,7 @@ class AuthService:
message="API key access granted",
user_email=token_user_email,
is_admin=token_is_admin,
api_key_id=api_key_id,
tenant_id=token_tenant_id
api_key_id=api_key_id
)
return AuthResult(
@@ -511,11 +437,10 @@ class AuthService:
def increment_token_usage(self, token: str, job_id: str) -> bool:
"""
Increment usage count for an auth_token and track the job.
Increment usage count for a token and track the job.
Note: Credit deduction for session-based users is handled in
JobManager.create_job(), not here. This method only tracks usage
for legacy auth_tokens stored in Firestore.
For session tokens (magic link auth), this deducts a credit from the user.
For auth_tokens, this increments the usage count.
Args:
token: The access token
@@ -540,11 +465,32 @@ class AuthService:
if token in self.admin_tokens:
return True
# Only increment usage for auth_tokens in Firestore
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
# Check if this is a session token (STRIPE type from validate_token means it's a session)
# Session tokens are not in auth_tokens collection
token_data = self.firestore.get_token(token)
if not token_data:
return True
# This is a session token - deduct credit from user
from backend.services.user_service import get_user_service
user_service = get_user_service()
# Get the user email from the session
is_valid, user, _ = user_service.validate_session(token)
if not is_valid or not user:
logger.error(f"Session token validation failed during usage increment")
return False
# Deduct one credit
success, new_balance, deduct_message = user_service.deduct_credit(
user.email, job_id, reason="job_creation"
)
if success:
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
return True
else:
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
return False
# Regular auth_token - increment usage in Firestore
try:
-225
View File
@@ -1,225 +0,0 @@
"""
Brand code allocation service using Firestore atomic counters.
Replaces the previous Dropbox folder-scanning approach which was susceptible to
TOCTOU race conditions when concurrent jobs read the same folder state.
Uses Firestore transactions to atomically allocate unique brand codes.
Supports recycling numbers when E2E cleanup deletes test brand codes.
"""
import logging
import re
from datetime import datetime, timezone
from typing import Optional
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
# Singleton instance
_brand_code_service: Optional["BrandCodeService"] = None
class BrandCodeService:
"""Atomic brand code allocation using Firestore counters."""
def __init__(self, db: Optional[firestore.Client] = None):
if db is None:
self.db = firestore.Client(project=settings.google_cloud_project)
else:
self.db = db
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
"""
Atomically allocate the next brand code for a given prefix.
Uses a Firestore transaction to ensure no two concurrent callers
receive the same number. If recycled numbers are available (from
E2E cleanup), the smallest recycled number is reused first.
On first use for a prefix, initializes the counter by scanning
Dropbox to find the current max number.
Args:
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
dropbox_path: Dropbox path for one-time initialization scan
Returns:
Brand code string (e.g., "NOMAD-1264")
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def allocate_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# First use — initialize from Dropbox scan
next_number = self._get_initial_next_number(prefix, dropbox_path)
allocated = next_number
transaction.set(doc_ref, {
"prefix": prefix,
"next_number": next_number + 1,
"recycled": [],
"initialized_at": datetime.now(timezone.utc),
"last_updated": datetime.now(timezone.utc),
})
return allocated
data = doc.to_dict()
recycled = data.get("recycled", [])
if recycled:
# Reuse smallest recycled number
recycled.sort()
allocated = recycled.pop(0)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
else:
# Allocate next sequential number
allocated = data["next_number"]
transaction.update(doc_ref, {
"next_number": allocated + 1,
"last_updated": datetime.now(timezone.utc),
})
return allocated
transaction = self.db.transaction()
number = allocate_in_transaction(transaction)
brand_code = f"{prefix}-{number:04d}"
logger.info(f"Allocated brand code: {brand_code}")
return brand_code
def recycle_brand_code(self, prefix: str, number: int) -> None:
"""
Return a brand code number to the recycled pool.
Called when E2E cleanup deletes a brand code's distribution.
The number will be reused on the next allocation for this prefix.
Args:
prefix: Brand prefix (e.g., "NOMAD")
number: The numeric portion to recycle (e.g., 1234)
"""
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
@firestore.transactional
def recycle_in_transaction(transaction):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
# Counter doesn't exist yet — nothing to recycle into.
# The number will naturally be below next_number when
# the counter is initialized, so it would create a gap.
# Just log and skip.
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
)
return
data = doc.to_dict()
next_number = data.get("next_number", 0)
if number <= 0 or number >= next_number:
logger.warning(
f"Cannot recycle {prefix}-{number:04d}: "
f"invalid (next_number={next_number})"
)
return
recycled = data.get("recycled", [])
if number not in recycled:
recycled.append(number)
transaction.update(doc_ref, {
"recycled": recycled,
"last_updated": datetime.now(timezone.utc),
})
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
else:
logger.warning(f"Number {number} already in recycled pool for {prefix}")
transaction = self.db.transaction()
recycle_in_transaction(transaction)
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
"""
Scan Dropbox once to determine the starting next_number.
This is only called when the counter document doesn't exist yet
(one-time migration from the old Dropbox-scanning approach).
Args:
prefix: Brand prefix to scan for
dropbox_path: Dropbox path containing organized folders
Returns:
The next number to allocate (max existing + 1, minimum 1)
"""
try:
from backend.services.dropbox_service import get_dropbox_service
dropbox = get_dropbox_service()
if not dropbox.is_configured:
logger.warning(
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
)
return 1
folders = dropbox.list_folders(dropbox_path)
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
if existing_nums:
next_num = max(existing_nums) + 1
logger.info(
f"Initialized {prefix} counter from Dropbox: "
f"max existing={max(existing_nums)}, next_number={next_num}"
)
return next_num
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
return 1
except Exception as e:
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
raise
@staticmethod
def parse_brand_code(brand_code: str) -> tuple[str, int]:
"""
Parse a brand code string into prefix and number.
Args:
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
Returns:
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
Raises:
ValueError: If brand_code doesn't match expected format
"""
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
if not match:
raise ValueError(f"Invalid brand code format: {brand_code}")
return match.group(1), int(match.group(2))
def get_brand_code_service() -> BrandCodeService:
"""Get or create the singleton BrandCodeService instance."""
global _brand_code_service
if _brand_code_service is None:
_brand_code_service = BrandCodeService()
return _brand_code_service
-105
View File
@@ -1,105 +0,0 @@
"""
Proxy service for karaoke-decide catalog API.
Forwards artist and track search requests to the karaoke-decide backend,
providing autocomplete data from MusicBrainz + Spotify catalogs.
"""
import logging
import time
from typing import Any
import httpx
from backend.config import settings
logger = logging.getLogger(__name__)
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
_cache: dict[str, tuple[float, Any]] = {}
CACHE_TTL_SECONDS = 300 # 5 minutes
KARAOKE_DECIDE_BASE_URL = "https://decide.nomadkaraoke.com"
REQUEST_TIMEOUT = 10 # seconds
def _get_base_url() -> str:
"""Get the karaoke-decide API base URL from config or default."""
return getattr(settings, "karaoke_decide_api_url", None) or KARAOKE_DECIDE_BASE_URL
def _cache_get(key: str) -> Any | None:
"""Get a value from cache if it exists and hasn't expired."""
if key in _cache:
expiry, data = _cache[key]
if time.monotonic() < expiry:
return data
del _cache[key]
return None
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
"""Store a value in cache with TTL."""
_cache[key] = (time.monotonic() + ttl, data)
async def search_artists(query: str, limit: int = 10) -> list[dict]:
"""
Search for artists via karaoke-decide catalog API.
Returns list of artist dicts with canonical MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"artists:{query.lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/artists"
params = {"q": query, "limit": limit}
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"artists": [...], "total": N}
results = data.get("artists", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search artists from karaoke-decide: {e}")
return []
async def search_tracks(query: str, artist: str | None = None, limit: int = 10) -> list[dict]:
"""
Search for tracks via karaoke-decide catalog API.
Returns list of track dicts with canonical Spotify/MusicBrainz names.
Falls back to empty list on any error.
"""
cache_key = f"tracks:{query.lower()}:{(artist or '').lower()}:{limit}"
cached = _cache_get(cache_key)
if cached is not None:
return cached
base_url = _get_base_url()
url = f"{base_url}/api/catalog/tracks"
params: dict[str, Any] = {"q": query, "limit": limit}
if artist:
params["artist"] = artist
try:
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
resp = await client.get(url, params=params)
resp.raise_for_status()
data = resp.json()
# karaoke-decide wraps results: {"tracks": [...], "total": N}
results = data.get("tracks", data) if isinstance(data, dict) else data
_cache_set(cache_key, results)
return results
except Exception as e:
logger.warning(f"Failed to search tracks from karaoke-decide: {e}")
return []
@@ -1,441 +0,0 @@
"""
AI-powered credit evaluation service.
Evaluates whether to grant free credits (welcome or feedback) by collecting
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
the AI call fails for any reason, credits are granted.
"""
import json
import logging
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Optional
from backend.config import get_settings
logger = logging.getLogger(__name__)
CREDIT_EVALUATIONS_COLLECTION = "credit_evaluations"
@dataclass
class CreditEvaluation:
"""Result of an AI credit evaluation."""
decision: str # "grant" or "deny"
reasoning: str
confidence: float
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
SYSTEM_PROMPT = """You are an anti-abuse evaluator for Nomad Karaoke, a web service that creates professional karaoke videos. Each job costs the service real money in API credits and cloud compute.
New users get 1 free welcome credit to try the service. Users who complete 2 jobs and submit feedback get 1 more credit. Some users abuse this by creating multiple accounts to get unlimited free karaoke generation.
Your job: Given the signals below, decide whether to GRANT or DENY free credits to this user.
## Decision Guidelines
**DENY if any of these are true:**
- The user's device fingerprint matches other accounts (strong signal of multi-accounting)
- The user's signup IP matches other accounts AND those accounts also have suspicious patterns (e.g., no spend, same user agent)
- There's a clear pattern of account creation from the same device/network to farm free credits
**GRANT if:**
- This appears to be a genuine new user with no suspicious correlations
- The IP match is explainable (e.g., different fingerprints suggest different actual users on a shared network like a school or office)
- There's insufficient evidence to confidently flag abuse
**When in doubt, GRANT.** False positives (blocking real users) are worse than false negatives (letting an occasional abuser through). We'd rather lose a few dollars than alienate a real customer.
## Response Format
Respond with ONLY a JSON object, no other text:
{"decision": "grant" or "deny", "reasoning": "Brief explanation", "confidence": 0.0 to 1.0}
"""
def _build_user_prompt(
email: str,
grant_type: str,
user_data: dict,
fingerprint_matches: list,
ip_matches: list,
ip_geo: Optional[dict],
recent_signups_ip: int,
recent_signups_fp: int,
user_agent: Optional[str],
feedback_content: Optional[dict] = None,
) -> str:
"""Build the user prompt with all abuse signals."""
lines = [
f"## Credit Grant Request",
f"- **Email:** {email}",
f"- **Grant type:** {grant_type} credits",
f"- **User's device fingerprint:** {user_data.get('device_fingerprint', 'unknown')}",
f"- **User's signup IP:** {user_data.get('signup_ip', 'unknown')}",
f"- **User agent:** {user_agent or 'unknown'}",
f"- **Account created:** {user_data.get('created_at', 'unknown')}",
f"- **Jobs created:** {user_data.get('total_jobs_created', 0)}",
f"- **Jobs completed:** {user_data.get('total_jobs_completed', 0)}",
f"- **Total spent (real money):** ${user_data.get('total_spent', 0) / 100:.2f}",
f"- **Current credits:** {user_data.get('credits', 0)}",
"",
]
# IP geolocation
if ip_geo and ip_geo.get("status") == "success":
lines.extend([
f"## IP Geolocation",
f"- **Country:** {ip_geo.get('country', 'unknown')} ({ip_geo.get('country_code', '')})",
f"- **City/Region:** {ip_geo.get('city', '')}, {ip_geo.get('region', '')}",
f"- **ISP:** {ip_geo.get('isp', 'unknown')}",
f"- **Organization:** {ip_geo.get('org', 'unknown')}",
f"- **ASN:** {ip_geo.get('as_number', '')} {ip_geo.get('as_name', '')}",
"",
])
# Recent signup patterns
lines.extend([
f"## Recent Signup Patterns (last 24 hours)",
f"- **New accounts from same IP:** {recent_signups_ip}",
f"- **New accounts from same fingerprint:** {recent_signups_fp}",
"",
])
# Fingerprint correlations
if fingerprint_matches:
lines.append("## Other Accounts with Same Device Fingerprint (RED FLAG)")
for match in fingerprint_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts with Same Device Fingerprint", "None found.", ""])
# IP correlations
if ip_matches:
lines.append("## Other Accounts from Same Signup IP")
for match in ip_matches:
lines.append(
f"- {match.get('email', '?')} | "
f"Fingerprint: {match.get('device_fingerprint', 'unknown')} | "
f"Jobs: {match.get('total_jobs_created', 0)} | "
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
f"Created: {match.get('created_at', '?')}"
)
lines.append("")
else:
lines.extend(["## Other Accounts from Same Signup IP", "None found.", ""])
# Feedback content (if applicable)
if feedback_content:
lines.extend([
"## Feedback Submitted",
f"- **Overall rating:** {feedback_content.get('overall_rating', '?')}/5",
f"- **What went well:** {feedback_content.get('what_went_well', 'N/A')[:200]}",
f"- **What could improve:** {feedback_content.get('what_could_improve', 'N/A')[:200]}",
f"- **Additional comments:** {feedback_content.get('additional_comments', 'N/A')[:200]}",
"",
])
lines.append("Based on these signals, should we GRANT or DENY free credits to this user?")
return "\n".join(lines)
def _parse_gemini_response(response_text: str) -> CreditEvaluation:
"""Parse Gemini's JSON response into a CreditEvaluation."""
try:
# Strip markdown code fences if present
text = response_text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
if text.endswith("```"):
text = text.rsplit("```", 1)[0]
text = text.strip()
if text.startswith("json"):
text = text[4:].strip()
data = json.loads(text)
decision = data.get("decision", "").lower()
if decision not in ("grant", "deny"):
decision = "pending_review" # fail-closed: unknown decision → manual review
return CreditEvaluation(
decision=decision,
reasoning=str(data.get("reasoning", "No reasoning provided")),
confidence=float(data.get("confidence", 0.5)),
)
except (json.JSONDecodeError, ValueError, TypeError) as e:
logger.warning(f"Failed to parse Gemini response: {e}. Response: {response_text[:200]}")
return CreditEvaluation(
decision="pending_review",
reasoning="Failed to parse AI response — pending manual review",
confidence=0.0,
error=f"Parse error: {e}",
)
class CreditEvaluationService:
"""Evaluates credit grant requests using Gemini AI."""
def __init__(self):
self.settings = get_settings()
def _collect_signals(self, email: str) -> dict:
"""Collect all abuse signals for evaluation."""
from backend.services.user_service import get_user_service
from backend.services.ip_geolocation_service import get_ip_geolocation_service
user_service = get_user_service()
user = user_service.get_user(email)
if not user:
return {
"user_data": {},
"fingerprint_matches": [],
"ip_matches": [],
"ip_geo": None,
"recent_signups_ip": 0,
"recent_signups_fp": 0,
"user_agent": None,
}
user_data = {
"email": user.email,
"device_fingerprint": user.device_fingerprint,
"signup_ip": user.signup_ip,
"credits": user.credits,
"total_jobs_created": user.total_jobs_created,
"total_jobs_completed": user.total_jobs_completed,
"total_spent": user.total_spent,
"created_at": str(user.created_at),
}
# Find other accounts with same fingerprint
fingerprint_matches = []
if user.device_fingerprint:
fp_users = user_service.find_users_by_fingerprint(user.device_fingerprint)
fingerprint_matches = [
{
"email": u.email,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in fp_users if u.email != email
]
# Find other accounts from same IP
ip_matches = []
if user.signup_ip:
ip_users = user_service.find_users_by_signup_ip(user.signup_ip)
ip_matches = [
{
"email": u.email,
"device_fingerprint": u.device_fingerprint,
"total_jobs_created": u.total_jobs_created,
"total_spent": u.total_spent,
"created_at": str(u.created_at),
}
for u in ip_users if u.email != email
]
# IP geolocation
ip_geo = None
if user.signup_ip:
try:
geo_service = get_ip_geolocation_service()
ip_geo = geo_service.lookup_ip(user.signup_ip)
except Exception:
pass
# Recent signup counts
recent_signups_ip = 0
recent_signups_fp = 0
if user.signup_ip:
recent_signups_ip = user_service.count_recent_signups_from_ip(user.signup_ip)
if user.device_fingerprint:
recent_signups_fp = user_service.count_recent_signups_from_fingerprint(user.device_fingerprint)
# User agent from latest session
user_agent = None
try:
from google.cloud.firestore_v1 import FieldFilter
from google.cloud import firestore as fs
sessions = (
user_service.db.collection("sessions")
.where(filter=FieldFilter("user_email", "==", email))
.order_by("created_at", direction=fs.Query.DESCENDING)
.limit(1)
.stream()
)
for session_doc in sessions:
user_agent = session_doc.to_dict().get("user_agent")
except Exception:
pass
return {
"user_data": user_data,
"fingerprint_matches": fingerprint_matches,
"ip_matches": ip_matches,
"ip_geo": ip_geo,
"recent_signups_ip": recent_signups_ip,
"recent_signups_fp": recent_signups_fp,
"user_agent": user_agent,
}
def _call_gemini(self, prompt: str) -> str:
"""Call Gemini and return the response text."""
from google import genai
client = genai.Client(
vertexai=True,
project=self.settings.google_cloud_project,
location="global",
)
response = client.models.generate_content(
model=self.settings.credit_eval_model,
contents=prompt,
config=genai.types.GenerateContentConfig(
system_instruction=SYSTEM_PROMPT,
),
)
return response.text
def _log_evaluation(
self,
email: str,
grant_type: str,
evaluation: CreditEvaluation,
signals: dict,
):
"""Log the evaluation decision to Firestore for audit trail."""
try:
from google.cloud import firestore as fs
db = fs.Client(project=self.settings.google_cloud_project)
doc_id = str(uuid.uuid4())
db.collection(CREDIT_EVALUATIONS_COLLECTION).document(doc_id).set({
"id": doc_id,
"email": email,
"grant_type": grant_type,
"decision": evaluation.decision,
"reasoning": evaluation.reasoning,
"confidence": evaluation.confidence,
"error": evaluation.error,
"model": self.settings.credit_eval_model,
"signals_snapshot": {
"fingerprint_match_count": len(signals.get("fingerprint_matches", [])),
"ip_match_count": len(signals.get("ip_matches", [])),
"recent_signups_ip": signals.get("recent_signups_ip", 0),
"recent_signups_fp": signals.get("recent_signups_fp", 0),
"user_ip": signals.get("user_data", {}).get("signup_ip"),
"user_fingerprint": signals.get("user_data", {}).get("device_fingerprint"),
},
"evaluated_at": datetime.now(timezone.utc).isoformat(),
})
except Exception:
logger.exception(f"Failed to log credit evaluation for {email}")
def evaluate(
self,
email: str,
grant_type: str,
feedback_content: Optional[dict] = None,
) -> CreditEvaluation:
"""
Evaluate whether to grant free credits.
Args:
email: User's email
grant_type: "welcome" or "feedback"
feedback_content: Feedback form data (for feedback grants only)
Returns:
CreditEvaluation with decision, reasoning, confidence.
Fail-closed: returns "pending_review" on any error (admin notified).
"""
if not self.settings.credit_eval_enabled:
return CreditEvaluation(
decision="grant",
reasoning="Credit evaluation disabled — auto-granting",
confidence=1.0,
)
try:
# Collect signals
signals = self._collect_signals(email)
# Quick-grant: if no correlations found at all, skip AI call
if (
not signals["fingerprint_matches"]
and not signals["ip_matches"]
and signals["recent_signups_ip"] <= 1
and signals["recent_signups_fp"] <= 1
):
evaluation = CreditEvaluation(
decision="grant",
reasoning="No suspicious correlations found — clean user",
confidence=1.0,
signals=signals,
)
self._log_evaluation(email, grant_type, evaluation, signals)
return evaluation
# Build prompt and call Gemini
prompt = _build_user_prompt(
email=email,
grant_type=grant_type,
user_data=signals["user_data"],
fingerprint_matches=signals["fingerprint_matches"],
ip_matches=signals["ip_matches"],
ip_geo=signals["ip_geo"],
recent_signups_ip=signals["recent_signups_ip"],
recent_signups_fp=signals["recent_signups_fp"],
user_agent=signals["user_agent"],
feedback_content=feedback_content,
)
response_text = self._call_gemini(prompt)
evaluation = _parse_gemini_response(response_text)
evaluation.signals = signals
self._log_evaluation(email, grant_type, evaluation, signals)
logger.info(
f"Credit evaluation for {email} ({grant_type}): "
f"{evaluation.decision} (confidence={evaluation.confidence:.2f})"
)
return evaluation
except Exception as e:
logger.exception(f"Credit evaluation failed for {email} — pending manual review (fail-closed)")
# Include whatever signals were collected before the failure
collected_signals = locals().get("signals")
evaluation = CreditEvaluation(
decision="pending_review",
reasoning="Evaluation failed — pending manual review",
confidence=0.0,
error=str(e),
signals=collected_signals,
)
try:
self._log_evaluation(email, grant_type, evaluation, collected_signals or {})
except Exception:
pass
return evaluation
_service_instance: Optional[CreditEvaluationService] = None
def get_credit_evaluation_service() -> CreditEvaluationService:
"""Get or create the singleton credit evaluation service."""
global _service_instance
if _service_instance is None:
_service_instance = CreditEvaluationService()
return _service_instance
@@ -1,129 +0,0 @@
"""
Disposable Domain Sync Service.
Fetches and syncs disposable email domain blocklists from external sources.
"""
import logging
from datetime import datetime, timezone
from typing import Set
import httpx
from backend.services.email_validation_service import (
BLOCKLISTS_COLLECTION,
BLOCKLIST_CONFIG_DOC,
DEFAULT_DISPOSABLE_DOMAINS,
)
logger = logging.getLogger(__name__)
EXTERNAL_BLOCKLIST_URL = (
"https://raw.githubusercontent.com/disposable-email-domains/"
"disposable-email-domains/refs/heads/main/disposable_email_blocklist.conf"
)
FETCH_TIMEOUT_SECONDS = 30
MAX_RESPONSE_BYTES = 2 * 1024 * 1024 # 2 MB
MAX_DOMAIN_COUNT = 50_000
def parse_blocklist_text(text: str) -> Set[str]:
"""Parse newline-delimited blocklist text into a set of lowercase domains."""
domains = set()
for line in text.splitlines():
domain = line.strip().lower()
if domain:
domains.add(domain)
return domains
async def fetch_external_blocklist() -> Set[str]:
"""Fetch the external disposable domain blocklist from GitHub.
Checks Content-Length header and response body size against safety limits.
"""
async with httpx.AsyncClient(timeout=FETCH_TIMEOUT_SECONDS) as client:
response = await client.get(EXTERNAL_BLOCKLIST_URL)
response.raise_for_status()
content_length = response.headers.get("content-length")
if content_length and int(content_length) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response size {content_length} exceeds maximum {MAX_RESPONSE_BYTES}"
)
text = response.text
if len(text.encode()) > MAX_RESPONSE_BYTES:
raise ValueError(
f"Response body size exceeds maximum {MAX_RESPONSE_BYTES}"
)
domains = parse_blocklist_text(text)
if len(domains) > MAX_DOMAIN_COUNT:
raise ValueError(
f"Domain count {len(domains)} exceeds maximum {MAX_DOMAIN_COUNT}"
)
logger.info(f"Fetched {len(domains)} domains from external blocklist")
return domains
def sync_disposable_domains(db, external_domains: Set[str]) -> dict:
"""
Sync external domains into Firestore.
Handles first-run migration from old disposable_domains field,
and cleans up redundant manual domains on subsequent runs.
Note: This is a synchronous function (Firestore SDK is sync).
Call from async code with `await asyncio.to_thread(sync_disposable_domains, db, domains)`.
Returns a summary dict with counts.
"""
doc_ref = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
data = doc.to_dict() if doc.exists else {}
result = {"external_count": len(external_domains), "added": 0, "removed": 0, "migrated_to_manual": []}
is_migration = "external_domains" not in data
if is_migration:
old_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
manual_only = old_domains - external_domains
result["migrated_to_manual"] = sorted(manual_only)
new_data = {
**data, # preserve all existing fields
"external_domains": sorted(external_domains),
"manual_domains": sorted(manual_only),
"allowlisted_domains": data.get("allowlisted_domains", []),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
new_data.pop("disposable_domains", None) # remove legacy field
doc_ref.set(new_data)
else:
old_external = set(data.get("external_domains", []))
manual_domains = set(data.get("manual_domains", []))
result["added"] = len(external_domains - old_external)
result["removed"] = len(old_external - external_domains)
cleaned_manual = manual_domains - external_domains
new_data = {
**data,
"external_domains": sorted(external_domains),
"manual_domains": sorted(cleaned_manual),
"last_sync_at": datetime.now(timezone.utc),
"last_sync_count": len(external_domains),
}
doc_ref.set(new_data)
logger.info(
f"Sync complete: {result['external_count']} external domains, "
f"{result['added']} added, {result['removed']} removed"
)
return result
+8 -24
View File
@@ -109,46 +109,30 @@ class DropboxService:
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
"""
Calculate next brand code from existing folders, filling gaps in the sequence.
NOTE: This method is only used for one-time initialization of the Firestore
counter (see brand_code_service.py). All runtime brand code allocation now
goes through BrandCodeService.allocate_brand_code() for atomic safety.
Calculate next brand code from existing folders.
Scans the folder for existing brand codes like "NOMAD-0001" and
returns the first available code, filling any gaps left by deleted outputs.
returns the next sequential code.
Args:
path: Dropbox path containing organized folders
brand_prefix: Brand prefix (e.g., "NOMAD")
Returns:
Next brand code (e.g., "NOMAD-0003" if 0003 is the first gap)
Next brand code (e.g., "NOMAD-1164")
"""
folders = self.list_folders(path)
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
existing_nums: set[int] = set()
max_num = 0
for folder in folders:
match = pattern.match(folder)
if match:
existing_nums.add(int(match.group(1)))
num = int(match.group(1))
max_num = max(max_num, num)
max_existing = max(existing_nums) if existing_nums else 0
# Only fill gaps for numbers >= 1001 (preserve legacy gaps below 1000)
if max_existing >= 1000:
next_num = 1001
while next_num in existing_nums:
next_num += 1
else:
next_num = max_existing + 1
next_code = f"{brand_prefix}-{next_num:04d}"
if max_existing >= 1000 and next_num <= max_existing:
logger.info(f"Next brand code for {brand_prefix}: {next_code} (filling gap, max existing: {max_existing})")
else:
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_existing})")
next_code = f"{brand_prefix}-{max_num + 1:04d}"
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
return next_code
def upload_file(self, local_path: str, remote_path: str) -> None:
File diff suppressed because it is too large. Load diff
@@ -1,856 +0,0 @@
"""
Email Validation Service.
Provides email validation, normalization, and blocklist checking
to prevent abuse during enrollment and other flows.
Uses a tiered disposable domain detection strategy:
1. Static blocklist (instant, from Firestore)
2. Verified-clean domain cache (skip API calls for 7 days)
3. DeBounce API (free, <50ms)
4. verifymail.io API (paid, only if DeBounce says clean)
Auto-learns: flagged domains are persisted to manual_domains,
clean domains are cached for 7 days in verified_clean_domains.
"""
import hashlib
import logging
import re
from datetime import datetime, timedelta, timezone
from typing import Optional, Tuple, Set
import httpx
from google.cloud import firestore
from backend.config import settings
logger = logging.getLogger(__name__)
# Module-level HTTP client for connection pooling (external API checks)
_http_client: Optional[httpx.Client] = None
_HTTP_TIMEOUT = 2.0 # seconds per external API call
def _get_http_client() -> httpx.Client:
"""Get or create the module-level HTTP client."""
global _http_client
if _http_client is None or _http_client.is_closed:
_http_client = httpx.Client(timeout=_HTTP_TIMEOUT)
return _http_client
def _mask_email(email: str) -> str:
"""Mask an email address for privacy-safe logging."""
if not email or "@" not in email:
return "***"
local, domain = email.split("@", 1)
if len(local) <= 2:
masked_local = "*" * len(local)
else:
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
return f"{masked_local}@{domain}"
# Firestore collection for blocklist data
BLOCKLISTS_COLLECTION = "blocklists"
BLOCKLIST_CONFIG_DOC = "config"
# Legacy default set — kept as empty set for import compatibility.
# The external sync (disposable_domain_sync_service) replaces this with ~4,800
# domains from the community-curated disposable-email-domains repo.
# The migration path in sync_disposable_domains() references this set to ensure
# no domains are lost during the transition.
DEFAULT_DISPOSABLE_DOMAINS: Set[str] = set()
# Gmail-like domains that support alias normalization
GMAIL_LIKE_DOMAINS = {
"gmail.com",
"googlemail.com",
}
# Well-known email providers that are never disposable.
# These skip all external API checks for zero-latency validation.
WELL_KNOWN_PROVIDERS = {
"gmail.com",
"googlemail.com",
"yahoo.com",
"yahoo.co.uk",
"yahoo.co.jp",
"yahoo.fr",
"hotmail.com",
"hotmail.co.uk",
"outlook.com",
"outlook.fr",
"live.com",
"live.co.uk",
"msn.com",
"icloud.com",
"me.com",
"mac.com",
"aol.com",
"protonmail.com",
"proton.me",
"zoho.com",
"yandex.com",
"yandex.ru",
"mail.com",
"gmx.com",
"gmx.de",
"fastmail.com",
"hey.com",
"tutanota.com",
"tuta.io",
}
# How long a verified-clean domain result is trusted before re-checking
VERIFIED_CLEAN_TTL = timedelta(days=7)
# External API URLs
DEBOUNCE_API_URL = "https://disposable.debounce.io/"
VERIFYMAIL_API_URL = "https://verifymail.io/api"
class EmailValidationService:
"""
Service for email validation and abuse prevention.
Features:
- Email normalization (Gmail alias stripping)
- Disposable domain detection
- Blocklist checking (emails, IPs)
- IP-based enrollment rate limiting
"""
_instance: Optional["EmailValidationService"] = None
_blocklist_cache: Optional[dict] = None
_blocklist_cache_time: Optional[datetime] = None
_cache_ttl_seconds: int = 300 # 5 minutes
def __init__(self, db: Optional[firestore.Client] = None):
"""Initialize the email validation service."""
if db is None:
from backend.services.firestore_service import get_firestore_client
db = get_firestore_client()
self.db = db
@classmethod
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
"""Get singleton instance of the service."""
if cls._instance is None:
cls._instance = cls(db)
return cls._instance
def normalize_email(self, email: str) -> str:
"""
Normalize an email address.
For Gmail-like domains:
- Removes dots from local part
- Removes everything after + in local part
- Converts to lowercase
For other domains:
- Only converts to lowercase
Args:
email: Email address to normalize
Returns:
Normalized email address
Examples:
j.o.h.n+spam@gmail.com -> john@gmail.com
John.Doe@example.com -> john.doe@example.com
"""
if not email or "@" not in email:
return email.lower().strip() if email else ""
email = email.lower().strip()
local, domain = email.rsplit("@", 1)
if domain in GMAIL_LIKE_DOMAINS:
# Remove dots from local part
local = local.replace(".", "")
# Remove everything after +
if "+" in local:
local = local.split("+")[0]
return f"{local}@{domain}"
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
"""
Get blocklist configuration from Firestore.
Uses caching to reduce Firestore reads.
Returns:
Dict with disposable_domains, blocked_emails, blocked_ips sets
"""
now = datetime.now(timezone.utc)
# Check cache validity
if (
not force_refresh
and self._blocklist_cache is not None
and self._blocklist_cache_time is not None
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
):
return self._blocklist_cache
# Fetch from Firestore
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
if "external_domains" in data:
# New data model: effective = (external + manual) - allowlisted
external = set(data.get("external_domains", []))
manual = set(data.get("manual_domains", []))
allowlisted = set(data.get("allowlisted_domains", []))
effective_domains = (external | manual) - allowlisted
else:
# Old data model (pre-migration): use disposable_domains + defaults
allowlisted = set()
effective_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
config = {
"disposable_domains": effective_domains,
"allowlisted_domains": allowlisted,
"blocked_emails": set(data.get("blocked_emails", [])),
"blocked_ips": set(data.get("blocked_ips", [])),
"verified_clean_domains": data.get("verified_clean_domains", {}),
}
else:
config = {
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
"allowlisted_domains": set(),
"blocked_emails": set(),
"blocked_ips": set(),
"verified_clean_domains": {},
}
# Update cache
EmailValidationService._blocklist_cache = config
EmailValidationService._blocklist_cache_time = now
return config
def is_disposable_domain(self, email: str) -> bool:
"""
Check if an email uses a disposable domain.
Uses a tiered checking strategy:
0. Well-known providers (gmail, yahoo, etc.) → instant False
1. Static blocklist from Firestore → instant
2. Verified-clean cache (7-day TTL) → skip API calls
3. DeBounce API (free) → fast external check
4. verifymail.io API (paid) → only if DeBounce says clean
Flagged domains are auto-learned to manual_domains.
Clean domains are cached for 7 days.
Args:
email: Email address to check
Returns:
True if the domain is known to be disposable
"""
if not email or "@" not in email:
return False
domain = email.lower().split("@")[-1]
# Tier 0: Well-known providers — skip everything
if domain in WELL_KNOWN_PROVIDERS:
return False
# Tier 1: Static blocklist (instant, always available)
config = self.get_blocklist_config()
if domain in config["disposable_domains"]:
return True
# Tier 1.5: Explicitly allowlisted domains skip external API checks.
# Without this, external APIs can override admin allowlist decisions
# (e.g. inbox.testmail.app is a legitimate testing domain we allow).
if domain in config.get("allowlisted_domains", set()):
return False
# Tier 2: Recently verified clean? Skip API calls
if self._is_domain_verified_clean(domain, config):
return False
# Tier 3: DeBounce API (free, fast)
debounce_result = self._check_debounce(email)
if debounce_result is True:
self._auto_learn_domain(domain)
return True
# Tier 4: verifymail.io (paid, only if DeBounce gave definitive clean)
if debounce_result is False:
verifymail_result = self._check_verifymail(email)
if verifymail_result is True:
self._auto_learn_domain(domain)
return True
# Both APIs say clean (or errored) — cache clean result
if debounce_result is False:
self._mark_domain_clean(domain)
return False
def _check_debounce(self, email: str) -> Optional[bool]:
"""
Check email against DeBounce disposable email API.
Returns:
True if disposable, False if clean, None if error/timeout.
"""
try:
client = _get_http_client()
response = client.get(DEBOUNCE_API_URL, params={"email": email})
response.raise_for_status()
data = response.json()
result = str(data.get("disposable", "")).lower() == "true"
logger.info(
f"DeBounce check for {_mask_email(email)}: "
f"disposable={result}"
)
return result
except httpx.TimeoutException:
logger.warning(f"DeBounce API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"DeBounce API error for {_mask_email(email)}: {e}")
return None
def _check_verifymail(self, email: str) -> Optional[bool]:
"""
Check email against verifymail.io API.
Only called when DeBounce says clean (to minimize paid API usage).
Returns:
True if disposable/blocked, False if clean, None if error/timeout
or API key not configured.
"""
api_key = settings.get_secret("verifymail-api-key")
if not api_key:
logger.debug("verifymail.io API key not configured, skipping")
return None
try:
client = _get_http_client()
response = client.get(
f"{VERIFYMAIL_API_URL}/{email}",
params={"key": api_key},
)
response.raise_for_status()
data = response.json()
# verifymail.io returns both "disposable" and "block" flags
is_disposable = data.get("disposable") is True
is_blocked = data.get("block") is True
result = is_disposable or is_blocked
logger.info(
f"verifymail.io check for {_mask_email(email)}: "
f"disposable={is_disposable}, block={is_blocked}"
)
return result
except httpx.TimeoutException:
logger.warning(f"verifymail.io API timeout for {_mask_email(email)}")
return None
except Exception as e:
logger.warning(f"verifymail.io API error for {_mask_email(email)}: {e}")
return None
def _is_domain_verified_clean(self, domain: str, config: dict) -> bool:
"""Check if a domain was verified clean within the TTL period."""
verified = config.get("verified_clean_domains", {})
entry = verified.get(domain)
if not entry:
return False
checked_at = entry.get("checked_at")
if not checked_at:
return False
# Handle both datetime objects and ISO strings
if isinstance(checked_at, str):
try:
checked_at = datetime.fromisoformat(checked_at)
except (ValueError, TypeError):
return False
# Make timezone-aware if naive
if checked_at.tzinfo is None:
checked_at = checked_at.replace(tzinfo=timezone.utc)
return (datetime.now(timezone.utc) - checked_at) < VERIFIED_CLEAN_TTL
def _auto_learn_domain(self, domain: str) -> None:
"""
Add a domain flagged by external API to manual_domains in Firestore.
This persists the result so future checks don't need the API call.
Fire-and-forget: errors are logged but don't affect the caller.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
manual = set(data.get("manual_domains", []))
else:
manual = set()
if domain not in manual:
manual.add(domain)
doc_ref.set(
{
"manual_domains": list(manual),
"updated_at": datetime.now(timezone.utc).isoformat(),
"updated_by": "auto-learn",
},
merge=True,
)
logger.info(f"Auto-learned disposable domain: {domain}")
# Invalidate cache so the domain is blocked immediately
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to auto-learn domain {domain}: {e}")
def _mark_domain_clean(self, domain: str) -> None:
"""
Record that a domain was verified clean by external APIs.
Stored in verified_clean_domains dict keyed by domain.
Future checks within VERIFIED_CLEAN_TTL will skip API calls.
Note: Uses read-modify-write instead of dot notation because
domain names contain dots which Firestore interprets as nested paths.
"""
try:
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
BLOCKLIST_CONFIG_DOC
)
doc = doc_ref.get()
verified = doc.to_dict().get("verified_clean_domains", {}) if doc.exists else {}
verified[domain] = {
"checked_at": datetime.now(timezone.utc).isoformat(),
}
doc_ref.set({"verified_clean_domains": verified}, merge=True)
logger.debug(f"Marked domain as verified clean: {domain}")
# Invalidate cache so new data is picked up
EmailValidationService._blocklist_cache = None
except Exception as e:
logger.error(f"Failed to mark domain clean {domain}: {e}")
def is_email_blocked(self, email: str) -> bool:
"""
Check if an email is explicitly blocked.
Checks both the raw email and normalized version.
Args:
email: Email address to check
Returns:
True if the email is blocked
"""
if not email:
return False
config = self.get_blocklist_config()
email_lower = email.lower().strip()
email_normalized = self.normalize_email(email)
return (
email_lower in config["blocked_emails"]
or email_normalized in config["blocked_emails"]
)
def is_ip_blocked(self, ip_address: str) -> bool:
"""
Check if an IP address is blocked.
Args:
ip_address: IP address to check
Returns:
True if the IP is blocked
"""
if not ip_address:
return False
config = self.get_blocklist_config()
return ip_address in config["blocked_ips"]
def hash_ip(self, ip_address: str) -> str:
"""
Hash an IP address for privacy-preserving storage.
Args:
ip_address: IP address to hash
Returns:
SHA-256 hash of the IP
"""
return hashlib.sha256(ip_address.encode()).hexdigest()
# -------------------------------------------------------------------------
# Blocklist Management (Admin)
# -------------------------------------------------------------------------
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
"""Add a domain to the disposable domains blocklist."""
domain = domain.lower().strip()
if not domain:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("manual_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["manual_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added disposable domain: {domain} by {admin_email}")
return True
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
"""Remove a domain from the disposable domains blocklist."""
domain = domain.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
manual = set(data.get("manual_domains", []))
external = set(data.get("external_domains", []))
if domain in manual:
result["found"] = True
manual.discard(domain)
data["manual_domains"] = list(manual)
elif domain in external:
result["found"] = True
allowlisted = set(data.get("allowlisted_domains", []))
allowlisted.add(domain)
data["allowlisted_domains"] = list(allowlisted)
else:
return
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
return True
def add_blocked_email(self, email: str, admin_email: str) -> bool:
"""Add an email to the blocked emails list."""
email = email.lower().strip()
if not email:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
emails = set(data.get("blocked_emails", []))
else:
data = {}
emails = set()
emails.add(email)
data["blocked_emails"] = list(emails)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added blocked email: {email} by {admin_email}")
return True
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
"""Remove an email from the blocked emails list."""
email = email.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
emails = set(data.get("blocked_emails", []))
if email not in emails:
return
result["found"] = True
emails.discard(email)
data["blocked_emails"] = list(emails)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed blocked email: {email} by {admin_email}")
return True
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
"""Add an IP address to the blocked IPs list."""
ip_address = ip_address.strip()
if not ip_address:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
ips = set(data.get("blocked_ips", []))
else:
data = {}
ips = set()
ips.add(ip_address)
data["blocked_ips"] = list(ips)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
return True
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
"""Remove an IP address from the blocked IPs list."""
ip_address = ip_address.strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
ips = set(data.get("blocked_ips", []))
if ip_address not in ips:
return
result["found"] = True
ips.discard(ip_address)
data["blocked_ips"] = list(ips)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
return True
def add_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Add a domain to the allowlisted domains list."""
domain = domain.lower().strip()
if not domain:
return False
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
@firestore.transactional
def update_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if doc.exists:
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
else:
data = {}
domains = set()
domains.add(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
update_in_transaction(transaction, doc_ref)
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Added allowlisted domain: {domain} by {admin_email}")
return True
def remove_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
"""Remove a domain from the allowlisted domains list."""
domain = domain.lower().strip()
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
result = {"found": False}
@firestore.transactional
def remove_in_transaction(transaction, doc_ref):
doc = doc_ref.get(transaction=transaction)
if not doc.exists:
return
data = doc.to_dict()
domains = set(data.get("allowlisted_domains", []))
if domain not in domains:
return
result["found"] = True
domains.discard(domain)
data["allowlisted_domains"] = list(domains)
data["updated_at"] = datetime.now(timezone.utc)
data["updated_by"] = admin_email
transaction.set(doc_ref, data, merge=True)
transaction = self.db.transaction()
remove_in_transaction(transaction, doc_ref)
if not result["found"]:
return False
# Invalidate cache
EmailValidationService._blocklist_cache = None
logger.info(f"Removed allowlisted domain: {domain} by {admin_email}")
return True
def get_blocklist_raw_data(self) -> dict:
"""Get raw blocklist data for admin display (not the effective set)."""
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if not doc.exists:
return {
"external_domains": [], "manual_domains": [], "allowlisted_domains": [],
"blocked_emails": [], "blocked_ips": [],
"last_sync_at": None, "last_sync_count": None,
"updated_at": None, "updated_by": None,
}
data = doc.to_dict()
return {
"external_domains": sorted(data.get("external_domains", [])),
"manual_domains": sorted(data.get("manual_domains", [])),
"allowlisted_domains": sorted(data.get("allowlisted_domains", [])),
"blocked_emails": sorted(data.get("blocked_emails", [])),
"blocked_ips": sorted(data.get("blocked_ips", [])),
"last_sync_at": data.get("last_sync_at"),
"last_sync_count": data.get("last_sync_count"),
"updated_at": data.get("updated_at"),
"updated_by": data.get("updated_by"),
}
def get_blocklist_stats(self) -> dict:
"""Get statistics about current blocklists."""
config = self.get_blocklist_config(force_refresh=True)
# Also get raw data for detailed counts
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
doc = doc_ref.get()
if doc.exists:
data = doc.to_dict()
external_count = len(data.get("external_domains", []))
manual_count = len(data.get("manual_domains", []))
allowlisted_count = len(data.get("allowlisted_domains", []))
else:
external_count = 0
manual_count = 0
allowlisted_count = 0
return {
"disposable_domains_count": len(config["disposable_domains"]),
"blocked_emails_count": len(config["blocked_emails"]),
"blocked_ips_count": len(config["blocked_ips"]),
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
"external_domains_count": external_count,
"manual_domains_count": manual_count,
"allowlisted_domains_count": allowlisted_count,
}
def get_email_validation_service(
db: Optional[firestore.Client] = None,
) -> EmailValidationService:
"""Get the singleton EmailValidationService instance."""
return EmailValidationService.get_instance(db)
+8 -25
View File
@@ -40,9 +40,6 @@ class EncodingInput:
# Output directory
output_dir: str = ""
# Instrumental selection (clean, with_backing, or custom)
instrumental_selection: str = "clean"
# Additional options
options: Dict[str, Any] = field(default_factory=dict)
@@ -200,7 +197,6 @@ class LocalEncodingBackend(EncodingBackend):
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
countdown_padding_seconds=input_config.options.get("countdown_padding_seconds"),
)
# Run encoding in thread pool to avoid blocking
@@ -332,15 +328,8 @@ class GCEEncodingBackend(EncodingBackend):
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
"artist": input_config.artist,
"title": input_config.title,
"instrumental_selection": input_config.instrumental_selection,
}
# Add countdown padding if present (for audio sync with countdown-padded vocals)
countdown_padding = input_config.options.get("countdown_padding_seconds")
if countdown_padding:
encoding_config["countdown_padding_seconds"] = countdown_padding
self.logger.info(f"GCE encoding with countdown_padding_seconds={countdown_padding}")
# Submit and wait for completion
result = await service.encode_videos(
job_id=job_id,
@@ -362,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
result = {}
raw_output_files = result.get("output_files", {})
# Convert output_files from list of paths to dict
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
# Convert output_files from list of paths to dict if needed
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
if isinstance(raw_output_files, list):
output_files = {}
@@ -371,20 +360,15 @@ class GCEEncodingBackend(EncodingBackend):
if not isinstance(path, str):
continue
filename = path.split("/")[-1] if "/" in path else path
filename_lower = filename.lower()
# Map filename patterns to output format keys
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
if "lossless 4k" in filename_lower:
if filename.endswith(".mkv"):
output_files["mkv_4k"] = path
else:
output_files["mp4_4k_lossless"] = path
elif "lossy 4k" in filename_lower:
if "4k_lossless" in filename:
output_files["mp4_4k_lossless"] = path
elif "4k_lossy" in filename:
output_files["mp4_4k_lossy"] = path
elif "720p" in filename_lower:
elif "720p" in filename:
output_files["mp4_720p"] = path
elif "with vocals" in filename_lower and filename.endswith(".mp4"):
output_files["with_vocals_mp4"] = path
elif filename.endswith(".mkv"):
output_files["mkv_4k"] = path
self.logger.info(f"Converted output_files list to dict: {output_files}")
else:
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
@@ -395,7 +379,6 @@ class GCEEncodingBackend(EncodingBackend):
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
lossless_mkv_path=output_files.get("mkv_4k"),
lossy_720p_mp4_path=output_files.get("mp4_720p"),
with_vocals_mp4_path=output_files.get("with_vocals_mp4"),
output_files=output_files,
encoding_time_seconds=encoding_time,
encoding_backend=self.name
+27 -495
View File
@@ -17,7 +17,6 @@ Usage:
import asyncio
import logging
import time
from typing import Optional, Dict, Any
import aiohttp
@@ -26,24 +25,6 @@ from backend.config import get_settings
logger = logging.getLogger(__name__)
# Retry configuration for handling transient failures (e.g., worker restarts during deployments).
#
# During a CI deployment, the encoding worker is restarted via systemctl. The restart
# involves downloading a new wheel from GCS, installing it, and starting uvicorn —
# which can take 30-90 seconds. The retry window must exceed this to avoid failing
# jobs that happen to hit the encoding stage during a deploy.
#
# With 7 retries and exponential backoff (5s, 10s, 15s, 15s, 15s, 15s, 15s),
# total retry window is ~90s, sufficient to survive a full worker restart.
MAX_RETRIES = 7
INITIAL_BACKOFF_SECONDS = 5.0
MAX_BACKOFF_SECONDS = 15.0
# Poll failure tolerance: number of consecutive poll failures allowed before giving up.
# This prevents a single transient network blip during status polling from killing a
# long-running encoding job. Similar pattern to flacfetch status polling (PR #446).
MAX_CONSECUTIVE_POLL_FAILURES = 5
class EncodingService:
"""Service for dispatching encoding jobs to GCE worker."""
@@ -53,34 +34,6 @@ class EncodingService:
self._url = None
self._api_key = None
self._initialized = False
self._worker_manager = None
self._cached_url = None
self._url_cached_at = 0
self._URL_CACHE_TTL = 30 # seconds
def set_worker_manager(self, manager):
"""Set the worker manager for dynamic URL resolution from Firestore."""
self._worker_manager = manager
def _get_worker_url(self) -> str:
"""Get the current primary worker URL from Firestore (with TTL cache).
Falls back to static URL from config if worker_manager is not set.
"""
now = time.time()
if self._cached_url and (now - self._url_cached_at) < self._URL_CACHE_TTL:
return self._cached_url
if self._worker_manager:
config = self._worker_manager.get_config()
self._cached_url = config.primary_url
self._url_cached_at = now
return self._cached_url
# Fallback to static URL
if not self._initialized:
self._load_credentials()
return self._url
def _load_credentials(self):
"""Load encoding worker URL and API key from config/secrets."""
@@ -110,110 +63,6 @@ class EncodingService:
"""Check if GCE encoding is enabled and configured."""
return self.settings.use_gce_encoding and self.is_configured
@property
def is_preview_enabled(self) -> bool:
"""Check if GCE preview encoding is enabled and configured."""
return self.settings.use_gce_preview_encoding and self.is_configured
def _warmup_encoding_worker_fallback(self, job_id: str) -> None:
"""Safety net: try to start the encoding worker VM on first connection failure.
If the worker_manager is available, calls ensure_primary_running() so the VM
starts booting during the retry window. If worker_manager is not set (e.g. dev
mode with static URL), this is a no-op.
"""
if not self._worker_manager:
return
try:
result = self._worker_manager.ensure_primary_running()
if result["started"]:
logger.warning(
f"[job:{job_id}] Encoding worker unreachable — started VM {result['vm_name']} as fallback"
)
else:
logger.info(
f"[job:{job_id}] Encoding worker unreachable — VM {result['vm_name']} already running/starting"
)
except Exception as e:
logger.warning(f"[job:{job_id}] Encoding worker warmup fallback failed (non-fatal): {e}")
async def _request_with_retry(
self,
method: str,
url: str,
headers: Dict[str, str],
json_payload: Optional[Dict[str, Any]] = None,
timeout: float = 30.0,
job_id: str = "unknown",
) -> Dict[str, Any]:
"""
Make an HTTP request with retry logic for transient failures.
This handles connection errors that occur when the GCE worker is
restarting (e.g., during deployments) by retrying with exponential backoff.
Args:
method: HTTP method (GET, POST)
url: Request URL
headers: Request headers
json_payload: JSON body for POST requests
timeout: Request timeout in seconds
job_id: Job ID for logging
Returns:
Dict with keys:
- status (int): HTTP status code
- json (Any): Parsed JSON response body (if status 200, else None)
- text (str): Raw response text (if status != 200, else None)
Raises:
aiohttp.ClientConnectorError: If all retries fail due to connection errors
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
asyncio.TimeoutError: If all retries fail due to timeout
"""
last_exception = None
backoff = INITIAL_BACKOFF_SECONDS
for attempt in range(MAX_RETRIES + 1):
try:
async with aiohttp.ClientSession() as session:
if method.upper() == "POST":
async with session.post(
url, json=json_payload, headers=headers, timeout=timeout
) as resp:
# Return a copy of the response data since we exit the context
return {
"status": resp.status,
"json": await resp.json() if resp.status == 200 else None,
"text": await resp.text() if resp.status != 200 else None,
}
else: # GET
async with session.get(
url, headers=headers, timeout=timeout
) as resp:
return {
"status": resp.status,
"json": await resp.json() if resp.status == 200 else None,
"text": await resp.text() if resp.status != 200 else None,
}
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
last_exception = e
if attempt == 0:
self._warmup_encoding_worker_fallback(job_id)
if attempt < MAX_RETRIES:
logger.warning(
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
f"Retrying in {backoff:.1f}s..."
)
await asyncio.sleep(backoff)
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
else:
logger.error(
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
)
raise last_exception
async def submit_encoding_job(
self,
job_id: str,
@@ -241,7 +90,7 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/encode"
url = f"{self._url}/encode"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
@@ -252,40 +101,17 @@ class EncodingService:
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
async with aiohttp.ClientSession() as session:
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 409:
raise RuntimeError(f"Encoding job {job_id} already exists")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Job already complete on GCE worker, returning cached result")
return {"status": "cached", "job_id": job_id, "output_files": status.get("output_files")}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Encoding job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
# Job was in worker memory but got cleared (restart) — safe to raise original error
raise RuntimeError(f"Encoding job {job_id} conflict: 409 but job not found on status check")
raise
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
return resp["json"]
return await resp.json()
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
"""
@@ -302,25 +128,20 @@ class EncodingService:
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/status/{job_id}"
url = f"{self._url}/status/{job_id}"
headers = {"X-API-Key": self._api_key}
resp = await self._request_with_retry(
method="GET",
url=url,
headers=headers,
timeout=30.0,
job_id=job_id,
)
async with aiohttp.ClientSession() as session:
async with session.get(url, headers=headers, timeout=30) as resp:
if resp.status == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp.status == 404:
raise RuntimeError(f"Encoding job {job_id} not found")
if resp.status != 200:
text = await resp.text()
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 404:
raise RuntimeError(f"Encoding job {job_id} not found")
if resp["status"] != 200:
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
return resp["json"]
return await resp.json()
async def wait_for_completion(
self,
@@ -330,12 +151,7 @@ class EncodingService:
progress_callback=None,
) -> Dict[str, Any]:
"""
Poll for encoding job completion with tolerance for transient failures.
During a deployment, the encoding worker may restart while we're polling.
Rather than failing the entire job on a single poll error, we tolerate up to
MAX_CONSECUTIVE_POLL_FAILURES consecutive failures before giving up. This is
the same pattern used in flacfetch status polling (PR #446).
Poll for encoding job completion.
Args:
job_id: Job identifier
@@ -348,40 +164,19 @@ class EncodingService:
Raises:
TimeoutError: If job doesn't complete within timeout
RuntimeError: If job fails or too many consecutive poll failures
RuntimeError: If job fails
"""
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
start_time = asyncio.get_event_loop().time()
last_progress = 0
consecutive_failures = 0
while True:
elapsed = asyncio.get_event_loop().time() - start_time
if elapsed > timeout:
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
try:
status = await self.get_job_status(job_id)
# Reset failure counter on successful poll
consecutive_failures = 0
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError,
asyncio.TimeoutError, RuntimeError) as e:
consecutive_failures += 1
if consecutive_failures >= MAX_CONSECUTIVE_POLL_FAILURES:
logger.error(
f"[job:{job_id}] Status polling failed {consecutive_failures} consecutive times, giving up: {e}"
)
raise RuntimeError(
f"Encoding job {job_id} lost contact with worker after "
f"{consecutive_failures} consecutive poll failures: {e}"
)
logger.warning(
f"[job:{job_id}] Status poll failed ({consecutive_failures}/{MAX_CONSECUTIVE_POLL_FAILURES}): {e}. "
f"Worker may be restarting, will retry..."
)
await asyncio.sleep(poll_interval)
continue
status = await self.get_job_status(job_id)
# Handle case where GCE worker returns a list instead of dict
if isinstance(status, list):
@@ -440,254 +235,7 @@ class EncodingService:
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
# Submit the job
submit_result = await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# If cached, return immediately — encoding already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Encoding already cached, returning immediately")
return {"status": "complete", "output_files": submit_result.get("output_files")}
# If in_progress, another request is encoding it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Encoding already in progress, joining poll")
# Wait for completion
return await self.wait_for_completion(
job_id, progress_callback=progress_callback
)
async def submit_preview_encoding_job(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
) -> Dict[str, Any]:
"""
Submit a preview video encoding job to the GCE worker.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
Returns:
Response from the encoding worker
Raises:
Exception: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/encode-preview"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {
"job_id": job_id,
"ass_gcs_path": ass_gcs_path,
"audio_gcs_path": audio_gcs_path,
"output_gcs_path": output_gcs_path,
"background_color": background_color,
}
if background_image_gcs_path:
payload["background_image_gcs_path"] = background_image_gcs_path
if font_gcs_path:
payload["font_gcs_path"] = font_gcs_path
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
raise RuntimeError(f"Preview encoding job {job_id} already exists")
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
return resp["json"]
async def encode_preview_video(
self,
job_id: str,
ass_gcs_path: str,
audio_gcs_path: str,
output_gcs_path: str,
background_color: str = "black",
background_image_gcs_path: Optional[str] = None,
font_gcs_path: Optional[str] = None,
timeout: float = 90.0,
poll_interval: float = 2.0,
) -> Dict[str, Any]:
"""
Submit preview encoding job and wait for completion.
This is a convenience method that combines submit + wait with shorter
timeouts suitable for preview videos.
Args:
job_id: Unique job identifier
ass_gcs_path: GCS path to ASS subtitles file
audio_gcs_path: GCS path to audio file
output_gcs_path: GCS path for output video
background_color: Background color (default: black)
background_image_gcs_path: Optional GCS path to background image
font_gcs_path: Optional GCS path to custom font file
timeout: Maximum time to wait (default 90s for preview)
poll_interval: Seconds between status checks (default 2s)
Returns:
Final job status with output files
"""
# Submit the job
submit_result = await self.submit_preview_encoding_job(
job_id=job_id,
ass_gcs_path=ass_gcs_path,
audio_gcs_path=audio_gcs_path,
output_gcs_path=output_gcs_path,
background_color=background_color,
background_image_gcs_path=background_image_gcs_path,
font_gcs_path=font_gcs_path,
)
# If cached, return immediately - video already exists in GCS
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
return {"status": "complete", "output_path": submit_result.get("output_path")}
# If in_progress, another request is encoding it - just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
# Wait for completion with shorter timeout
return await self.wait_for_completion(
job_id=job_id,
poll_interval=poll_interval,
timeout=timeout,
)
async def submit_render_video_job(
self,
job_id: str,
render_config: Dict[str, Any],
) -> Dict[str, Any]:
"""
Submit a render-video job to the GCE worker.
Args:
job_id: Unique job identifier
render_config: Configuration dict containing original_corrections_gcs_path,
audio_gcs_path, output_gcs_prefix, artist, title, and any other render params.
Returns:
Response from the encoding worker
Raises:
RuntimeError: If submission fails
"""
self._load_credentials()
if not self.is_configured:
raise RuntimeError("Encoding service not configured")
url = f"{self._get_worker_url()}/render-video"
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
payload = {"job_id": job_id, **render_config}
logger.info(f"[job:{job_id}] Submitting render-video job to GCE worker: {url}")
resp = await self._request_with_retry(
method="POST",
url=url,
headers=headers,
json_payload=payload,
timeout=30.0,
job_id=job_id,
)
if resp["status"] == 401:
raise RuntimeError("Invalid API key for encoding worker")
if resp["status"] == 409:
# Job already exists on worker — check if it already completed
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
try:
status = await self.get_job_status(job_id)
job_status = status.get("status", "unknown")
if job_status == "complete":
logger.info(f"[job:{job_id}] Render-video job already complete on GCE worker, returning cached result")
return {
"status": "cached",
"job_id": job_id,
"output_files": status.get("output_files"),
"metadata": status.get("metadata"),
}
elif job_status in ("pending", "running"):
logger.info(f"[job:{job_id}] Render-video job still in progress on GCE worker")
return {"status": "in_progress", "job_id": job_id}
else:
raise RuntimeError(f"Render-video job {job_id} already exists with status: {job_status}")
except RuntimeError as e:
if "not found" in str(e).lower():
raise RuntimeError(f"Render-video job {job_id} conflict: 409 but job not found on status check")
raise
if resp["status"] != 200:
raise RuntimeError(f"Failed to submit render-video job: {resp['status']} - {resp['text']}")
return resp["json"]
async def render_video_on_gce(
self,
job_id: str,
render_config: Dict[str, Any],
progress_callback=None,
) -> Dict[str, Any]:
"""
Submit render-video job and wait for completion.
This is a convenience method that combines submit + wait.
Args:
job_id: Unique job identifier
render_config: Configuration dict for the render-video worker endpoint
progress_callback: Optional callback for progress updates
Returns:
Final job status with output files and metadata
"""
# Submit the job
submit_result = await self.submit_render_video_job(job_id, render_config)
# If cached, return immediately — rendering already done
submit_status = submit_result.get("status")
if submit_status == "cached":
logger.info(f"[job:{job_id}] Render-video already cached, returning immediately")
return {
"status": "complete",
"output_files": submit_result.get("output_files"),
"metadata": submit_result.get("metadata"),
}
# If in_progress, another request is rendering it — just wait for that
if submit_status == "in_progress":
logger.info(f"[job:{job_id}] Render-video already in progress, joining poll")
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
# Wait for completion
return await self.wait_for_completion(
@@ -706,7 +254,7 @@ class EncodingService:
if not self.is_configured:
return {"status": "not_configured"}
url = f"{self._get_worker_url()}/health"
url = f"{self._url}/health"
headers = {"X-API-Key": self._api_key}
try:
@@ -728,20 +276,4 @@ def get_encoding_service() -> EncodingService:
global _encoding_service
if _encoding_service is None:
_encoding_service = EncodingService()
# Wire up worker manager for dynamic URL resolution
try:
from backend.services.encoding_worker_manager import EncodingWorkerManager
from google.cloud import compute_v1, firestore
settings = get_settings()
db = firestore.Client(project=settings.google_cloud_project)
compute_client = compute_v1.InstancesClient()
manager = EncodingWorkerManager(
db=db,
compute_client=compute_client,
project_id=settings.google_cloud_project,
)
_encoding_service.set_worker_manager(manager)
except Exception:
# Fallback to static URL if worker manager setup fails
pass
return _encoding_service
-239
View File
@@ -1,239 +0,0 @@
"""
Blue-Green Encoding Worker Manager.
Manages the lifecycle and role assignment of two GCE encoding worker VMs
(blue/green) via a Firestore config document as the single source of truth.
The config document at `config/encoding-worker` tracks which VM is primary
(actively serving encoding requests) and which is secondary (idle or being
deployed to). This enables zero-downtime deployments by swapping roles
after deploying new code to the secondary.
Usage:
from google.cloud import firestore
from google.cloud.compute_v1 import InstancesClient
db = firestore.Client()
compute = InstancesClient()
manager = EncodingWorkerManager(db, compute)
# Get current config
config = manager.get_config()
print(f"Primary: {config.primary_vm} at {config.primary_url}")
# Ensure primary VM is running before dispatching work
result = manager.ensure_primary_running()
"""
import logging
from dataclasses import dataclass
from datetime import datetime, UTC
from typing import Optional
from google.cloud import firestore
logger = logging.getLogger(__name__)
CONFIG_COLLECTION = "config"
CONFIG_DOCUMENT = "encoding-worker"
ENCODING_WORKER_PORT = 8080
@dataclass
class EncodingWorkerConfig:
"""Represents the Firestore config/encoding-worker document."""
primary_vm: str
primary_ip: str
primary_version: str
primary_deployed_at: Optional[str]
secondary_vm: str
secondary_ip: str
secondary_version: Optional[str]
secondary_deployed_at: Optional[str]
last_swap_at: Optional[str]
last_activity_at: Optional[str]
deploy_in_progress: bool
deploy_in_progress_since: Optional[str]
@property
def primary_url(self) -> str:
return f"http://{self.primary_ip}:{ENCODING_WORKER_PORT}"
@property
def secondary_url(self) -> str:
return f"http://{self.secondary_ip}:{ENCODING_WORKER_PORT}"
class EncodingWorkerManager:
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
self._db = db
self._compute = compute_client
self._project_id = project_id
self._zone = zone
def _doc_ref(self):
"""Return a reference to the config/encoding-worker document."""
return self._db.collection(CONFIG_COLLECTION).document(CONFIG_DOCUMENT)
# ------------------------------------------------------------------
# Firestore config operations (Task 1)
# ------------------------------------------------------------------
def get_config(self) -> EncodingWorkerConfig:
"""Read config/encoding-worker from Firestore and return as dataclass."""
snapshot = self._doc_ref().get()
if not snapshot.exists:
raise ValueError(
f"Encoding worker config document not found at "
f"{CONFIG_COLLECTION}/{CONFIG_DOCUMENT}"
)
data = snapshot.to_dict()
return EncodingWorkerConfig(
primary_vm=data["primary_vm"],
primary_ip=data["primary_ip"],
primary_version=data["primary_version"],
primary_deployed_at=data.get("primary_deployed_at"),
secondary_vm=data["secondary_vm"],
secondary_ip=data["secondary_ip"],
secondary_version=data.get("secondary_version"),
secondary_deployed_at=data.get("secondary_deployed_at"),
last_swap_at=data.get("last_swap_at"),
last_activity_at=data.get("last_activity_at"),
deploy_in_progress=data.get("deploy_in_progress", False),
deploy_in_progress_since=data.get("deploy_in_progress_since"),
)
def update_activity(self) -> None:
"""Update last_activity_at timestamp to record recent usage."""
now = datetime.now(UTC).isoformat()
self._doc_ref().update({"last_activity_at": now})
logger.info("Updated encoding worker last_activity_at to %s", now)
def swap_roles(
self,
new_primary_vm: str,
new_primary_ip: str,
new_primary_version: str,
old_primary_vm: str,
old_primary_ip: str,
old_primary_version: str,
) -> None:
"""Atomically swap primary and secondary roles using a Firestore transaction.
Args:
new_primary_vm: VM name that will become the new primary.
new_primary_ip: Internal IP of the new primary.
new_primary_version: Version deployed on the new primary.
old_primary_vm: Current primary VM name (becomes secondary).
old_primary_ip: Internal IP of the old primary.
old_primary_version: Version on the old primary.
"""
transaction = self._db.transaction()
doc_ref = self._doc_ref()
now = datetime.now(UTC).isoformat()
@firestore.transactional
def _swap_in_transaction(txn, ref):
txn.update(ref, {
"primary_vm": new_primary_vm,
"primary_ip": new_primary_ip,
"primary_version": new_primary_version,
"primary_deployed_at": now,
"secondary_vm": old_primary_vm,
"secondary_ip": old_primary_ip,
"secondary_version": old_primary_version,
"secondary_deployed_at": None,
"last_swap_at": now,
"deploy_in_progress": False,
"deploy_in_progress_since": None,
})
_swap_in_transaction(transaction, doc_ref)
logger.info(
"Swapped encoding worker roles: %s -> primary, %s -> secondary",
new_primary_vm, old_primary_vm,
)
# ------------------------------------------------------------------
# VM lifecycle operations (Task 2)
# ------------------------------------------------------------------
def get_vm_status(self, vm_name: str) -> str:
"""Get the current status of a GCE VM (e.g. RUNNING, TERMINATED, STAGING)."""
instance = self._compute.get(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
return instance.status
def start_vm(self, vm_name: str) -> None:
"""Start a VM if it is not already running or starting.
Fire-and-forget: returns immediately. Caller should poll get_vm_status
if they need to wait for the VM to be fully RUNNING.
"""
status = self.get_vm_status(vm_name)
if status in ("RUNNING", "STAGING"):
logger.info("VM %s is already %s, skipping start", vm_name, status)
return
logger.info("Starting VM %s (current status: %s)", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def stop_vm(self, vm_name: str) -> None:
"""Stop a VM if it is not already stopped or stopping."""
status = self.get_vm_status(vm_name)
if status in ("TERMINATED", "STOPPING"):
logger.info("VM %s is already %s, skipping stop", vm_name, status)
return
logger.info("Stopping VM %s (current status: %s)", vm_name, status)
self._compute.stop(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
def ensure_primary_running(self) -> dict:
"""Ensure the primary encoding worker VM is running.
Reads config, starts primary VM if stopped, and updates activity timestamp.
Returns:
dict with keys:
- started (bool): True if VM was started, False if already running
- vm_name (str): Name of the primary VM
- primary_url (str): URL of the primary encoding worker
"""
config = self.get_config()
vm_name = config.primary_vm
status = self.get_vm_status(vm_name)
started = False
if status not in ("RUNNING", "STAGING"):
logger.info("Primary VM %s is %s, starting it", vm_name, status)
self._compute.start(
project=self._project_id,
zone=self._zone,
instance=vm_name,
)
started = True
else:
logger.info("Primary VM %s is already %s", vm_name, status)
self.update_activity()
return {
"started": started,
"vm_name": vm_name,
"primary_url": config.primary_url,
}
@@ -1 +0,0 @@
"""Production error monitoring service for Nomad Karaoke."""
-177
View File
@@ -1,177 +0,0 @@
"""Configuration for the production error monitoring service.
All constants, service lists, and environment-driven configuration
functions used across the error monitor modules.
"""
import os
# ---------------------------------------------------------------------------
# GCP project settings
# ---------------------------------------------------------------------------
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
# ---------------------------------------------------------------------------
# Log querying settings
# ---------------------------------------------------------------------------
#: How far back to look for errors on each run (minutes).
LOOKBACK_MINUTES: int = 15
#: Maximum number of log entries to fetch per service per run.
MAX_LOG_ENTRIES: int = 500
# ---------------------------------------------------------------------------
# Monitored services
# ---------------------------------------------------------------------------
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
"karaoke-backend",
"karaoke-decide",
"audio-separator",
# Gen2 Cloud Functions appear as Cloud Run services in logs
"gdrive-validator",
"github-runner-manager",
"backup-to-aws",
"divebar-mirror",
"kn-data-sync",
"divebar-lookup",
"encoding-worker-idle-shutdown",
]
MONITORED_CLOUD_RUN_JOBS: list[str] = [
"video-encoding-job",
"lyrics-transcription-job",
"audio-separation-job",
"audio-download-job",
]
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
MONITORED_CLOUD_FUNCTIONS: list[str] = []
MONITORED_GCE_INSTANCES: list[str] = [
"encoding-worker-a",
"encoding-worker-b",
"flacfetch-vm",
"divebar-sync-vm",
]
# ---------------------------------------------------------------------------
# Service dependency map
# Human-readable description of which services depend on which workers.
# Used as LLM context when generating root-cause analysis.
# ---------------------------------------------------------------------------
SERVICE_DEPENDENCY_MAP: str = """\
karaoke-backend (main API)
→ video-encoding-job (Cloud Run Job, triggered per order)
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
→ gdrive-validator (Cloud Function, validates output files)
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
karaoke-decide (catalog / discovery API)
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
→ divebar_lookup (Cloud Function, single-song lookup)
audio-separator (Cloud Run service, GPU stem separation)
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
flacfetch-vm (GCE VM)
→ audio-download-job (initiated by karaoke-backend)
divebar-sync-vm (GCE VM)
→ divebar_mirror (Cloud Function, reads from this VM)
Shared infrastructure:
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
"""
# ---------------------------------------------------------------------------
# LLM analysis settings
# ---------------------------------------------------------------------------
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-2.0-flash-001")
LLM_VERTEX_LOCATION: str = os.environ.get("LLM_VERTEX_LOCATION", "us-central1")
#: Minimum number of active error patterns required before triggering LLM analysis.
MIN_PATTERNS_FOR_ANALYSIS: int = 3
# ---------------------------------------------------------------------------
# Spike detection
# ---------------------------------------------------------------------------
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
SPIKE_MULTIPLIER: float = 5.0
#: Minimum absolute error count before spike detection fires (avoids noise).
SPIKE_MIN_COUNT: int = 5
# ---------------------------------------------------------------------------
# Auto-resolution settings
# ---------------------------------------------------------------------------
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
AUTO_RESOLVE_MULTIPLIER: int = 8
#: Minimum hours before a pattern can be auto-resolved.
AUTO_RESOLVE_MIN_HOURS: int = 6
#: Maximum hours before a pattern is force-resolved even if still active.
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
#: Fallback resolve window (hours) if baseline data is unavailable.
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
# ---------------------------------------------------------------------------
# Discord settings
# ---------------------------------------------------------------------------
#: Maximum Discord messages to send per monitor run (rate-limit guard).
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
#: Discord API hard limit on message length.
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
# ---------------------------------------------------------------------------
# Pattern tracking settings
# ---------------------------------------------------------------------------
#: Maximum number of active error patterns to track in Firestore.
MAX_ACTIVE_PATTERNS: int = 500
#: Rolling window used when computing error baselines.
ROLLING_WINDOW_DAYS: int = 7
#: Maximum length of a normalized error message stored in Firestore.
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
# ---------------------------------------------------------------------------
# Helper functions
# ---------------------------------------------------------------------------
def get_llm_enabled() -> bool:
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
def get_discord_webhook_secret_name() -> str:
"""Return the Secret Manager secret name for the Discord webhook URL."""
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
def get_digest_mode() -> bool:
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
In digest mode all alerts are bundled into a single Discord message sent at
the end of the run, rather than sending one message per error pattern.
"""
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
-323
View File
@@ -1,323 +0,0 @@
"""Discord alert client for the production error monitoring service.
Provides formatting functions for each alert type, and an ErrorMonitorDiscord
webhook client that rate-limits outbound messages to MAX_DISCORD_MESSAGES_PER_RUN
per monitor run.
Usage:
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_new_pattern_alert,
format_incident_alert,
format_auto_resolved_alert,
format_spike_alert,
format_daily_digest,
)
discord = ErrorMonitorDiscord(webhook_url="https://discord.com/api/webhooks/...")
msg = format_new_pattern_alert(...)
discord.send_message(msg)
"""
import logging
from datetime import datetime
from zoneinfo import ZoneInfo
import requests
from backend.services.error_monitor.config import (
DISCORD_MAX_MESSAGE_LENGTH,
MAX_DISCORD_MESSAGES_PER_RUN,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
SEVERITY_EMOJI: dict[str, str] = {
"P0": "🚨",
"P1": "🔴",
"P2": "⚠️",
"P3": "🟡",
}
_EASTERN = ZoneInfo("America/New_York")
# ---------------------------------------------------------------------------
# Private helpers
# ---------------------------------------------------------------------------
def _now_eastern() -> str:
"""Return the current Eastern time as 'HH:MM AM/PM ET' (DST-aware)."""
now = datetime.now(tz=_EASTERN)
return now.strftime("%I:%M %p ET").lstrip("0")
def _truncate(msg: str) -> str:
"""Truncate *msg* to DISCORD_MAX_MESSAGE_LENGTH characters."""
if len(msg) <= DISCORD_MAX_MESSAGE_LENGTH:
return msg
suffix = "…"
return msg[: DISCORD_MAX_MESSAGE_LENGTH - len(suffix)] + suffix
# ---------------------------------------------------------------------------
# Formatting functions
# ---------------------------------------------------------------------------
def format_new_pattern_alert(
service: str,
resource_type: str,
normalized_message: str,
sample_message: str,
count: int,
first_seen: str,
) -> str:
"""Format a Discord alert for a newly-detected error pattern.
Args:
service: Name of the service emitting the error.
resource_type: GCP resource type (e.g. 'cloud_run_service', 'gce_instance').
normalized_message: Normalised error pattern string.
sample_message: A representative raw log message.
count: Number of occurrences seen so far.
first_seen: ISO 8601 timestamp when the pattern was first seen.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"🔴 **New Error Pattern Detected**",
f"",
f"**Service:** {service}",
f"**Resource type:** {resource_type}",
f"**Pattern:** {normalized_message}",
f"**Sample:** {sample_message}",
f"**Count:** {count}",
f"**First seen:** {first_seen}",
f"**Alerted at:** {_now_eastern()}",
f"",
f"To investigate: `/prod-investigate {service} \"{normalized_message}\"`",
]
return _truncate("\n".join(lines))
def format_incident_alert(
title: str,
severity: str,
root_cause: str,
suggested_fix: str,
primary_service: str,
patterns: list[dict],
) -> str:
"""Format a Discord alert for an analysed incident (LLM-generated).
Args:
title: Short incident title.
severity: Severity level — one of P0, P1, P2, P3.
root_cause: Human-readable root-cause analysis.
suggested_fix: Recommended remediation steps.
primary_service: The main service involved in the incident.
patterns: List of dicts, each with keys: service, normalized_message, count.
Only the first 5 are shown; normalized_message is truncated to 80 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
emoji = SEVERITY_EMOJI.get(severity, "⚠️")
lines = [
f"{emoji} **[{severity}] {title}**",
f"",
f"**Primary service:** {primary_service}",
f"**Root cause:** {root_cause}",
f"**Suggested fix:** {suggested_fix}",
]
shown_patterns = patterns[:5]
if shown_patterns:
lines.append("")
lines.append(f"**Patterns ({len(shown_patterns)} of {len(patterns)}):**")
for p in shown_patterns:
svc = p.get("service", "unknown")
msg = p.get("normalized_message", "")[:80]
cnt = p.get("count", 0)
lines.append(f" • `{svc}` — {msg} ({cnt}x)")
lines.append("")
lines.append(f"**Alerted at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_auto_resolved_alert(resolved_patterns: list[dict]) -> str:
"""Format a consolidated auto-resolved alert.
Args:
resolved_patterns: List of dicts, each with keys: service, message, hours_silent.
Only the first 8 are shown; message is truncated to 60 chars.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"✅ **Patterns Auto-Resolved**",
f"",
]
shown = resolved_patterns[:8]
if shown:
for p in shown:
svc = p.get("service", "unknown")
msg = p.get("message", "")[:60]
hours = p.get("hours_silent", 0)
lines.append(f" • `{svc}` — {msg} (silent {hours}h)")
else:
lines.append(" *(no patterns)*")
if len(resolved_patterns) > 8:
lines.append(f" … and {len(resolved_patterns) - 8} more")
lines.append("")
lines.append(f"**Resolved at:** {_now_eastern()}")
return _truncate("\n".join(lines))
def format_spike_alert(
service: str,
normalized_message: str,
current_count: int,
rolling_average: float,
) -> str:
"""Format a Discord alert for an error rate spike.
Args:
service: Name of the service.
normalized_message: Normalised error pattern string.
current_count: Error count in the current window.
rolling_average: Baseline rolling average count.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
if rolling_average > 0:
multiplier = current_count / rolling_average
multiplier_str = f"{multiplier:.1f}x"
else:
multiplier_str = "∞x"
lines = [
f"⚠️ **Error Rate Spike Detected**",
f"",
f"**Service:** {service}",
f"**Pattern:** {normalized_message}",
f"**Current count:** {current_count}",
f"**Normal (rolling avg):** {rolling_average}",
f"**Multiplier:** {multiplier_str} above normal",
f"",
f"**Alerted at:** {_now_eastern()}",
]
return _truncate("\n".join(lines))
def format_daily_digest(
total_errors: int,
new_patterns: int,
resolved_patterns: int,
active_patterns: int,
per_service_counts: dict,
) -> str:
"""Format the daily 24h error digest.
Args:
total_errors: Total error count in the last 24 hours.
new_patterns: Number of new error patterns detected.
resolved_patterns: Number of patterns auto-resolved.
active_patterns: Total number of currently active patterns.
per_service_counts: Mapping of service name → error count.
Displayed sorted by count descending.
Returns:
Formatted Discord message string, truncated to 2000 chars.
"""
lines = [
f"📊 **Daily Error Digest (24h)**",
f"",
f"**Total errors:** {total_errors}",
f"**New patterns:** {new_patterns}",
f"**Resolved patterns:** {resolved_patterns}",
f"**Active patterns:** {active_patterns}",
]
if per_service_counts:
lines.append("")
lines.append("**Per-service breakdown:**")
sorted_services = sorted(per_service_counts.items(), key=lambda kv: kv[1], reverse=True)
for svc, cnt in sorted_services:
lines.append(f" • `{svc}`: {cnt}")
lines.append("")
lines.append(f"**Generated at:** {_now_eastern()}")
return _truncate("\n".join(lines))
# ---------------------------------------------------------------------------
# Discord webhook client
# ---------------------------------------------------------------------------
class ErrorMonitorDiscord:
"""Webhook client for sending error monitor alerts to Discord.
Enforces a per-run rate limit of MAX_DISCORD_MESSAGES_PER_RUN to avoid
flooding the channel during high-volume incident windows.
Attributes:
webhook_url: The Discord incoming webhook URL.
messages_sent: Counter of messages attempted this run (success or failure).
"""
def __init__(self, webhook_url: str) -> None:
self.webhook_url = webhook_url
self.messages_sent: int = 0
def send_message(self, content: str) -> bool:
"""Send a message to the configured Discord webhook.
Rate-limits: if messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN, the
message is silently skipped and False is returned without incrementing
the counter.
Args:
content: The message text to send (should be pre-formatted and
pre-truncated to ≤2000 chars).
Returns:
True if the message was sent successfully, False otherwise.
"""
if self.messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN:
logger.warning(
"Discord rate limit reached (%d messages sent this run); skipping message.",
self.messages_sent,
)
return False
try:
response = requests.post(
self.webhook_url,
json={"content": content},
timeout=30,
)
response.raise_for_status()
logger.debug("Discord message sent successfully.")
return True
except Exception as exc:
logger.error("Failed to send Discord message: %s", exc)
return False
finally:
self.messages_sent += 1
@@ -1,502 +0,0 @@
"""Firestore adapter for the production error monitoring service.
Handles all CRUD for three Firestore collections:
- error_patterns: normalized error patterns (pattern_id as doc ID)
- error_incidents: groups of related patterns (incident_id as doc ID)
- discord_alerts: audit trail for Discord messages (alert_id as doc ID)
"""
from __future__ import annotations
import secrets
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any
from backend.services.error_monitor.config import (
AUTO_RESOLVE_FALLBACK_HOURS,
AUTO_RESOLVE_MAX_HOURS,
AUTO_RESOLVE_MIN_HOURS,
AUTO_RESOLVE_MULTIPLIER,
ROLLING_WINDOW_DAYS,
)
# ---------------------------------------------------------------------------
# Data classes
# ---------------------------------------------------------------------------
@dataclass
class PatternData:
"""Input data for creating or updating an error pattern."""
pattern_id: str
service: str
resource_type: str
normalized_message: str
sample_message: str
count: int
timestamp: datetime
@dataclass
class UpsertResult:
"""Result of an upsert_pattern operation."""
pattern_id: str
is_new: bool
previous_status: str | None = None
# ---------------------------------------------------------------------------
# Collection names
# ---------------------------------------------------------------------------
_COL_PATTERNS = "error_patterns"
_COL_INCIDENTS = "error_incidents"
_COL_ALERTS = "discord_alerts"
# Statuses that can be queried as "active"
_ACTIVE_STATUSES = ["new", "acknowledged", "known"]
# Statuses from which a pattern is reactivated when the error recurs
_REACTIVATION_STATUSES = {"auto_resolved", "fixed"}
# Statuses eligible for auto-resolve checking
_AUTO_RESOLVE_ELIGIBLE = {"new", "acknowledged"}
def _utcnow() -> datetime:
"""Return current UTC datetime."""
return datetime.now(tz=timezone.utc)
def _iso(dt: datetime) -> str:
"""Serialise a datetime to ISO-8601 string."""
return dt.isoformat()
def _generate_id(prefix: str, fmt: str) -> str:
"""Generate a time-stamped ID with 8 hex random suffix.
Args:
prefix: e.g. "inc" or "alert"
fmt: strftime format string applied to utcnow(), e.g. "%Y%m%d_%H%M"
"""
ts = _utcnow().strftime(fmt)
suffix = secrets.token_hex(4) # 8 hex chars
return f"{prefix}_{ts}_{suffix}"
# ---------------------------------------------------------------------------
# Adapter
# ---------------------------------------------------------------------------
class ErrorPatternsAdapter:
"""Firestore CRUD adapter for error monitoring collections.
Args:
db: Optional Firestore client. If omitted, a default client for the
``nomadkaraoke`` GCP project is constructed on first use.
"""
def __init__(self, db=None) -> None:
if db is None:
from google.cloud import firestore # type: ignore[import]
db = firestore.Client(project="nomadkaraoke")
self._db = db
# ------------------------------------------------------------------
# Patterns — read
# ------------------------------------------------------------------
def get_pattern(self, pattern_id: str) -> dict | None:
"""Return the pattern document as a dict, or None if it doesn't exist.
Args:
pattern_id: Firestore document ID in the ``error_patterns`` collection.
"""
snap = self._db.collection(_COL_PATTERNS).document(pattern_id).get()
if not snap.exists:
return None
return snap.to_dict()
def get_active_patterns(self) -> list[dict]:
"""Return all patterns with status in [new, acknowledged, known]."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", _ACTIVE_STATUSES
)
return [snap.to_dict() for snap in query.stream()]
def get_patterns_for_auto_resolve(self) -> list[dict]:
"""Return active patterns eligible for auto-resolution (new or acknowledged)."""
query = self._db.collection(_COL_PATTERNS).where(
"status", "in", list(_AUTO_RESOLVE_ELIGIBLE)
)
return [snap.to_dict() for snap in query.stream()]
def get_unalerted_new_patterns(self) -> list[dict]:
"""Return patterns with status="new" and alerted_at=None.
These are typically patterns written by a path other than the current
monitor cycle — most notably frontend crashes ingested via
``POST /api/client-errors``. The monitor uses this to ensure out-of-band
patterns still get a Discord alert on the next scheduled run.
"""
query = self._db.collection(_COL_PATTERNS).where(
"status", "==", "new"
).where("alerted_at", "==", None)
return [snap.to_dict() for snap in query.stream()]
# ------------------------------------------------------------------
# Patterns — upsert
# ------------------------------------------------------------------
def upsert_pattern(self, data: PatternData) -> UpsertResult:
"""Create a new pattern or update an existing one.
New pattern:
- status="new", severity="P3"
- Initialize ``rolling_counts`` with one entry for this run.
Existing pattern:
- Increment ``total_count`` and append to ``rolling_counts``.
- If status was ``auto_resolved`` or ``fixed`` → reactivate (set status="new").
- Prune ``rolling_counts`` entries older than ROLLING_WINDOW_DAYS.
Returns:
UpsertResult with ``is_new=True`` for brand-new patterns *and* for
reactivated patterns (auto_resolved → new, fixed → new).
"""
doc_ref = self._db.collection(_COL_PATTERNS).document(data.pattern_id)
snap = doc_ref.get()
ts_iso = _iso(data.timestamp)
new_rolling_entry = {"ts": ts_iso, "count": data.count}
if not snap.exists:
# ── brand new pattern ──────────────────────────────────────────
doc_ref.set(
{
"pattern_id": data.pattern_id,
"service": data.service,
"resource_type": data.resource_type,
"normalized_message": data.normalized_message,
"sample_message": data.sample_message,
"status": "new",
"severity": "P3",
"total_count": data.count,
"rolling_counts": [new_rolling_entry],
"first_seen": ts_iso,
"last_seen": ts_iso,
"alerted_at": None,
"created_at": ts_iso,
"updated_at": ts_iso,
}
)
return UpsertResult(pattern_id=data.pattern_id, is_new=True)
# ── existing pattern ───────────────────────────────────────────────
existing = snap.to_dict()
prev_status = existing.get("status", "new")
rolling_counts: list[dict] = list(existing.get("rolling_counts") or [])
rolling_counts.append(new_rolling_entry)
rolling_counts = self._prune_rolling_counts(rolling_counts)
update: dict[str, Any] = {
"total_count": existing.get("total_count", 0) + data.count,
"rolling_counts": rolling_counts,
"last_seen": ts_iso,
"sample_message": data.sample_message,
"updated_at": ts_iso,
}
# Reactivation: pattern was considered resolved/fixed but error recurred
is_new = False
if prev_status in _REACTIVATION_STATUSES:
update["status"] = "new"
update["alerted_at"] = None
is_new = True
doc_ref.update(update)
return UpsertResult(
pattern_id=data.pattern_id,
is_new=is_new,
previous_status=prev_status,
)
# ------------------------------------------------------------------
# Patterns — status changes
# ------------------------------------------------------------------
def auto_resolve_pattern(self, pattern_id: str, hours_silent: float) -> None:
"""Mark a pattern as auto-resolved.
Args:
pattern_id: Firestore document ID.
hours_silent: How many hours the pattern has been silent.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "auto_resolved",
"auto_resolve_hours_silent": hours_silent,
"auto_resolved_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def resolve_pattern(
self,
pattern_id: str,
pr_url: str | None = None,
note: str = "",
) -> None:
"""Mark a pattern as fixed (human-resolved).
Args:
pattern_id: Firestore document ID.
pr_url: Optional URL of the PR that fixed this pattern.
note: Optional human-readable note about the fix.
"""
now_iso = _iso(_utcnow())
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"status": "fixed",
"fixed_by": {
"pr_url": pr_url,
"note": note,
"fixed_at": now_iso,
},
"updated_at": now_iso,
}
)
def update_pattern_alerted(self, pattern_id: str) -> None:
"""Set the ``alerted_at`` timestamp on a pattern.
Called after a Discord alert has been successfully sent for the pattern.
"""
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"alerted_at": _iso(_utcnow()),
"updated_at": _iso(_utcnow()),
}
)
def merge_pattern(self, source_id: str, target_id: str, reason: str) -> None:
"""Merge *source* into *target* by muting the source pattern.
Args:
source_id: Pattern to mute.
target_id: Pattern that the source is being merged into.
reason: Human-readable rationale for the merge.
"""
self._db.collection(_COL_PATTERNS).document(source_id).update(
{
"status": "muted",
"merged_into": target_id,
"merge_reason": reason,
"updated_at": _iso(_utcnow()),
}
)
# ------------------------------------------------------------------
# Auto-resolve logic
# ------------------------------------------------------------------
def check_auto_resolve(self, pattern: dict) -> float | None:
"""Determine whether *pattern* should be auto-resolved.
Computes the frequency-aware silence threshold and compares it against
how long the pattern has been silent.
Args:
pattern: A pattern document dict (as returned by get_pattern).
Returns:
The number of hours the pattern has been silent if the threshold is
exceeded, or ``None`` if the pattern should not yet be resolved.
"""
status = pattern.get("status", "")
if status not in _AUTO_RESOLVE_ELIGIBLE:
return None
last_seen_str = pattern.get("last_seen")
if not last_seen_str:
return None
last_seen = datetime.fromisoformat(last_seen_str)
now = datetime.now(tz=timezone.utc)
# Ensure both datetimes are tz-aware for subtraction
if last_seen.tzinfo is None:
last_seen = last_seen.replace(tzinfo=timezone.utc)
hours_silent = (now - last_seen).total_seconds() / 3600.0
rolling_counts = pattern.get("rolling_counts") or []
threshold = self._compute_resolve_threshold(rolling_counts)
if hours_silent >= threshold:
return hours_silent
return None
def _compute_resolve_threshold(self, rolling_counts: list[dict]) -> float:
"""Compute the frequency-aware auto-resolve threshold in hours.
Algorithm:
1. If fewer than 3 data points → return AUTO_RESOLVE_FALLBACK_HOURS (48).
2. Compute mean interval between consecutive timestamps.
3. threshold = mean_interval_hours * AUTO_RESOLVE_MULTIPLIER.
4. Clamp result to [AUTO_RESOLVE_MIN_HOURS, AUTO_RESOLVE_MAX_HOURS].
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Threshold in hours (float).
"""
if len(rolling_counts) < 3:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
# Parse timestamps and sort ascending
timestamps: list[datetime] = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
timestamps.append(ts)
timestamps.sort()
# Compute pairwise intervals
intervals_hours: list[float] = []
for i in range(1, len(timestamps)):
delta = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600.0
intervals_hours.append(delta)
if not intervals_hours:
return float(AUTO_RESOLVE_FALLBACK_HOURS)
mean_interval = sum(intervals_hours) / len(intervals_hours)
threshold = mean_interval * AUTO_RESOLVE_MULTIPLIER
# Clamp to configured bounds
threshold = max(float(AUTO_RESOLVE_MIN_HOURS), threshold)
threshold = min(float(AUTO_RESOLVE_MAX_HOURS), threshold)
return threshold
def _prune_rolling_counts(self, rolling_counts: list[dict]) -> list[dict]:
"""Remove rolling_count entries older than ROLLING_WINDOW_DAYS.
Args:
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
Returns:
Filtered list with only recent entries.
"""
if not rolling_counts:
return []
cutoff = _utcnow() - timedelta(days=ROLLING_WINDOW_DAYS)
result = []
for entry in rolling_counts:
ts = datetime.fromisoformat(entry["ts"])
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
if ts >= cutoff:
result.append(entry)
return result
# ------------------------------------------------------------------
# Incidents
# ------------------------------------------------------------------
def create_incident(
self,
title: str,
root_cause: str,
severity: str,
suggested_fix: str,
primary_service: str,
pattern_ids: list[str],
used_llm: bool,
) -> str:
"""Create a new incident document and link associated patterns.
Args:
title: Short human-readable title.
root_cause: LLM-generated or manual root-cause description.
severity: Severity string (e.g. "P1", "P2", "P3").
suggested_fix: Suggested remediation steps.
primary_service: The primary affected service.
pattern_ids: List of pattern_ids that belong to this incident.
used_llm: Whether LLM analysis was used to generate this incident.
Returns:
The generated incident_id string.
"""
incident_id = _generate_id("inc", "%Y%m%d_%H%M")
now_iso = _iso(_utcnow())
self._db.collection(_COL_INCIDENTS).document(incident_id).set(
{
"incident_id": incident_id,
"title": title,
"root_cause": root_cause,
"severity": severity,
"suggested_fix": suggested_fix,
"primary_service": primary_service,
"pattern_ids": pattern_ids,
"used_llm": used_llm,
"created_at": now_iso,
"updated_at": now_iso,
"status": "open",
}
)
# Link each pattern back to this incident
for pattern_id in pattern_ids:
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
{
"incident_id": incident_id,
"updated_at": now_iso,
}
)
return incident_id
# ------------------------------------------------------------------
# Discord audit trail
# ------------------------------------------------------------------
def log_discord_alert(
self,
alert_type: str,
content: str,
success: bool,
metadata: dict | None = None,
) -> None:
"""Write an audit record to the ``discord_alerts`` collection.
Args:
alert_type: Category of alert (e.g. "new_pattern", "digest").
content: The message body that was (or would have been) sent.
success: Whether the Discord webhook call succeeded.
metadata: Optional additional data (pattern_ids, incident_id, etc.).
"""
alert_id = _generate_id("alert", "%Y%m%d_%H%M%S")
now_iso = _iso(_utcnow())
self._db.collection(_COL_ALERTS).document(alert_id).set(
{
"alert_id": alert_id,
"alert_type": alert_type,
"content": content,
"success": success,
"metadata": metadata or {},
"created_at": now_iso,
}
)
@@ -1,127 +0,0 @@
"""Frontend error ingestion helpers.
Converts an inbound browser crash report into a ``PatternData`` suitable for
the shared ``ErrorPatternsAdapter``. The adapter + existing error-monitor
Cloud Run Job handle all alerting / Discord plumbing.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
from urllib.parse import urlsplit, urlunsplit
from backend.services.error_monitor.firestore_adapter import PatternData
from backend.services.error_monitor.normalizer import (
compute_pattern_hash,
normalize_message,
)
MAX_SAMPLE_MESSAGE_CHARS = 4000
MAX_URL_CHARS = 512
@dataclass
class FrontendErrorReport:
"""In-memory representation of an inbound crash report."""
message: str
stack: str | None
url: str
user_agent: str
release: str
user_email: str | None
viewport: dict | None
locale: str
extra: dict | None
def sanitize_url(url: str) -> str:
"""Strip query and fragment from a URL; cap length; tolerate junk."""
if not url:
return ""
try:
parts = urlsplit(url)
if not parts.scheme:
return url[:MAX_URL_CHARS]
cleaned = urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
return cleaned[:MAX_URL_CHARS]
except ValueError:
return url[:MAX_URL_CHARS]
def _stack_for_hashing(report: FrontendErrorReport) -> str:
"""Pick the most stable signal we have for pattern dedup.
Prefer the stack (stable across invocations) over the message (sometimes
has interpolated values). Falls back to message if no stack.
"""
if report.stack:
return report.stack
return report.message
def build_pattern_data(
report: FrontendErrorReport, now: datetime | None = None
) -> PatternData:
"""Convert an inbound report to a PatternData ready for upsert."""
if now is None:
now = datetime.now(tz=timezone.utc)
raw = _stack_for_hashing(report)
normalized = normalize_message(raw)
pattern_id = compute_pattern_hash("frontend", normalized)
# sample_message is human-readable context. Keep the error message plus a
# trimmed stack + sanitized URL so the Discord alert is self-contained.
sample_parts: list[str] = []
if report.message:
sample_parts.append(report.message.strip())
if report.stack and report.stack.strip() != (report.message or "").strip():
sample_parts.append(report.stack.strip())
clean_url = sanitize_url(report.url)
if clean_url:
sample_parts.append(f"URL: {clean_url}")
if report.user_agent:
sample_parts.append(f"UA: {report.user_agent[:200]}")
if report.release:
sample_parts.append(f"Build: {report.release}")
sample_message = "\n".join(sample_parts)[:MAX_SAMPLE_MESSAGE_CHARS]
return PatternData(
pattern_id=pattern_id,
service="frontend",
resource_type="browser",
normalized_message=normalized,
sample_message=sample_message,
count=1,
timestamp=now,
)
class RateLimiter:
"""In-memory sliding-window limiter. One instance per process is enough —
this runs inside Cloud Run which scales to multiple instances, so the
effective limit is (per_ip_per_minute * num_instances). That's fine for our
threat model (non-malicious browsers reporting their own crashes).
"""
def __init__(self, max_per_minute: int = 60) -> None:
self._max = max_per_minute
self._hits: dict[str, list[float]] = {}
def allow(self, ip: str, now_ts: float) -> bool:
cutoff = now_ts - 60.0
hits = [t for t in self._hits.get(ip, []) if t >= cutoff]
if len(hits) >= self._max:
self._hits[ip] = hits
return False
hits.append(now_ts)
self._hits[ip] = hits
# simple cleanup: if the map gets huge, drop stale keys
if len(self._hits) > 10_000:
self._hits = {
k: [t for t in v if t >= cutoff]
for k, v in self._hits.items()
if any(t >= cutoff for t in v)
}
return True
@@ -1,106 +0,0 @@
"""Known infrastructure noise patterns for the production error monitoring service.
Filters out transient, expected log messages so the error monitor is not flooded
with Cloud Run cold-start probes, deploy events, spot VM preemptions, and similar
operational noise.
Usage:
from backend.services.error_monitor.known_issues import should_ignore
reason = should_ignore(service, message)
if reason:
# skip this log entry — it's known infrastructure noise
...
"""
import re
from dataclasses import dataclass
@dataclass
class IgnoreReason:
"""Describes why a log entry should be suppressed.
Attributes:
pattern_name: Machine-readable name of the matching ignore pattern.
reason: Human-readable explanation of why this is infrastructure noise.
"""
pattern_name: str
reason: str
# ---------------------------------------------------------------------------
# Compiled ignore patterns (applied in order)
# All patterns use re.IGNORECASE for case-insensitive matching.
# ---------------------------------------------------------------------------
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
(
"startup_probe",
re.compile(r"startup probe failed", re.IGNORECASE),
"Cloud Run cold start — startup probe failure is transient and expected",
),
(
"ready_condition",
re.compile(r"ready condition status changed", re.IGNORECASE),
"Cloud Run deploy event — readiness condition change is expected on deploy",
),
(
"spot_preemption",
re.compile(r"instance was preempted", re.IGNORECASE),
"Spot/preemptible VM preemption — expected behaviour for spot instances",
),
(
"idle_shutdown",
re.compile(r"stopping idle encoding worker", re.IGNORECASE),
"Encoding worker idle shutdown — expected behaviour when no jobs are queued",
),
(
"health_check_404",
re.compile(
r"(?:GET|HEAD)\b[^\n]*?/health[^\n]*?\b(?:404|not found)\b",
re.IGNORECASE,
),
"Load balancer health check returning 404 — transient during cold start",
),
(
"scheduler_retry",
re.compile(r"cloud scheduler.*retry", re.IGNORECASE),
"Cloud Scheduler transient retry — expected on intermittent job failures",
),
(
"runner_startup",
re.compile(r"github\.runner\..*(?:starting|stopping|idle)", re.IGNORECASE),
"GitHub Actions runner lifecycle event — expected during runner start/stop",
),
(
"container_shutdown",
re.compile(r"container called exit\(0\)", re.IGNORECASE),
"Container clean exit — exit code 0 indicates a graceful, expected shutdown",
),
]
def should_ignore(service: str, message: str) -> IgnoreReason | None:
"""Check whether a log entry matches a known infrastructure noise pattern.
The check is purely message-based; ``service`` is accepted for API symmetry
and future service-specific overrides but does not gate any pattern today.
Args:
service: The originating service name (e.g. ``"karaoke-backend"``).
message: The raw log message string to evaluate.
Returns:
An :class:`IgnoreReason` if the message matches a known noise pattern,
or ``None`` if the message should be processed normally.
"""
if not message:
return None
for pattern_name, pattern, reason in _IGNORE_PATTERNS:
if pattern.search(message):
return IgnoreReason(pattern_name=pattern_name, reason=reason)
return None
@@ -1,429 +0,0 @@
"""LLM-powered incident analysis and duplicate detection for the error monitor.
Uses Gemini Flash via the google-generativeai library to:
1. Group related error patterns into incidents with root-cause analysis.
2. Identify near-duplicate patterns that the regex normalizer missed.
Both functions fall back gracefully when the LLM is unavailable or returns
unparseable output.
"""
from __future__ import annotations
import json
import logging
import re
import time
from collections import defaultdict
from dataclasses import dataclass, field
from google import genai
from google.genai import types as genai_types
from backend.services.error_monitor.config import (
GCP_PROJECT,
LLM_ANALYSIS_MODEL,
LLM_VERTEX_LOCATION,
MIN_PATTERNS_FOR_ANALYSIS,
SERVICE_DEPENDENCY_MAP,
)
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Dataclasses
# ---------------------------------------------------------------------------
@dataclass
class Incident:
"""A grouped incident produced by LLM analysis or the fallback grouper."""
title: str
root_cause: str | None
severity: str # P0 / P1 / P2 / P3
suggested_fix: str | None
primary_service: str
pattern_indices: list[int]
@dataclass
class IncidentAnalysis:
"""The result of an incident grouping pass."""
incidents: list[Incident]
used_llm: bool = True
@dataclass
class DuplicateGroup:
"""A canonical pattern plus the new patterns that duplicate it."""
canonical_index: int
duplicate_indices: list[int]
reason: str
# ---------------------------------------------------------------------------
# LLM call helper
# ---------------------------------------------------------------------------
_RETRY_ATTEMPTS = 3
_RETRY_BASE_DELAY = 2.0 # seconds
def _call_llm(
system_prompt: str,
user_prompt: str,
temperature: float = 0.2,
) -> str:
"""Call Gemini and return the response text.
Retries up to _RETRY_ATTEMPTS times with exponential back-off on failure.
Args:
system_prompt: System instruction for the model.
user_prompt: User-facing prompt content.
temperature: Sampling temperature (lower = more deterministic).
Returns:
The model's response text.
Raises:
Exception: Re-raises the last exception after all retries are exhausted.
"""
last_exc: Exception | None = None
for attempt in range(_RETRY_ATTEMPTS):
try:
client = genai.Client(
vertexai=True,
project=GCP_PROJECT,
location=LLM_VERTEX_LOCATION,
)
response = client.models.generate_content(
model=LLM_ANALYSIS_MODEL,
contents=user_prompt,
config=genai_types.GenerateContentConfig(
system_instruction=system_prompt,
temperature=temperature,
),
)
return response.text
except Exception as exc: # noqa: BLE001
last_exc = exc
delay = _RETRY_BASE_DELAY * (2**attempt)
logger.warning(
"LLM call failed (attempt %d/%d): %s — retrying in %.1fs",
attempt + 1,
_RETRY_ATTEMPTS,
exc,
delay,
)
time.sleep(delay)
raise last_exc # type: ignore[misc]
# ---------------------------------------------------------------------------
# Response parsing
# ---------------------------------------------------------------------------
_MD_CODE_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
def _strip_markdown_code_block(text: str) -> str:
"""Remove surrounding ```json ... ``` or ``` ... ``` fences if present."""
match = _MD_CODE_BLOCK_RE.search(text)
if match:
return match.group(1).strip()
return text.strip()
def _parse_llm_response(text: str) -> IncidentAnalysis | None:
"""Parse the LLM response text into an IncidentAnalysis.
Args:
text: Raw LLM output, possibly wrapped in a markdown code block.
Returns:
IncidentAnalysis on success, None on any parse failure.
"""
if not text:
return None
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("LLM returned non-JSON response (first 200 chars): %.200s", text)
return None
if "incidents" not in data:
logger.warning("LLM response missing 'incidents' key: %s", list(data.keys()))
return None
incidents: list[Incident] = []
for raw in data["incidents"]:
try:
incidents.append(
Incident(
title=raw.get("title", "Unknown incident"),
root_cause=raw.get("root_cause"),
severity=raw.get("severity", "P2"),
suggested_fix=raw.get("suggested_fix"),
primary_service=raw.get("primary_service", "unknown"),
pattern_indices=raw.get("pattern_indices", []),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed incident entry: %s — %s", raw, exc)
return IncidentAnalysis(incidents=incidents, used_llm=True)
# ---------------------------------------------------------------------------
# Fallback grouper
# ---------------------------------------------------------------------------
def _fallback_group_by_service(patterns: list[dict]) -> IncidentAnalysis:
"""Group patterns by service name when the LLM is unavailable.
Each unique service becomes one P2 incident containing all of that
service's pattern indices.
Args:
patterns: List of error pattern dicts (must include 'service' key).
Returns:
IncidentAnalysis with used_llm=False.
"""
service_to_indices: dict[str, list[int]] = defaultdict(list)
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
service_to_indices[service].append(idx)
incidents = [
Incident(
title=f"Errors in {service}",
root_cause=None,
severity="P2",
suggested_fix=None,
primary_service=service,
pattern_indices=indices,
)
for service, indices in service_to_indices.items()
]
return IncidentAnalysis(incidents=incidents, used_llm=False)
# ---------------------------------------------------------------------------
# Main analysis functions
# ---------------------------------------------------------------------------
_ANALYZE_SYSTEM_PROMPT = f"""\
You are an expert site-reliability engineer for Nomad Karaoke, a karaoke video
generation platform running on Google Cloud Platform.
Your task is to analyse a list of active error patterns detected across
production services, group them into meaningful incidents, and identify root
causes.
## Service dependency map
{SERVICE_DEPENDENCY_MAP}
## Severity definitions
- P0: Service is completely down / total outage
- P1: Major pipeline broken — orders cannot complete
- P2: Degraded performance — some orders affected
- P3: Minor / cosmetic issue with no direct user impact
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{{
"incidents": [
{{
"title": "<short human-readable title>",
"root_cause": "<root cause or null>",
"severity": "P0|P1|P2|P3",
"suggested_fix": "<actionable suggestion or null>",
"primary_service": "<service name>",
"pattern_indices": [<list of 0-based indices from the input list>]
}}
]
}}
Group related patterns together into a single incident where they share a
common root cause (e.g., a downstream dependency failure). Unrelated patterns
should be in separate incidents. Every input pattern must appear in exactly
one incident.
"""
_DEDUP_SYSTEM_PROMPT = """\
You are an expert at identifying near-duplicate error patterns in production
monitoring systems.
Your task is to compare a list of NEW error patterns against a list of
EXISTING (canonical) patterns and identify which new patterns are essentially
duplicates of existing ones — i.e., they describe the same underlying error
but with slightly different wording due to variation in log messages.
## Output format
Respond with a single JSON object (no markdown fencing) with this schema:
{
"duplicates": [
{
"canonical_index": <0-based index into the EXISTING patterns list>,
"duplicate_indices": [<0-based indices into the NEW patterns list>],
"reason": "<brief explanation>"
}
]
}
If no duplicates are found, return {"duplicates": []}.
"""
def analyze_patterns(patterns: list[dict]) -> IncidentAnalysis | None:
"""Group error patterns into incidents using Gemini LLM analysis.
Returns None if there are fewer than MIN_PATTERNS_FOR_ANALYSIS patterns.
Falls back to service-based grouping when the LLM fails or returns an
unparseable response.
Args:
patterns: List of error pattern dicts to analyse.
Returns:
IncidentAnalysis, or None if insufficient patterns.
"""
if len(patterns) < MIN_PATTERNS_FOR_ANALYSIS:
return None
user_prompt = _build_analyze_user_prompt(patterns)
try:
raw_response = _call_llm(_ANALYZE_SYSTEM_PROMPT, user_prompt, temperature=0.2)
result = _parse_llm_response(raw_response)
if result is not None:
return result
logger.warning("LLM returned unparseable response; falling back to service grouping")
except Exception as exc: # noqa: BLE001
logger.warning("LLM analysis failed: %s — falling back to service grouping", exc)
return _fallback_group_by_service(patterns)
def find_duplicate_patterns(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> list[DuplicateGroup]:
"""Identify near-duplicate patterns that the regex normalizer missed.
Returns an empty list if either input list is empty, or if the LLM call
fails for any reason.
Args:
new_patterns: Newly detected error patterns to check.
existing_patterns: Currently tracked (canonical) error patterns.
Returns:
List of DuplicateGroup objects (may be empty).
"""
if not new_patterns or not existing_patterns:
return []
user_prompt = _build_dedup_user_prompt(new_patterns, existing_patterns)
try:
raw_response = _call_llm(_DEDUP_SYSTEM_PROMPT, user_prompt, temperature=0.1)
return _parse_duplicate_response(raw_response)
except Exception as exc: # noqa: BLE001
logger.warning("Duplicate detection LLM call failed: %s", exc)
return []
# ---------------------------------------------------------------------------
# Prompt builders
# ---------------------------------------------------------------------------
def _build_analyze_user_prompt(patterns: list[dict]) -> str:
"""Build the user prompt listing patterns for incident grouping."""
lines = ["Error patterns to analyse:\n"]
for idx, pattern in enumerate(patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
count = pattern.get("count", 0)
lines.append(f"[{idx}] service={service!r} count={count} message={message!r}")
return "\n".join(lines)
def _build_dedup_user_prompt(
new_patterns: list[dict],
existing_patterns: list[dict],
) -> str:
"""Build the user prompt listing existing and new patterns for dedup."""
lines = ["EXISTING (canonical) patterns:\n"]
for idx, pattern in enumerate(existing_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
lines.append("\nNEW patterns to check:\n")
for idx, pattern in enumerate(new_patterns):
service = pattern.get("service", "unknown")
message = pattern.get("normalized_message", "")
lines.append(f"[{idx}] service={service!r} message={message!r}")
return "\n".join(lines)
def _parse_duplicate_response(text: str) -> list[DuplicateGroup]:
"""Parse the LLM response for duplicate detection.
Args:
text: Raw LLM output.
Returns:
List of DuplicateGroup objects, or [] on any failure.
"""
if not text:
return []
cleaned = _strip_markdown_code_block(text)
try:
data = json.loads(cleaned)
except json.JSONDecodeError:
logger.warning("Dedup LLM returned non-JSON: %.200s", text)
return []
if "duplicates" not in data:
logger.warning("Dedup LLM response missing 'duplicates' key")
return []
groups: list[DuplicateGroup] = []
for raw in data["duplicates"]:
try:
groups.append(
DuplicateGroup(
canonical_index=int(raw["canonical_index"]),
duplicate_indices=list(raw.get("duplicate_indices", [])),
reason=raw.get("reason", ""),
)
)
except Exception as exc: # noqa: BLE001
logger.warning("Skipping malformed duplicate entry: %s — %s", raw, exc)
return groups
-673
View File
@@ -1,673 +0,0 @@
"""Monitor orchestrator for the production error monitoring service.
This is the main entry point. Run as::
python -m backend.services.error_monitor.monitor
The ``ErrorMonitor.run()`` method executes the full pipeline:
* Monitor cycle (default) — query Cloud Logging, detect new patterns, send
Discord alerts, check auto-resolution.
* Daily digest mode (``DIGEST_MODE=true``) — bundle 24h stats into one message.
"""
from __future__ import annotations
import logging
import os
from datetime import datetime, timedelta, timezone
from typing import Any
from google.cloud import logging as cloud_logging
from google.cloud import secretmanager
from backend.services.error_monitor.config import (
GCP_PROJECT,
LOOKBACK_MINUTES,
MAX_LOG_ENTRIES,
MONITORED_CLOUD_FUNCTIONS,
MONITORED_CLOUD_RUN_JOBS,
MONITORED_CLOUD_RUN_SERVICES,
MONITORED_GCE_INSTANCES,
SPIKE_MIN_COUNT,
SPIKE_MULTIPLIER,
get_digest_mode,
get_discord_webhook_secret_name,
get_llm_enabled,
)
from backend.services.error_monitor.discord import (
ErrorMonitorDiscord,
format_auto_resolved_alert,
format_daily_digest,
format_incident_alert,
format_new_pattern_alert,
format_spike_alert,
)
from backend.services.error_monitor.firestore_adapter import (
ErrorPatternsAdapter,
PatternData,
UpsertResult,
)
from backend.services.error_monitor.known_issues import should_ignore
from backend.services.error_monitor.llm_analysis import analyze_patterns, find_duplicate_patterns
from backend.services.error_monitor.normalizer import compute_pattern_hash, normalize_message
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Module-level constants
# ---------------------------------------------------------------------------
#: Maps log resource type → (label_key, service_name_list)
RESOURCE_TYPE_FILTERS: dict[str, tuple[str, list[str]]] = {
"cloud_run_revision": ("resource.labels.service_name", MONITORED_CLOUD_RUN_SERVICES),
"cloud_run_job": ("resource.labels.job_name", MONITORED_CLOUD_RUN_JOBS),
"cloud_function": ("resource.labels.function_name", MONITORED_CLOUD_FUNCTIONS),
"gce_instance": ('labels."compute.googleapis.com/resource_name"', MONITORED_GCE_INSTANCES),
}
#: Maps log resource type → canonical resource_type string stored in Firestore.
RESOURCE_TYPE_MAP: dict[str, str] = {
"cloud_run_revision": "cloud_run_service",
"cloud_run_job": "cloud_run_job",
"cloud_function": "cloud_function",
"gce_instance": "gce_instance",
}
# ---------------------------------------------------------------------------
# Free functions
# ---------------------------------------------------------------------------
def _build_log_filter(
resource_type: str,
resource_names: list[str],
lookback_minutes: int,
) -> str:
"""Build a Cloud Logging filter string.
Args:
resource_type: GCP resource type (e.g. ``"cloud_run_revision"``).
resource_names: List of service/job/function/instance names to include.
lookback_minutes: How many minutes back to query.
Returns:
A filter string suitable for ``logging_client.list_entries(filter_=...)``.
"""
label_key = RESOURCE_TYPE_FILTERS[resource_type][0]
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=lookback_minutes)
cutoff_iso = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
name_clause = " OR ".join(f'{label_key}="{name}"' for name in resource_names)
if len(resource_names) > 1:
name_clause = f"({name_clause})"
return (
f'resource.type="{resource_type}" '
f"severity>=ERROR "
f'timestamp>="{cutoff_iso}" '
f"{name_clause}"
)
def _extract_message(entry: Any) -> str | None:
"""Extract the log message text from a Cloud Logging entry.
Handles several payload shapes:
* ``str`` payload — returned directly.
* ``dict`` payload — tries ``"message"``, ``"textPayload"``, ``"error"``,
``"msg"`` keys in that order.
* ``None`` payload — returns ``None``.
Args:
entry: A Cloud Logging ``LogEntry`` object.
Returns:
The message string, or ``None`` if not extractable.
"""
payload = entry.payload
if payload is None:
return None
if isinstance(payload, str):
return payload if payload else None
if isinstance(payload, dict):
for key in ("message", "textPayload", "error", "msg"):
value = payload.get(key)
if value:
return str(value)
return None
# Fallback: convert to string representation
text = str(payload)
return text if text else None
def _extract_service_name(entry: Any, resource_type: str) -> str:
"""Extract the service/job/function/instance name from a log entry.
Args:
entry: A Cloud Logging ``LogEntry`` object.
resource_type: GCP log resource type string.
Returns:
The service name string (falls back to ``"unknown"`` if not found).
"""
labels = getattr(entry.resource, "labels", {}) or {}
entry_labels = getattr(entry, "labels", {}) or {}
if resource_type == "cloud_run_revision":
return labels.get("service_name") or "unknown"
if resource_type == "cloud_run_job":
return labels.get("job_name") or "unknown"
if resource_type == "cloud_function":
return labels.get("function_name") or "unknown"
if resource_type == "gce_instance":
# Prefer the human-readable resource name label
name = entry_labels.get("compute.googleapis.com/resource_name")
if name:
return name
return labels.get("instance_id") or "unknown"
return "unknown"
def _classify_resource_type(log_resource_type: str) -> str:
"""Map a Cloud Logging resource type to the canonical type stored in Firestore.
Args:
log_resource_type: Raw GCP resource type (e.g. ``"cloud_run_revision"``).
Returns:
Canonical resource type string (falls back to the input if not in map).
"""
return RESOURCE_TYPE_MAP.get(log_resource_type, log_resource_type)
def _group_by_pattern(entries: list[dict]) -> dict[str, dict]:
"""Group normalised log entries by their pattern hash.
Args:
entries: List of dicts with ``service``, ``normalized``, and ``message`` keys.
Returns:
Dict mapping ``pattern_id`` → group dict containing ``service``,
``resource_type``, ``normalized``, ``sample_message``, ``count``.
"""
groups: dict[str, dict] = {}
for entry in entries:
service = entry["service"]
normalized = entry["normalized"]
pattern_id = compute_pattern_hash(service, normalized)
if pattern_id not in groups:
groups[pattern_id] = {
"pattern_id": pattern_id,
"service": service,
"resource_type": entry.get("resource_type", "unknown"),
"normalized": normalized,
"sample_message": entry["message"],
"count": 0,
}
groups[pattern_id]["count"] += 1
return groups
def _rolling_average(rolling_counts: list[dict]) -> float:
"""Compute the mean count from a list of rolling_count dicts.
Args:
rolling_counts: List of ``{"count": int, ...}`` dicts.
Returns:
The arithmetic mean as a float, or ``0.0`` for an empty list.
"""
if not rolling_counts:
return 0.0
total = sum(entry.get("count", 0) for entry in rolling_counts)
return total / len(rolling_counts)
def _is_spike(current_count: int, rolling_counts: list[dict]) -> bool:
"""Determine whether ``current_count`` represents an error rate spike.
A spike is detected when:
- ``current_count`` >= ``SPIKE_MIN_COUNT`` (avoids noise from rare errors), AND
- ``current_count`` > ``rolling_average * SPIKE_MULTIPLIER``
Args:
current_count: Error count in the current monitoring window.
rolling_counts: Historical rolling counts from Firestore.
Returns:
``True`` if a spike is detected, ``False`` otherwise.
"""
if not rolling_counts:
return False
if current_count < SPIKE_MIN_COUNT:
return False
avg = _rolling_average(rolling_counts)
if avg == 0.0:
return False
return current_count > avg * SPIKE_MULTIPLIER
# ---------------------------------------------------------------------------
# ErrorMonitor class
# ---------------------------------------------------------------------------
class ErrorMonitor:
"""Orchestrates the full production error monitoring pipeline.
On each run (typically every 15 minutes via Cloud Scheduler), it:
1. Queries Cloud Logging for recent errors across all monitored services.
2. Groups errors into patterns using normalisation + hashing.
3. Upserts patterns to Firestore, tracking new vs. recurrent.
4. Optionally uses an LLM to deduplicate patterns and create incidents.
5. Sends Discord alerts for new patterns, incidents, and spikes.
6. Auto-resolves patterns that have been silent long enough.
"""
def __init__(self) -> None:
self.logging_client = cloud_logging.Client(project=GCP_PROJECT)
self.firestore_adapter = ErrorPatternsAdapter()
self.discord = ErrorMonitorDiscord(webhook_url=self._get_discord_webhook())
def _get_discord_webhook(self) -> str:
"""Retrieve the Discord webhook URL.
Checks ``DISCORD_WEBHOOK_URL`` environment variable first; falls back
to Secret Manager.
Returns:
Discord webhook URL string.
"""
url = os.environ.get("DISCORD_WEBHOOK_URL")
if url:
return url
secret_name = get_discord_webhook_secret_name()
client = secretmanager.SecretManagerServiceClient()
secret_path = f"projects/{GCP_PROJECT}/secrets/{secret_name}/versions/latest"
response = client.access_secret_version(name=secret_path)
return response.payload.data.decode("utf-8").strip()
# ------------------------------------------------------------------
# Main entry point
# ------------------------------------------------------------------
def run(self) -> None:
"""Execute the monitor pipeline (digest or cycle mode)."""
if get_digest_mode():
self._run_daily_digest()
else:
self._run_monitor_cycle()
# ------------------------------------------------------------------
# Monitor cycle
# ------------------------------------------------------------------
def _run_monitor_cycle(self) -> None:
"""Execute one monitor cycle: collect → group → upsert → alert → auto-resolve."""
now = datetime.now(tz=timezone.utc)
now_iso = now.isoformat()
# ── Step 1: Collect log entries ────────────────────────────────────
raw_entries: list[dict] = []
for resource_type, (_, resource_names) in RESOURCE_TYPE_FILTERS.items():
log_filter = _build_log_filter(resource_type, resource_names, LOOKBACK_MINUTES)
logger.info("Querying %s with filter: %s", resource_type, log_filter)
try:
entries = self.logging_client.list_entries(
filter_=log_filter,
max_results=MAX_LOG_ENTRIES,
)
for entry in entries:
message = _extract_message(entry)
if not message:
continue
service = _extract_service_name(entry, resource_type)
ignore_reason = should_ignore(service, message)
if ignore_reason:
logger.debug(
"Ignoring entry from %s (%s): %s",
service,
ignore_reason.pattern_name,
ignore_reason.reason,
)
continue
normalized = normalize_message(message)
if not normalized:
continue
raw_entries.append(
{
"service": service,
"resource_type": _classify_resource_type(resource_type),
"normalized": normalized,
"message": message,
}
)
except Exception as exc: # noqa: BLE001
logger.error("Failed to query %s logs: %s", resource_type, exc)
logger.info("Collected %d log entries after filtering", len(raw_entries))
# ── Step 2: Group by pattern ───────────────────────────────────────
groups = _group_by_pattern(raw_entries)
logger.info("Grouped into %d distinct patterns", len(groups))
# ── Step 3: Upsert to Firestore ────────────────────────────────────
new_pattern_ids: list[str] = []
spike_patterns: list[dict] = []
for pattern_id, group in groups.items():
data = PatternData(
pattern_id=pattern_id,
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
timestamp=now,
)
result: UpsertResult = self.firestore_adapter.upsert_pattern(data)
if result.is_new:
new_pattern_ids.append(pattern_id)
logger.info("New pattern detected: %s (%s)", pattern_id, group["service"])
# Check for spike on existing patterns
existing = self.firestore_adapter.get_pattern(pattern_id)
if existing and not result.is_new:
rolling = existing.get("rolling_counts") or []
if _is_spike(group["count"], rolling):
spike_patterns.append(
{
"service": group["service"],
"normalized_message": group["normalized"],
"current_count": group["count"],
"rolling_average": _rolling_average(rolling),
}
)
logger.info("Spike detected for pattern %s", pattern_id)
# ── Step 3b: Pick up out-of-band patterns (e.g. frontend crashes) ──
# Patterns written via POST /api/client-errors land directly in
# Firestore without going through the log-scraping pipeline, so they
# won't be in new_pattern_ids / groups from Steps 1-3. Ensure any
# pattern with status="new" and alerted_at=None still gets a Discord
# alert on this cycle.
try:
unalerted = self.firestore_adapter.get_unalerted_new_patterns()
except Exception as exc: # noqa: BLE001
logger.error("Failed to query unalerted patterns: %s", exc)
unalerted = []
for pattern in unalerted:
pid = pattern.get("pattern_id")
if not pid or pid in groups:
continue
groups[pid] = {
"service": pattern.get("service", "unknown"),
"resource_type": pattern.get("resource_type", "unknown"),
"normalized": pattern.get("normalized_message", ""),
"sample_message": pattern.get("sample_message", ""),
"count": pattern.get("total_count", 1),
}
if pid not in new_pattern_ids:
new_pattern_ids.append(pid)
logger.info(
"Picked up out-of-band pattern %s (%s)",
pid,
pattern.get("service", "unknown"),
)
# If nothing to alert on this cycle, go straight to auto-resolve.
if not new_pattern_ids and not spike_patterns:
logger.info("No new or unalerted patterns this cycle; checking auto-resolve.")
self._check_auto_resolve()
return
# ── Step 4: LLM dedup ─────────────────────────────────────────────
if new_pattern_ids and get_llm_enabled():
new_patterns_data = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
]
existing_patterns = self.firestore_adapter.get_active_patterns()
# Exclude newly added patterns from the canonical list
canonical = [p for p in existing_patterns if p.get("pattern_id") not in new_pattern_ids]
if canonical:
duplicate_groups = find_duplicate_patterns(new_patterns_data, canonical)
for dup_group in duplicate_groups:
# Merge each duplicate new pattern into the canonical
canonical_pattern = canonical[dup_group.canonical_index]
canonical_id = canonical_pattern.get("pattern_id", "")
for dup_idx in dup_group.duplicate_indices:
if dup_idx < len(new_pattern_ids):
dup_id = new_pattern_ids[dup_idx]
logger.info(
"Merging duplicate pattern %s into %s: %s",
dup_id,
canonical_id,
dup_group.reason,
)
self.firestore_adapter.merge_pattern(dup_id, canonical_id, dup_group.reason)
# Remove from new_pattern_ids since it was merged
new_pattern_ids = [pid for pid in new_pattern_ids if pid != dup_id]
# ── Step 5: LLM incident analysis ─────────────────────────────────
incident_pattern_ids: set[str] = set()
if len(new_pattern_ids) >= 2 and get_llm_enabled():
patterns_for_analysis = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in new_pattern_ids
if pid in groups
]
analysis = analyze_patterns(patterns_for_analysis)
if analysis and analysis.incidents:
for incident in analysis.incidents:
incident_pids = [
new_pattern_ids[idx]
for idx in incident.pattern_indices
if idx < len(new_pattern_ids)
]
if not incident_pids:
continue
incident_id = self.firestore_adapter.create_incident(
title=incident.title,
root_cause=incident.root_cause or "",
severity=incident.severity,
suggested_fix=incident.suggested_fix or "",
primary_service=incident.primary_service,
pattern_ids=incident_pids,
used_llm=analysis.used_llm,
)
logger.info(
"Created incident %s (%s) with %d patterns",
incident_id,
incident.severity,
len(incident_pids),
)
# Build pattern list for the Discord alert
pattern_dicts = [
{
"service": groups[pid]["service"],
"normalized_message": groups[pid]["normalized"],
"count": groups[pid]["count"],
}
for pid in incident_pids
if pid in groups
]
msg = format_incident_alert(
title=incident.title,
severity=incident.severity,
root_cause=incident.root_cause or "Unknown",
suggested_fix=incident.suggested_fix or "Investigate logs",
primary_service=incident.primary_service,
patterns=pattern_dicts,
)
if self.discord.send_message(msg):
for pid in incident_pids:
try:
self.firestore_adapter.update_pattern_alerted(pid)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pid, exc)
incident_pattern_ids.update(incident_pids)
# ── Step 6: Individual alerts for new patterns not in incidents ────
for pattern_id in new_pattern_ids:
if pattern_id in incident_pattern_ids:
continue
if pattern_id not in groups:
continue
group = groups[pattern_id]
# Fetch fresh copy to get first_seen timestamp
existing = self.firestore_adapter.get_pattern(pattern_id)
first_seen = existing.get("first_seen", now_iso) if existing else now_iso
msg = format_new_pattern_alert(
service=group["service"],
resource_type=group["resource_type"],
normalized_message=group["normalized"],
sample_message=group["sample_message"],
count=group["count"],
first_seen=first_seen,
)
if self.discord.send_message(msg):
try:
self.firestore_adapter.update_pattern_alerted(pattern_id)
except Exception as exc: # noqa: BLE001
logger.warning("Could not update alerted_at for %s: %s", pattern_id, exc)
# ── Step 7: Spike alerts ───────────────────────────────────────────
for spike in spike_patterns:
msg = format_spike_alert(
service=spike["service"],
normalized_message=spike["normalized_message"],
current_count=spike["current_count"],
rolling_average=spike["rolling_average"],
)
self.discord.send_message(msg)
# ── Step 8: Auto-resolve check ─────────────────────────────────────
self._check_auto_resolve()
# ------------------------------------------------------------------
# Auto-resolve helper
# ------------------------------------------------------------------
def _check_auto_resolve(self) -> None:
"""Check all active patterns for auto-resolution and send a consolidated alert."""
patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved: list[dict] = []
for pattern in patterns:
pattern_id = pattern.get("pattern_id", "")
hours_silent = self.firestore_adapter.check_auto_resolve(pattern)
if hours_silent is not None:
logger.info(
"Auto-resolving pattern %s (silent %.1f hours)", pattern_id, hours_silent
)
self.firestore_adapter.auto_resolve_pattern(pattern_id, round(hours_silent, 1))
resolved.append(
{
"service": pattern.get("service", "unknown"),
"message": pattern.get("normalized_message", ""),
"hours_silent": round(hours_silent, 1),
}
)
if resolved:
msg = format_auto_resolved_alert(resolved)
self.discord.send_message(msg)
# ------------------------------------------------------------------
# Daily digest
# ------------------------------------------------------------------
def _run_daily_digest(self) -> None:
"""Gather 24h stats and send a daily digest to Discord."""
# Optional dedup sweep (best-effort; LLM errors are non-fatal)
if get_llm_enabled():
try:
active = self.firestore_adapter.get_active_patterns()
if len(active) >= 2:
mid = len(active) // 2
duplicates = find_duplicate_patterns(active[:mid], active[mid:])
for dup in duplicates:
for idx in dup.duplicate_indices:
target_half = active[mid:]
source_half = active[:mid]
if idx < len(target_half) and dup.canonical_index < len(source_half):
self.firestore_adapter.merge_pattern(
target_half[idx]["pattern_id"],
source_half[dup.canonical_index]["pattern_id"],
dup.reason,
)
except Exception as exc: # noqa: BLE001
logger.warning("Digest dedup sweep failed (non-fatal): %s", exc)
# Gather stats from active patterns
active_patterns = self.firestore_adapter.get_active_patterns()
per_service_counts: dict[str, int] = {}
new_count = 0
total_errors = 0
for pattern in active_patterns:
service = pattern.get("service", "unknown")
count = pattern.get("total_count", 0)
per_service_counts[service] = per_service_counts.get(service, 0) + count
total_errors += count
if pattern.get("status") == "new":
new_count += 1
# Count recently auto-resolved patterns
resolved_patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
resolved_count = sum(
1
for p in resolved_patterns
if self.firestore_adapter.check_auto_resolve(p) is not None
)
msg = format_daily_digest(
total_errors=total_errors,
new_patterns=new_count,
resolved_patterns=resolved_count,
active_patterns=len(active_patterns),
per_service_counts=per_service_counts,
)
self.discord.send_message(msg)
# ---------------------------------------------------------------------------
# Entry point
# ---------------------------------------------------------------------------
def main() -> None:
"""Configure logging and run the error monitor."""
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
)
monitor = ErrorMonitor()
monitor.run()
if __name__ == "__main__":
main()
@@ -1,137 +0,0 @@
"""Message normalizer for the production error monitoring service.
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
with stable placeholders so the same logical error always produces the same
pattern hash, regardless of which specific resource or timestamp was involved.
Normalizer rules are applied in order — more specific patterns first.
"""
import hashlib
import re
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
# ---------------------------------------------------------------------------
# Compiled regex patterns (applied in order — more specific first)
# ---------------------------------------------------------------------------
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
# Must come before Firestore and URL normalization.
(
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
r"gs://<BUCKET>/<PATH>",
),
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
# a character class that excludes '<' to avoid re-matching placeholders.
(
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
r"gs://<BUCKET>",
),
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
(
re.compile(
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
),
r"<DOC_PATH>",
),
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
(
re.compile(
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
),
"<ID>",
),
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
(
re.compile(
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
),
"<TS>",
),
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
(
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
"<EPOCH>",
),
# 6. Email addresses
(
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
"<EMAIL>",
),
# 7. IP addresses (IPv4) — must come before large-number replacement
(
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
"<IP>",
),
# 8. URLs — strip to domain only (https://host/path → https://host)
# Handles http and https; preserves the scheme+domain for context.
(
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
r"https://\1",
),
# 9. Job IDs in paths (/jobs/<alphanumeric>)
(
re.compile(r"/jobs/[A-Za-z0-9]+"),
"/jobs/<ID>",
),
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
# Must contain both letters and digits (to avoid replacing plain words).
(
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
"<UID>",
),
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
# Uses a lookahead to require at least one digit AND one letter.
(
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
"<ID>",
),
# 12. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
(
re.compile(r"\b\d{4,}\b"),
"<NUM>",
),
]
def normalize_message(message: str) -> str:
"""Normalize a raw log message by replacing variable parts with placeholders.
Applies each normalizer rule in order (more specific patterns first) and
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
Args:
message: Raw log message string.
Returns:
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
"""
if not message:
return ""
result = message
for pattern, replacement in _NORMALIZERS:
result = pattern.sub(replacement, result)
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
def compute_pattern_hash(service: str, normalized_message: str) -> str:
"""Compute a stable SHA-256 fingerprint for an error pattern.
Args:
service: The originating service name (e.g. "karaoke-backend").
normalized_message: The output of normalize_message().
Returns:
64-character lowercase hex SHA-256 digest.
"""
payload = f"{service}:{normalized_message}"
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
+2 -566
View File
@@ -10,8 +10,6 @@ from google.cloud.firestore_v1 import FieldFilter
from backend.config import settings
from backend.models.job import Job, JobStatus, TimelineEvent
from backend.models.worker_log import WorkerLogEntry
from backend.models.review_session import ReviewSession
from backend.models.audio_edit_session import AudioEditSession
logger = logging.getLogger(__name__)
@@ -77,20 +75,18 @@ class FirestoreService:
status: JobStatus,
progress: Optional[int] = None,
message: Optional[str] = None,
timeline_metadata: Optional[Dict[str, Any]] = None,
**additional_fields
) -> None:
"""Update job status and add timeline event."""
try:
doc_ref = self.db.collection(self.collection).document(job_id)
# Create timeline event
timeline_event = TimelineEvent(
status=status.value,
timestamp=datetime.utcnow().isoformat(),
progress=progress,
message=message,
metadata=timeline_metadata,
message=message
)
# Prepare updates
@@ -120,7 +116,6 @@ class FirestoreService:
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100
) -> List[Job]:
"""
@@ -133,7 +128,6 @@ class FirestoreService:
created_after: Filter jobs created after this datetime
created_before: Filter jobs created before this datetime
user_email: Filter by user_email (owner of the job)
tenant_id: Filter by tenant_id (white-label portal scoping)
limit: Maximum number of jobs to return
Returns:
@@ -156,11 +150,6 @@ class FirestoreService:
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Filter by tenant_id (white-label portal scoping)
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
# Date range filters
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
@@ -178,86 +167,6 @@ class FirestoreService:
logger.error(f"Error listing jobs: {e}")
raise
# Fields needed by the dashboard summary view
SUMMARY_FIELD_PATHS = [
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email', 'is_private',
'url', 'filename', 'audio_search_artist', 'audio_search_title',
'audio_source_type', 'source_name', 'source_id', 'target_file', 'download_url',
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
'state_data.audio_progress', 'state_data.lyrics_progress',
'state_data.audio_complete', 'state_data.lyrics_complete',
'state_data.backing_vocals_analysis', 'state_data.visibility_change_in_progress',
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
'processing_metadata',
]
def list_jobs_summary(
self,
status: Optional[JobStatus] = None,
exclude_statuses: Optional[List[str]] = None,
environment: Optional[str] = None,
client_id: Optional[str] = None,
created_after: Optional[datetime] = None,
created_before: Optional[datetime] = None,
user_email: Optional[str] = None,
tenant_id: Optional[str] = None,
limit: int = 100,
) -> List[Dict[str, Any]]:
"""
List jobs with Firestore field projection for reduced payload.
Uses ``query.select()`` so Firestore only reads the requested fields,
drastically cutting bandwidth and deserialization cost.
Returns raw dicts (not Job models) because projected documents would
fail Pydantic validation for missing required fields.
"""
try:
query = self.db.collection(self.collection)
if status:
query = query.where(filter=FieldFilter('status', '==', status.value))
# NOTE: exclude_statuses is filtered in Python after the query
# because Firestore's not-in filter combined with order_by(created_at)
# requires a composite index. Since documents are tiny after select(),
# Python filtering is fast and avoids the index requirement.
if environment:
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
if client_id:
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
if user_email:
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
if tenant_id is not None:
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
if created_after:
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
if created_before:
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
query = query.select(self.SUMMARY_FIELD_PATHS)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
# Apply exclude_statuses filter in Python (see note above)
if exclude_statuses:
results = [r for r in results if r.get('status') not in exclude_statuses]
return results
except Exception as e:
logger.error(f"Error listing jobs summary: {e}")
raise
def delete_jobs_by_filter(
self,
environment: Optional[str] = None,
@@ -601,476 +510,3 @@ class FirestoreService:
logger.error(f"Error listing tokens: {e}")
return []
# ============================================
# Search Session Methods
# ============================================
def create_search_session(self, session_data: dict) -> str:
"""Store a search session for the guided job creation flow.
Sessions are short-lived (TTL enforced via Firestore TTL policy) and
represent a completed audio search that hasn't yet been converted into
a job. The document is consumed (deleted) when create-from-search runs.
Args:
session_data: Dict with session_id, user_email, tenant_id, artist,
title, results, remote_search_id, ttl_expiry.
Returns:
session_id from session_data.
"""
try:
session_id = session_data['session_id']
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.set(session_data)
logger.info(f"Created search session {session_id}")
return session_id
except Exception as e:
logger.error(f"Error creating search session: {e}")
raise
def get_search_session(self, session_id: str) -> Optional[dict]:
"""Retrieve a search session by ID.
Returns None only if the document does not exist.
Raises on Firestore errors so callers can distinguish a missing session
from a transient datastore failure.
TTL-expired documents may still appear briefly; callers should check
ttl_expiry and treat expired sessions as missing.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
doc = doc_ref.get()
if not doc.exists:
return None
return doc.to_dict()
def consume_search_session(self, session_id: str) -> Optional[dict]:
"""Atomically read and delete a search session.
Uses a Firestore transaction to prevent two concurrent job-creation
requests from both reading the same session and creating duplicate jobs.
Returns the session dict if it existed, None if it did not exist.
Raises on Firestore errors.
Args:
session_id: UUID string identifying the session.
Returns:
Session dict or None.
"""
doc_ref = self.db.collection('search_sessions').document(session_id)
@firestore.transactional
def _read_and_delete(transaction, ref):
doc = ref.get(transaction=transaction)
if not doc.exists:
return None
session_data = doc.to_dict()
transaction.delete(ref)
return session_data
transaction = self.db.transaction()
return _read_and_delete(transaction, doc_ref)
# ============================================
# Review Session Subcollection Methods
# ============================================
# These methods store review session snapshots in a subcollection
# (jobs/{job_id}/review_sessions) for lyrics review backup/restore.
def save_review_session(self, job_id: str, session: ReviewSession) -> str:
"""
Save a review session to the subcollection.
Stores at: jobs/{job_id}/review_sessions/{session_id}
Args:
job_id: Job ID
session: ReviewSession instance
Returns:
session_id
"""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved review session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving review session for job {job_id}: {e}")
raise
def list_review_sessions(self, job_id: str) -> List[ReviewSession]:
"""
List all review sessions for a job, ordered by updated_at descending.
Args:
job_id: Job ID
Returns:
List of ReviewSession instances (metadata only, no correction_data)
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [ReviewSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing review sessions for job {job_id}: {e}")
return []
def get_review_session(self, job_id: str, session_id: str) -> Optional[ReviewSession]:
"""
Get a single review session by ID.
Args:
job_id: Job ID
session_id: Session ID
Returns:
ReviewSession or None
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return ReviewSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_session(self, job_id: str, session_id: str) -> None:
"""
Delete a review session.
Args:
job_id: Job ID
session_id: Session ID
"""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("review_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted review session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting review session {session_id} for job {job_id}: {e}")
raise
def delete_review_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""
Delete all review sessions for a job (used when deleting a job).
Args:
job_id: Job ID
batch_size: Number of documents to delete per batch
Returns:
Number of sessions deleted
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} review sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting review sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_review_session_hash(self, job_id: str) -> Optional[str]:
"""
Get the data_hash of the most recent review session for deduplication.
Args:
job_id: Job ID
Returns:
data_hash string or None if no sessions exist
"""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest review session hash for job {job_id}: {e}")
return None
def search_review_sessions(
self,
query_text: str = "",
job_id: Optional[str] = None,
limit: int = 20,
) -> List[Dict[str, Any]]:
"""
Search review sessions across all jobs using collection group query.
Args:
query_text: Search text (prefix-matched against artist_lower or title_lower)
job_id: If provided, search only within this job
limit: Maximum results
Returns:
List of session dicts with job context
"""
try:
if job_id:
# Search within a specific job
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
# Collection group query across all jobs
sessions_ref = self.db.collection_group("review_sessions")
if query_text:
# Over-fetch since we filter in Python (Firestore doesn't support LIKE/contains)
fetch_limit = max(limit * 25, 500)
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(fetch_limit)
docs = query.stream()
results = [doc.to_dict() for doc in docs]
q_lower = query_text.lower()
results = [
r for r in results
if q_lower in (r.get("artist") or "").lower()
or q_lower in (r.get("title") or "").lower()
or q_lower in (r.get("job_id") or "").lower()
]
return results[:limit]
else:
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
docs = query.stream()
return [doc.to_dict() for doc in docs]
except Exception as e:
logger.error(f"Error searching review sessions: {e}")
return []
# ---- Audio Edit Sessions ----
def save_audio_edit_session(self, job_id: str, session: AudioEditSession) -> str:
"""Save an audio edit session to the subcollection."""
try:
session.job_id = job_id
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
doc_ref = sessions_ref.document(session.session_id)
doc_ref.set(session.to_dict())
logger.info(f"Saved audio edit session {session.session_id} for job {job_id}")
return session.session_id
except Exception as e:
logger.error(f"Error saving audio edit session for job {job_id}: {e}")
raise
def list_audio_edit_sessions(self, job_id: str) -> List[AudioEditSession]:
"""List all audio edit sessions for a job, ordered by updated_at descending."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
docs = query.stream()
return [AudioEditSession.from_dict(doc.to_dict()) for doc in docs]
except Exception as e:
logger.error(f"Error listing audio edit sessions for job {job_id}: {e}")
return []
def get_audio_edit_session(self, job_id: str, session_id: str) -> Optional[AudioEditSession]:
"""Get a single audio edit session by ID."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc = doc_ref.get()
if not doc.exists:
return None
return AudioEditSession.from_dict(doc.to_dict())
except Exception as e:
logger.error(f"Error getting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_session(self, job_id: str, session_id: str) -> None:
"""Delete a single audio edit session."""
try:
doc_ref = (
self.db.collection(self.collection)
.document(job_id)
.collection("audio_edit_sessions")
.document(session_id)
)
doc_ref.delete()
logger.info(f"Deleted audio edit session {session_id} for job {job_id}")
except Exception as e:
logger.error(f"Error deleting audio edit session {session_id} for job {job_id}: {e}")
raise
def delete_audio_edit_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
"""Delete all audio edit sessions for a job (used when deleting a job)."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
deleted_count = 0
while True:
docs = sessions_ref.limit(batch_size).stream()
deleted_in_batch = 0
batch = self.db.batch()
for doc in docs:
batch.delete(doc.reference)
deleted_in_batch += 1
if deleted_in_batch == 0:
break
batch.commit()
deleted_count += deleted_in_batch
if deleted_in_batch < batch_size:
break
if deleted_count > 0:
logger.info(f"Deleted {deleted_count} audio edit sessions for job {job_id}")
return deleted_count
except Exception as e:
logger.error(f"Error deleting audio edit sessions subcollection for job {job_id}: {e}")
return 0
def get_latest_audio_edit_session_hash(self, job_id: str) -> Optional[str]:
"""Get the data_hash of the most recent audio edit session for deduplication."""
try:
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
query = (
sessions_ref
.order_by("updated_at", direction=firestore.Query.DESCENDING)
.limit(1)
.select(["data_hash"])
)
docs = list(query.stream())
if docs:
return docs[0].to_dict().get("data_hash", "")
return None
except Exception as e:
logger.error(f"Error getting latest audio edit session hash for job {job_id}: {e}")
return None
def delete_search_session(self, session_id: str) -> None:
"""Delete a search session after it has been consumed by job creation.
Errors are logged but not re-raised — a missing or already-deleted
session is a no-op.
Args:
session_id: UUID string identifying the session.
"""
try:
doc_ref = self.db.collection('search_sessions').document(session_id)
doc_ref.delete()
logger.debug(f"Deleted search session {session_id}")
except Exception as e:
logger.error(f"Error deleting search session {session_id}: {e}")
# Singleton client instance
_firestore_client: Optional[firestore.Client] = None
def get_firestore_client() -> firestore.Client:
"""
Get a shared Firestore client instance.
This returns a raw Firestore client (not the FirestoreService) for use
in services that need direct Firestore access without the job-specific
abstractions provided by FirestoreService.
Returns:
Firestore client instance
"""
global _firestore_client
if _firestore_client is None:
_firestore_client = firestore.Client(project=settings.google_cloud_project)
return _firestore_client
_log_service: Optional["FirestoreService"] = None
def _get_log_service() -> "FirestoreService":
"""Return a cached FirestoreService instance for log_to_job."""
global _log_service
if _log_service is None:
_log_service = FirestoreService()
return _log_service
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Write a log entry to a job's Firestore log subcollection.
This is a convenience function for API endpoints that need to log
to a job's log history (workers already have JobLogAdapter for this).
Args:
job_id: Job ID to log to
worker: Worker/source name (e.g., "api", "edit", "admin")
level: Log level (INFO, WARNING, ERROR)
message: Log message
metadata: Optional structured metadata dict
"""
try:
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
_get_log_service().append_log_to_subcollection(job_id, entry)
except Exception as e:
logger.debug(f"Failed to write job log for {job_id}: {e}")
+84 -283
View File
@@ -34,8 +34,6 @@ from typing import Any, Dict, List, Optional
import httpx
from backend.config import get_settings
logger = logging.getLogger(__name__)
@@ -44,104 +42,6 @@ class FlacfetchServiceError(Exception):
pass
def _format_httpx_error(e: Exception) -> str:
"""
Format an httpx exception into a descriptive error string.
httpx.RequestError subclasses (ConnectError, ReadError, etc.) often have
empty str() representations, which produces useless messages like
"Status request failed: ". This helper always returns something meaningful.
"""
error_str = str(e)
error_type = type(e).__name__
if error_str:
return f"{error_type}: {error_str}"
# str(e) is empty - try to include request URL for context
if isinstance(e, httpx.RequestError):
try:
request = e.request
if request:
return f"{error_type} for {request.method} {request.url}"
except RuntimeError:
pass # .request not set
return error_type
def should_retry_flacfetch_error(exception: Exception) -> bool:
"""
Determine if a flacfetch exception should be retried.
Retry on:
- httpx.RequestError (network failures: timeout, connection refused, etc.)
- httpx.HTTPStatusError with 5xx status (server errors)
Don't retry on:
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
"""
if isinstance(exception, httpx.RequestError):
# Network-level failures - always retry
return True
if isinstance(exception, httpx.HTTPStatusError):
# HTTP errors - only retry on 5xx (server errors)
return exception.response.status_code >= 500
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
return False
async def with_retry(func, *args, **kwargs):
"""
Execute an async function with retry logic using exponential backoff.
This implementation uses manual retry logic instead of tenacity decorators
for better testability and runtime configuration.
"""
settings = get_settings()
max_attempts = settings.flacfetch_retry_max_attempts
min_wait = settings.flacfetch_retry_min_wait
max_wait = settings.flacfetch_retry_max_wait
last_exception = None
for attempt in range(1, max_attempts + 1):
try:
return await func(*args, **kwargs)
except Exception as e:
last_exception = e
# Check if we should retry this exception
if not should_retry_flacfetch_error(e):
# Don't retry - raise immediately
raise
# Check if we have more attempts left
if attempt >= max_attempts:
# No more attempts - raise the last exception
logger.warning(
f"Flacfetch request failed after {max_attempts} attempts: "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}"
)
raise
# Calculate wait time with exponential backoff
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
logger.warning(
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): "
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}. "
f"Retrying in {wait_time:.1f}s..."
)
# Wait before next attempt
await asyncio.sleep(wait_time)
# Should never reach here, but just in case
raise last_exception
class FlacfetchClient:
"""
Client for remote flacfetch HTTP API.
@@ -200,7 +100,7 @@ class FlacfetchClient:
return data
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
@@ -245,10 +145,10 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Search request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Search request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
async def download(
self,
search_id: str,
@@ -258,59 +158,47 @@ class FlacfetchClient:
) -> str:
"""
Start downloading an audio file.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
search_id: Search ID from previous search
result_index: Index of result to download
output_filename: Optional custom filename (without extension)
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
Returns:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure (after all retries exhausted)
FlacfetchServiceError: On download start failure
"""
try:
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Download request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
async def _download_impl(
self,
search_id: str,
result_index: int,
output_filename: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"search_id": search_id,
"result_index": result_index,
}
if output_filename:
payload["output_filename"] = output_filename
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def download_by_id(
self,
source_name: str,
@@ -326,9 +214,6 @@ class FlacfetchClient:
This is useful when you have stored the source_id from a previous search
and want to download later without re-searching.
Retries automatically on network errors and server errors (5xx) using
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
Args:
source_name: Provider name (RED, OPS, YouTube, Spotify)
source_id: Source-specific ID (torrent ID, video ID, track ID)
@@ -341,68 +226,52 @@ class FlacfetchClient:
Download ID for tracking progress
Raises:
FlacfetchServiceError: On download start failure (after all retries exhausted)
FlacfetchServiceError: On download start failure
"""
try:
return await with_retry(
self._download_by_id_impl,
source_name, source_id, output_filename, target_file, download_url, gcs_path
)
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Download by ID request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
async def _download_by_id_impl(
self,
source_name: str,
source_id: str,
output_filename: Optional[str] = None,
target_file: Optional[str] = None,
download_url: Optional[str] = None,
gcs_path: Optional[str] = None,
) -> str:
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
async with httpx.AsyncClient() as client:
payload = {
"source_name": source_name,
"source_id": source_id,
}
if output_filename:
payload["output_filename"] = output_filename
if target_file:
payload["target_file"] = target_file
if download_url:
payload["download_url"] = download_url
if gcs_path:
payload["upload_to_gcs"] = True
payload["gcs_path"] = gcs_path
resp = await client.post(
f"{self.base_url}/download-by-id",
headers=self._headers(),
json=payload,
timeout=self.timeout,
)
resp.raise_for_status()
data = resp.json()
return data["download_id"]
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
"""
Get the current status of a download.
Args:
download_id: Download ID from start_download
Returns:
Status dict with progress, speed, path info
Raises:
httpx.RequestError: On network-level failures (connection refused, timeout, etc.)
These are NOT wrapped so wait_for_download can handle retries.
FlacfetchServiceError: On application-level failures (404, other HTTP errors)
FlacfetchServiceError: On status check failure
"""
try:
async with httpx.AsyncClient() as client:
@@ -413,10 +282,9 @@ class FlacfetchClient:
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError:
# Let network errors bubble up so wait_for_download can handle retries
raise
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Status request failed: {e}")
except httpx.HTTPStatusError as e:
if e.response.status_code == 404:
raise FlacfetchServiceError(f"Download not found: {download_id}")
@@ -428,59 +296,35 @@ class FlacfetchClient:
timeout: int = 600,
poll_interval: float = 2.0,
progress_callback: Optional[callable] = None,
max_consecutive_errors: int = 3,
) -> Dict[str, Any]:
"""
Wait for a download to complete.
Tolerates transient network errors during status polling. A single
connection hiccup won't fail the entire download -- only N consecutive
poll failures will.
Args:
download_id: Download ID to wait for
timeout: Maximum wait time in seconds
poll_interval: Time between status checks
progress_callback: Optional callback(status_dict) for progress updates
max_consecutive_errors: Number of consecutive poll failures before giving up
Returns:
Final status dict with output_path or gcs_path
Raises:
FlacfetchServiceError: On download failure, timeout, or too many consecutive poll errors
FlacfetchServiceError: On download failure or timeout
"""
elapsed = 0.0
consecutive_errors = 0
while elapsed < timeout:
try:
status = await self.get_download_status(download_id)
consecutive_errors = 0 # Reset on success
except httpx.RequestError as e:
consecutive_errors += 1
error_desc = _format_httpx_error(e)
logger.warning(
f"Status poll failed for download {download_id} "
f"({consecutive_errors}/{max_consecutive_errors}): {error_desc}"
)
if consecutive_errors >= max_consecutive_errors:
raise FlacfetchServiceError(
f"Status polling failed after {consecutive_errors} consecutive "
f"network errors (last: {error_desc})"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
continue
status = await self.get_download_status(download_id)
if progress_callback:
try:
progress_callback(status)
except Exception as e:
logger.warning(f"Progress callback error: {e}")
download_status = status.get("status")
if download_status in ["complete", "seeding"]:
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
return status
@@ -489,7 +333,7 @@ class FlacfetchClient:
raise FlacfetchServiceError(f"Download failed: {error}")
elif download_status == "cancelled":
raise FlacfetchServiceError("Download was cancelled")
# Log progress
progress = status.get("progress", 0)
speed = status.get("download_speed_kbps", 0)
@@ -498,10 +342,10 @@ class FlacfetchClient:
f"Download {download_id}: {progress:.1f}% "
f"({speed:.1f} KB/s, {peers} peers)"
)
await asyncio.sleep(poll_interval)
elapsed += poll_interval
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
async def list_torrents(self) -> Dict[str, Any]:
@@ -522,7 +366,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"List torrents failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"List torrents failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
@@ -553,7 +397,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
@@ -587,7 +431,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Cleanup failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Cleanup failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
@@ -628,7 +472,7 @@ class FlacfetchClient:
return deleted
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear search cache request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
@@ -658,55 +502,12 @@ class FlacfetchClient:
return deleted_count
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Clear all cache request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# YouTube Availability Check
# =========================================================================
async def check_youtube(self, url: str) -> Dict[str, Any]:
"""
Check if a YouTube video is available for download from this server's location.
Calls the flacfetch /check-youtube endpoint which uses yt-dlp to detect
geo-restrictions, private/removed videos, and other unavailability reasons.
Args:
url: YouTube video URL or video ID
Returns:
Dict with keys: available, video_id, title, error,
is_geo_restricted, is_age_restricted, is_private, is_removed
Raises:
FlacfetchServiceError: On request failure
"""
try:
async with httpx.AsyncClient() as client:
resp = await client.post(
f"{self.base_url}/check-youtube",
headers=self._headers(),
json={"url": url},
timeout=15,
)
resp.raise_for_status()
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"YouTube availability check failed: {_format_httpx_error(e)}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"YouTube availability check failed: {e.response.status_code} - {e.response.text}"
)
# =========================================================================
# Cache Management
# =========================================================================
async def get_cache_stats(self) -> Dict[str, Any]:
"""
Get statistics about the cache.
@@ -728,7 +529,7 @@ class FlacfetchClient:
return resp.json()
except httpx.RequestError as e:
raise FlacfetchServiceError(f"Get cache stats request failed: {_format_httpx_error(e)}")
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
except httpx.HTTPStatusError as e:
raise FlacfetchServiceError(
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
-141
View File
@@ -1,141 +0,0 @@
"""Service for generating personalized referral flyer PDFs."""
import logging
import os
import shutil
import subprocess
import tempfile
logger = logging.getLogger(__name__)
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "..", "templates", "printables")
CHROMIUM_PATHS = [
"/usr/bin/chromium",
"/usr/bin/chromium-browser",
"/usr/bin/google-chrome",
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
]
VALID_THEMES = {"light", "dark"}
MAX_QR_DATA_URL_BYTES = 500_000
class FlyerError(Exception):
"""Error during flyer generation."""
class FlyerService:
"""Generates personalized referral flyer PDFs."""
def __init__(self):
self._chromium_path: str | None = None
def _get_chromium(self) -> str:
"""Lazily find Chromium binary. Only needed when generating PDFs."""
if self._chromium_path:
return self._chromium_path
for path in CHROMIUM_PATHS:
if os.path.exists(path):
self._chromium_path = path
return path
for name in ["chromium", "chromium-browser", "google-chrome"]:
found = shutil.which(name)
if found:
self._chromium_path = found
return found
raise FlyerError("Chromium not found. Install chromium to generate flyer PDFs.")
def _validate_qr_data_url(self, qr_data_url: str) -> None:
if not qr_data_url.startswith("data:image/"):
raise FlyerError("Invalid QR data URL: must be a data:image/ URL")
if len(qr_data_url) > MAX_QR_DATA_URL_BYTES:
raise FlyerError(f"QR data URL too large (>{MAX_QR_DATA_URL_BYTES} bytes)")
def _render_template(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> str:
if theme not in VALID_THEMES:
raise FlyerError(f"Invalid theme: {theme}. Must be one of: {VALID_THEMES}")
suffix = "-dark" if theme == "dark" else ""
template_path = os.path.join(TEMPLATE_DIR, f"website-referral-flyer{suffix}.html")
with open(template_path) as f:
html = f.read()
html = html.replace("{{REFERRAL_CODE}}", referral_code.upper())
html = html.replace("{{REFERRAL_CODE_LOWER}}", referral_code.lower())
html = html.replace("{{DISCOUNT_PERCENT}}", str(discount_percent))
html = html.replace("{{QR_DATA_URL}}", qr_data_url)
return html
def generate_pdf(
self,
theme: str,
referral_code: str,
discount_percent: int,
qr_data_url: str,
) -> bytes:
"""Generate a personalized flyer PDF. Returns raw PDF bytes."""
self._validate_qr_data_url(qr_data_url)
html = self._render_template(theme, referral_code, discount_percent, qr_data_url)
with tempfile.NamedTemporaryFile(
suffix=".html", delete=False, mode="w", encoding="utf-8"
) as tmp_html:
tmp_html.write(html)
tmp_html_path = tmp_html.name
tmp_pdf_path = tmp_html_path.replace(".html", ".pdf")
try:
cmd = [
self._get_chromium(),
"--headless",
f"--print-to-pdf={tmp_pdf_path}",
"--no-margins",
"--print-background",
"--no-pdf-header-footer",
"--virtual-time-budget=5000",
"--paper-width=8.5",
"--paper-height=11",
"--disable-gpu",
"--no-sandbox",
tmp_html_path,
]
result = subprocess.run(
cmd, capture_output=True, text=True, timeout=30
)
if result.returncode != 0:
logger.error("Chromium PDF generation failed: %s", result.stderr)
raise FlyerError(f"PDF generation failed: {result.stderr[:200]}")
if not os.path.exists(tmp_pdf_path):
raise FlyerError("PDF generation failed: no output file created")
with open(tmp_pdf_path, "rb") as f:
return f.read()
finally:
for path in [tmp_html_path, tmp_pdf_path]:
try:
os.unlink(path)
except OSError:
pass
_flyer_service = None
def get_flyer_service() -> FlyerService:
global _flyer_service
if _flyer_service is None:
_flyer_service = FlyerService()
return _flyer_service
-72
View File
@@ -1,72 +0,0 @@
# GCE Encoding Worker
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
## Overview
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
## Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/encode` | POST | Submit a full encoding job |
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
| `/status/{job_id}` | GET | Get job status and progress |
| `/health` | GET | Health check |
## Authentication
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
## Encoding Process
### Full Encoding (`/encode`)
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
- Lossless 4K MP4
- Lossy 4K MP4
- Lossless MKV
- 720p MP4
3. Uploads results to GCS
4. Reports progress via status endpoint
### Preview Encoding (`/encode-preview`)
1. Downloads ASS subtitle and audio files from GCS
2. Runs FFmpeg to create a quick 480x270 preview video
3. Uploads result to GCS
4. Used for real-time preview in the web UI
## Dependencies
- **FFmpeg**: Static build from John Van Sickle
- **Python 3.13**: Built from source (required for karaoke-gen)
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
## Deployment
This service is deployed via the VM's startup script. The script:
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
2. Builds Python 3.13 from source
3. Creates venv and installs dependencies
4. Writes `main.py` to `/opt/encoding-worker/`
5. Creates systemd service
6. Starts the service on port 8080
## Local Development
This code is extracted from the infrastructure for maintainability. To test locally:
```bash
cd backend/services/gce_encoding
pip install -r requirements.txt
pip install ../../../ # Install karaoke-gen wheel
ENCODING_API_KEY=test python main.py
```
## Architecture Notes
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
- **GCS integration**: Direct download/upload via google-cloud-storage client
-22
View File
@@ -1,22 +0,0 @@
"""
GCE Encoding Worker Service.
This package contains the FastAPI application that runs on the GCE encoding worker VM.
It provides HTTP endpoints for video encoding jobs.
The service:
- Downloads input files from GCS
- Runs FFmpeg encoding via LocalEncodingService
- Uploads results back to GCS
- Provides job status tracking
Endpoints:
- POST /encode - Submit a full encoding job
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
- GET /status/{job_id} - Get job status
- GET /health - Health check
"""
from .main import app
__all__ = ["app"]
Loaded 100 of 5334 files, more files were not shown because too many files have changed in this diff. Show more