mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-22 05:47:55 +09:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d0a657610b |
No files matched your search
@@ -1,109 +0,0 @@
|
||||
# Plan: Admin Feedback Visibility
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1454-feedback-email-debug
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
|
||||
|
||||
1. **Admin UI page** to browse and review all submitted feedback
|
||||
2. **Email notification** to admin@nomadkaraoke.com on each new submission
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
|
||||
- [ ] Sortable by date (newest first by default) and filterable by search (email)
|
||||
- [ ] Star ratings rendered visually (not just numbers)
|
||||
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
|
||||
- [ ] Email includes all feedback content (ratings, text fields, user email)
|
||||
- [ ] Test data filtering works (exclude_test toggle)
|
||||
- [ ] Sidebar navigation includes new "Feedback" item
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Admin API Endpoint
|
||||
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
|
||||
|
||||
### Admin UI Page
|
||||
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
|
||||
|
||||
### Email Notification
|
||||
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Admin feedback list endpoint
|
||||
|
||||
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
|
||||
2. [ ] Add `GET /api/admin/feedback` endpoint with:
|
||||
- Pagination: `limit=50`, `offset=0`
|
||||
- Search by user email
|
||||
- `exclude_test` filter (reuse `is_test_email`)
|
||||
- Sort by `created_at` descending
|
||||
- Return: items, total, offset, limit, has_more
|
||||
3. [ ] Add aggregate stats in response: average ratings, total count
|
||||
|
||||
### Step 2: Backend — Email notification on submission
|
||||
|
||||
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
|
||||
- User email
|
||||
- All 4 star ratings
|
||||
- All text fields
|
||||
- Would recommend / would use again
|
||||
- Submitted timestamp
|
||||
- Link to admin feedback page
|
||||
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
|
||||
|
||||
### Step 3: Frontend — Admin API client
|
||||
|
||||
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
|
||||
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
|
||||
|
||||
### Step 4: Frontend — Admin feedback page
|
||||
|
||||
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
|
||||
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
|
||||
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
|
||||
- Search by email
|
||||
- Pagination (50 per page)
|
||||
- Refresh button
|
||||
- Test data toggle (via `useAdminSettings`)
|
||||
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
|
||||
|
||||
### Step 5: Testing
|
||||
|
||||
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
|
||||
11. [ ] Backend unit test for `send_feedback_notification` email method
|
||||
12. [ ] Frontend component test for feedback page (render, empty state)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
|
||||
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
|
||||
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
|
||||
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
|
||||
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
|
||||
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
|
||||
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
|
||||
- **Integration**: Submit feedback → verify email service called
|
||||
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
|
||||
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Backend: Remove endpoint from admin.py, revert email call in users.py
|
||||
- Frontend: Delete feedback page, remove sidebar item
|
||||
- No data migration needed — Firestore collection unchanged
|
||||
@@ -1,393 +0,0 @@
|
||||
# Plan: Admin Job Details Page
|
||||
|
||||
**Created:** 2026-01-09
|
||||
**Branch:** feat/sess-20260109-1738-admin-job-details
|
||||
**Status:** In Progress (Phase 1 Complete)
|
||||
|
||||
## Overview
|
||||
|
||||
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
|
||||
|
||||
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
|
||||
|
||||
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Must Have
|
||||
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
|
||||
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
|
||||
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
|
||||
- [ ] Download links for all existing files (generate signed URLs)
|
||||
- [ ] View worker logs with filtering by level/worker
|
||||
- [ ] Restart job from beginning (reset to PENDING)
|
||||
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
|
||||
|
||||
### Should Have
|
||||
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
|
||||
- [ ] View/edit raw JSON fields (state_data, file_urls)
|
||||
- [ ] Quick actions: send completion email, clear cache, impersonate user
|
||||
|
||||
### Nice to Have
|
||||
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
|
||||
- [ ] Diff view when editing fields
|
||||
- [ ] Audit log of admin changes
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Routing Fix (COMPLETED)
|
||||
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
|
||||
|
||||
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
|
||||
- Uses `useSearchParams()` to detect `?id=` parameter
|
||||
- Shows detail view when `selectedJobId` is present, list view otherwise
|
||||
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
|
||||
- Deleted old `/admin/jobs/detail/` directory
|
||||
|
||||
### Backend Enhancements
|
||||
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
|
||||
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
|
||||
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
|
||||
|
||||
### Frontend Structure
|
||||
The page will have collapsible sections:
|
||||
1. **Header**: Job ID, status badge, artist/title, quick actions
|
||||
2. **Overview Cards**: User, dates, progress, source info
|
||||
3. **Stage Timeline**: Visual breakdown of each stage with times
|
||||
4. **Files**: Downloadable links for all GCS files
|
||||
5. **Logs**: Filterable worker logs
|
||||
6. **Configuration**: Editable job settings
|
||||
7. **Raw Data**: JSON views of state_data, file_urls, full job
|
||||
|
||||
### Job Reset Logic
|
||||
Reset to specific states requires:
|
||||
- Clearing downstream state_data keys
|
||||
- Resetting status
|
||||
- Optionally clearing associated files
|
||||
- Adding timeline event for the reset
|
||||
|
||||
Valid reset targets:
|
||||
| Target State | Clears | Use Case |
|
||||
|--------------|--------|----------|
|
||||
| PENDING | Everything except input | Full restart |
|
||||
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
|
||||
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
|
||||
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
|
||||
|
||||
### Phase 1: Fix Routing (COMPLETED)
|
||||
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
|
||||
2. [x] Combined list and detail views in single component
|
||||
3. [x] Use `useSearchParams()` to detect `?id=` parameter
|
||||
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
|
||||
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
|
||||
6. [x] Verified: All 9 E2E tests pass, build succeeds
|
||||
|
||||
### Phase 2: Backend - File Downloads with Signed URLs
|
||||
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
|
||||
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
|
||||
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
|
||||
10. [ ] Traverse job.file_urls recursively
|
||||
11. [ ] Generate signed URLs for each GCS path
|
||||
12. [ ] Return structured response with file metadata
|
||||
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
|
||||
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
|
||||
|
||||
### Phase 3: Frontend - Enhanced File Display
|
||||
15. [ ] Create "Files" accordion section with grouped downloads
|
||||
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
|
||||
17. [ ] Show file name, type, download button
|
||||
18. [ ] Handle missing/unavailable files gracefully
|
||||
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
|
||||
20. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 4: Backend - Job Update Endpoint
|
||||
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
|
||||
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
|
||||
23. [ ] Define editable fields whitelist
|
||||
24. [ ] Validate field values (e.g., status must be valid enum)
|
||||
25. [ ] Log admin changes for audit trail
|
||||
26. [ ] Verify: Backend tests pass
|
||||
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
|
||||
|
||||
### Phase 5: Frontend - Editable Fields
|
||||
28. [ ] Create "Configuration" section with editable fields
|
||||
29. [ ] Use inline edit pattern (click to edit, enter to save)
|
||||
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
|
||||
31. [ ] Show validation errors inline
|
||||
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
|
||||
33. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 6: Backend - Job Reset Endpoint
|
||||
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
|
||||
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
|
||||
36. [ ] Accept `target_state` parameter
|
||||
37. [ ] Implement reset logic for each target state
|
||||
38. [ ] Clear appropriate state_data keys
|
||||
39. [ ] Optionally trigger workers after reset
|
||||
40. [ ] Verify: Backend tests pass
|
||||
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
|
||||
|
||||
### Phase 7: Frontend - Reset Actions
|
||||
42. [ ] Add "Reset Job" dropdown/menu in header
|
||||
43. [ ] Show available reset targets based on current state
|
||||
44. [ ] Confirm dialog explaining what will be cleared
|
||||
45. [ ] Show success/error toast after reset
|
||||
46. [ ] **E2E Test**: Add reset confirmation dialog test
|
||||
47. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 8: Enhanced Stage Timeline
|
||||
48. [ ] Parse timeline events to calculate stage durations
|
||||
49. [ ] Display stages as cards or timeline items
|
||||
50. [ ] Show duration for each stage
|
||||
51. [ ] Highlight current/blocking stages
|
||||
52. [ ] **E2E Test**: Add timeline display test
|
||||
|
||||
### Phase 9: Final Verification
|
||||
53. [ ] Run full backend test suite: `make test-backend`
|
||||
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
|
||||
55. [ ] Manual production testing checklist (see Testing Strategy)
|
||||
56. [ ] Verify CI passes on PR
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
|
||||
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
|
||||
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
|
||||
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
|
||||
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
|
||||
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Based on `docs/TESTING.md` guidelines, this feature requires:
|
||||
|
||||
### Test Types Required
|
||||
|
||||
| Type | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
|
||||
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
|
||||
| Manual Production E2E | N/A | Final validation on staging/prod |
|
||||
|
||||
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
|
||||
|
||||
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
|
||||
|
||||
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
|
||||
```python
|
||||
def test_get_job_files_returns_signed_urls_for_all_files():
|
||||
"""Test that all files in file_urls get signed URLs."""
|
||||
|
||||
def test_get_job_files_handles_nested_file_urls():
|
||||
"""Test recursive traversal of stems.instrumental_clean, etc."""
|
||||
|
||||
def test_get_job_files_handles_missing_files_gracefully():
|
||||
"""Test files that don't exist in GCS are omitted or marked."""
|
||||
|
||||
def test_get_job_files_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_get_job_files_returns_404_for_missing_job():
|
||||
"""Test invalid job_id returns 404."""
|
||||
```
|
||||
|
||||
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
|
||||
```python
|
||||
def test_update_job_allows_editable_fields():
|
||||
"""Test artist, title, user_email, theme_id can be updated."""
|
||||
|
||||
def test_update_job_rejects_non_editable_fields():
|
||||
"""Test job_id, created_at, etc. cannot be updated."""
|
||||
|
||||
def test_update_job_validates_status_enum():
|
||||
"""Test invalid status value returns 422."""
|
||||
|
||||
def test_update_job_validates_theme_id_exists():
|
||||
"""Test non-existent theme_id returns 422."""
|
||||
|
||||
def test_update_job_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_update_job_returns_updated_job():
|
||||
"""Test response includes updated job data."""
|
||||
```
|
||||
|
||||
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
|
||||
```python
|
||||
def test_reset_to_pending_clears_all_state_data():
|
||||
"""Test full reset clears everything except input."""
|
||||
|
||||
def test_reset_to_audio_selection_clears_search_results():
|
||||
"""Test audio selection reset clears audio_search_results."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_review_data():
|
||||
"""Test review reset clears screens, videos, review data."""
|
||||
|
||||
def test_reset_to_instrumental_selection_clears_finals():
|
||||
"""Test instrumental reset clears encoding results."""
|
||||
|
||||
def test_reset_adds_timeline_event():
|
||||
"""Test reset adds admin action to timeline."""
|
||||
|
||||
def test_reset_invalid_target_returns_422():
|
||||
"""Test invalid target_state returns validation error."""
|
||||
|
||||
def test_reset_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_reset_from_complete_to_pending_works():
|
||||
"""Test can reset completed job back to start."""
|
||||
```
|
||||
|
||||
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
|
||||
|
||||
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
|
||||
|
||||
test.describe('Admin Job Detail Page', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await setAdminAuthToken(page);
|
||||
});
|
||||
|
||||
test('navigates from jobs list to detail page', async ({ page }) => {
|
||||
// Mock jobs list and single job endpoints
|
||||
await page.route('**/api/admin/jobs', route => route.fulfill({
|
||||
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
|
||||
}));
|
||||
await page.route('**/api/jobs/test123', route => route.fulfill({
|
||||
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
|
||||
}));
|
||||
|
||||
await page.goto('/admin/jobs');
|
||||
await page.click('text=test123');
|
||||
await expect(page).toHaveURL('/admin/jobs/test123');
|
||||
await expect(page.getByText('Job test123')).toBeVisible();
|
||||
});
|
||||
|
||||
test('displays all job metadata sections', async ({ page }) => {
|
||||
// Test overview cards, timeline, logs accordion, etc.
|
||||
});
|
||||
|
||||
test('files section shows downloadable links', async ({ page }) => {
|
||||
// Mock files endpoint, verify download buttons appear
|
||||
});
|
||||
|
||||
test('edit field inline and save', async ({ page }) => {
|
||||
// Click edit, change value, press Enter, verify PATCH called
|
||||
});
|
||||
|
||||
test('reset job shows confirmation dialog', async ({ page }) => {
|
||||
// Click reset, verify dialog, confirm, verify POST called
|
||||
});
|
||||
|
||||
test('handles API errors gracefully', async ({ page }) => {
|
||||
// Mock 500 response, verify error toast shown
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test File Organization
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── test_admin_job_detail.py # NEW - all backend tests for this feature
|
||||
├── test_admin_api.py # Existing admin tests (keep separate)
|
||||
└── conftest.py # Shared fixtures
|
||||
|
||||
frontend/e2e/
|
||||
├── fixtures/
|
||||
│ ├── test-helper.ts # Add setAdminAuthToken helper
|
||||
│ └── mock-responses.ts # Add admin job detail mocks
|
||||
└── regression/
|
||||
└── admin-job-detail.spec.ts # NEW - E2E regression tests
|
||||
```
|
||||
|
||||
### Mocking Guidelines
|
||||
|
||||
**Backend - What to mock:**
|
||||
- `StorageService.generate_signed_url()` - return predictable test URLs
|
||||
- `StorageService.file_exists()` - control which files "exist"
|
||||
- Firestore operations via existing `mock_firestore` fixture
|
||||
|
||||
**Backend - What NOT to mock:**
|
||||
- `JobManager` logic - test the real reset/update logic
|
||||
- Validation logic - test real Pydantic validation
|
||||
- Authorization checks - test real `require_admin` dependency
|
||||
|
||||
**Frontend - What to mock:**
|
||||
- All API responses via Playwright route interception
|
||||
- Time (if testing token expiry)
|
||||
|
||||
**Frontend - What NOT to mock:**
|
||||
- React component behavior
|
||||
- User interactions (click, type, etc.)
|
||||
|
||||
### CI Requirements
|
||||
|
||||
This feature will be validated by existing CI checks:
|
||||
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
|
||||
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
|
||||
- `frontend-build` - Must pass (verifies new page compiles)
|
||||
|
||||
### Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Backend tests only (for this feature)
|
||||
poetry run pytest backend/tests/test_admin_job_detail.py -v
|
||||
|
||||
# All backend tests
|
||||
make test-backend
|
||||
|
||||
# Frontend E2E regression (includes new tests)
|
||||
cd frontend && npm run test:e2e
|
||||
|
||||
# Single E2E test file
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
|
||||
|
||||
# With browser visible for debugging
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
|
||||
```
|
||||
|
||||
### Manual Production Testing Checklist
|
||||
|
||||
After deployment, manually verify on production:
|
||||
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
|
||||
- [ ] All file download links work (valid signed URLs)
|
||||
- [ ] Edit artist/title → refresh → changes persisted
|
||||
- [ ] Reset job to pending → job status shows "pending"
|
||||
- [ ] Logs section loads and filters work
|
||||
- [ ] Error states handled (invalid job ID shows friendly error)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
|
||||
- **Recommendation**: Delete it. No external links depend on it.
|
||||
|
||||
2. [ ] Should job reset delete GCS files or just clear state_data references?
|
||||
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
|
||||
|
||||
3. [ ] Should edits be immediately saved or require explicit "Save" button?
|
||||
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
|
||||
2. Backend: New endpoints are additive, won't affect existing functionality
|
||||
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
|
||||
|
||||
## Notes
|
||||
|
||||
- The existing detail page has good foundational code - we're enhancing rather than replacing
|
||||
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
|
||||
- Job reset is powerful - consider adding confirmation dialogs and logging
|
||||
@@ -1,120 +0,0 @@
|
||||
# Plan: Admin Lyrics Review Dashboard
|
||||
|
||||
**Created:** 2026-03-05
|
||||
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
|
||||
|
||||
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
|
||||
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
|
||||
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
|
||||
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
|
||||
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
|
||||
- [ ] Admin page showing list of recently reviewed jobs with key metrics
|
||||
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
|
||||
- [ ] Edit log replay/stepper UI to walk through user changes one by one
|
||||
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Backend: 2 new admin endpoints
|
||||
|
||||
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
|
||||
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
|
||||
- Filter by status (completed jobs, or optionally all post-review statuses)
|
||||
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
|
||||
- Support pagination, search, exclude_test filtering
|
||||
|
||||
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
|
||||
- Returns immutable copy of:
|
||||
- Original corrections (`corrections.json` from GCS)
|
||||
- Updated corrections (`corrections_updated.json` from GCS)
|
||||
- Edit log (`edit_log_{session_id}.json` from GCS)
|
||||
- Annotations (`annotations.json` from GCS)
|
||||
- Job metadata (artist, title, user_email, timestamps)
|
||||
- Audio URL (signed, for playback in review UI)
|
||||
- This is a **read-only** endpoint - no job state mutations
|
||||
- Uses `require_admin` dependency (not `require_review_auth`)
|
||||
- Does NOT transition job status or modify any state
|
||||
|
||||
### Frontend: New admin page + Read-only review mode
|
||||
|
||||
1. **`/admin/edit-reviews/page.tsx`** - List page
|
||||
- Table of recently reviewed jobs
|
||||
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
|
||||
- "View Review" button → opens read-only lyrics review
|
||||
- "View Edit Log" button → opens edit replay view
|
||||
- "View Raw Data" button → shows raw JSON
|
||||
|
||||
2. **Read-only review viewer** - Reuse LyricsAnalyzer
|
||||
- Load data from the new admin endpoint (not the regular review endpoint)
|
||||
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
|
||||
- Header shows "Admin Review - Read Only" indicator
|
||||
- No submit/save buttons, no state transitions
|
||||
- Audio playback still works (via signed URL)
|
||||
|
||||
3. **Edit log replay** - New component
|
||||
- Timeline/stepper showing each edit entry
|
||||
- For each entry: operation type, text before → text after, timestamp
|
||||
- Step forward/backward through edits
|
||||
- Show feedback reason if user provided one
|
||||
- Summary stats: total edits, edits with feedback, correction types breakdown
|
||||
|
||||
4. **Raw data viewer** - JSON viewer tabs
|
||||
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
|
||||
- Tab 2: Annotations
|
||||
- Tab 3: Corrections diff (original vs updated)
|
||||
|
||||
### Key safety guarantee
|
||||
|
||||
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
|
||||
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
|
||||
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
|
||||
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
|
||||
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
|
||||
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
|
||||
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
|
||||
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
|
||||
9. [ ] **Tests: Backend unit tests** for new admin endpoints
|
||||
10. [ ] **Tests: Frontend component tests** for new page
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
|
||||
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
|
||||
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
|
||||
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
|
||||
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
|
||||
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for backend endpoints (mock Firestore + GCS)
|
||||
- Verify admin auth is required on all new endpoints
|
||||
- Verify no write operations occur on read-only endpoints
|
||||
- Frontend component tests for key interactions
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
|
||||
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
|
||||
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
|
||||
@@ -1,151 +0,0 @@
|
||||
# Plan: AI-Powered Credit Gating
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1716-ai-credit-gating
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Welcome credit grants go through AI evaluation before granting
|
||||
- [ ] Feedback credit grants go through AI evaluation before granting
|
||||
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
|
||||
- [ ] Approved users get credits as before (seamless experience)
|
||||
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
|
||||
- [ ] Rejection email invites them to reply if it's a mistake
|
||||
- [ ] Decision is logged to Firestore for audit trail
|
||||
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
|
||||
- [ ] Admin can override decisions via existing admin credit grant endpoint
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User verifies email / submits feedback
|
||||
↓
|
||||
Collect abuse signals (sync, fast):
|
||||
- User's fingerprint + IP
|
||||
- Other accounts with same fingerprint (from correlations)
|
||||
- Other accounts with same IP
|
||||
- IP geolocation info
|
||||
- Recent signup count from same IP/fingerprint (last 24h)
|
||||
- User agent string
|
||||
- For feedback: the feedback content itself
|
||||
↓
|
||||
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
|
||||
- Present all signals
|
||||
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
|
||||
↓
|
||||
If "grant": grant credits normally, log decision
|
||||
If "deny": skip credit grant, send rejection email, log decision
|
||||
If AI call fails: grant credits (fail-open), log error
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
|
||||
|
||||
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
|
||||
|
||||
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
|
||||
|
||||
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
|
||||
|
||||
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add google-generativeai dependency
|
||||
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
|
||||
- Verify it works with Vertex AI auth in the nomadkaraoke project
|
||||
|
||||
### Step 2: Create credit evaluation service
|
||||
- New file: `backend/services/credit_evaluation_service.py`
|
||||
- `CreditEvaluationService` class with:
|
||||
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
|
||||
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
|
||||
- Builds prompt with signals
|
||||
- Calls Gemini for decision
|
||||
- Logs decision to Firestore
|
||||
- Returns structured result
|
||||
|
||||
### Step 3: Build the Gemini prompt
|
||||
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
|
||||
- Present all signals in a structured format
|
||||
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
|
||||
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
|
||||
|
||||
### Step 4: Create rejection email template
|
||||
- Add `send_credit_denied_email(email, reasoning)` to EmailService
|
||||
- Friendly-but-firm tone
|
||||
- CC andrew@beveridge.uk
|
||||
- Invite reply if it's a mistake
|
||||
- Use existing email template infrastructure (branding, styling)
|
||||
|
||||
### Step 5: Integrate into welcome credit flow
|
||||
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
|
||||
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
|
||||
- If denied: send rejection email, return False
|
||||
- If error: log warning, grant anyway (fail-open)
|
||||
|
||||
### Step 6: Integrate into feedback credit flow
|
||||
- Modify feedback endpoint in users.py
|
||||
- Before granting credits: evaluate with feedback content included
|
||||
- If denied: return different response message, send rejection email
|
||||
- If error: grant anyway
|
||||
|
||||
### Step 7: Add Firestore collection for audit trail
|
||||
- Collection: `credit_evaluations`
|
||||
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
|
||||
- TTL: keep indefinitely (small volume, high audit value)
|
||||
|
||||
### Step 8: Tests
|
||||
- Unit tests for signal collection
|
||||
- Unit tests for prompt building
|
||||
- Unit tests for decision parsing (grant/deny/error cases)
|
||||
- Integration test for the evaluation service with mocked Gemini
|
||||
- Integration test for welcome credit flow with evaluation
|
||||
- Integration test for feedback credit flow with evaluation
|
||||
- Test fail-open behavior
|
||||
|
||||
### Step 9: Configuration
|
||||
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
|
||||
- Add to config.py
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `pyproject.toml` | Modify | Add google-generativeai dependency |
|
||||
| `backend/config.py` | Modify | Add credit evaluation config vars |
|
||||
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
|
||||
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
|
||||
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
|
||||
| `backend/services/email_service.py` | Modify | Add credit denied email template |
|
||||
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
|
||||
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
|
||||
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
|
||||
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
|
||||
- **Integration tests**: Full evaluation flow with mocked Gemini responses
|
||||
- **E2E tests**: Manual verification in production with test accounts
|
||||
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
|
||||
|
||||
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
|
||||
|
||||
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
|
||||
@@ -1,150 +0,0 @@
|
||||
# Plan: Block Disposable Email Registration
|
||||
|
||||
**Created:** 2026-03-19
|
||||
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
|
||||
|
||||
## Context
|
||||
|
||||
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
|
||||
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
|
||||
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
|
||||
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
|
||||
- Each new account gets 2 free welcome credits automatically
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Reject disposable email domains at the magic link endpoint (backend)
|
||||
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
|
||||
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
|
||||
- [x] Add `emailhook.site` to the default disposable domains list
|
||||
- [x] Add backend unit tests for the integration
|
||||
- [x] Add frontend test coverage for the new error state
|
||||
|
||||
## Design Decision: Explicit Rejection vs Silent Failure
|
||||
|
||||
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
|
||||
|
||||
1. We *want* the user to know their disposable email won't work (so they can use a real one)
|
||||
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
|
||||
3. The friendly message serves our goal of converting them to real email users
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Add disposable email check to magic link endpoint
|
||||
|
||||
**File:** `backend/api/routes/users.py`
|
||||
|
||||
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
|
||||
|
||||
```python
|
||||
# Check for disposable email domains
|
||||
email_validation = get_email_validation_service()
|
||||
if email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="disposable_email_not_allowed"
|
||||
)
|
||||
```
|
||||
|
||||
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
|
||||
|
||||
### Step 2: Backend — Also check email and IP blocklists
|
||||
|
||||
In the same location, add checks for explicitly blocked emails and IPs:
|
||||
|
||||
```python
|
||||
if email_validation.is_email_blocked(email):
|
||||
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
|
||||
# Silent reject — return generic success to prevent enumeration
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"Blocked IP signup attempt: {ip_address}")
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
```
|
||||
|
||||
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
|
||||
|
||||
### Step 3: Backend — Add `emailhook.site` to default disposable domains
|
||||
|
||||
**File:** `backend/services/email_validation_service.py`
|
||||
|
||||
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
|
||||
|
||||
### Step 4: Frontend — Handle the disposable email error in AuthDialog
|
||||
|
||||
**File:** `frontend/components/auth/AuthDialog.tsx`
|
||||
|
||||
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
|
||||
|
||||
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
|
||||
|
||||
```tsx
|
||||
const success = await sendMagicLink(email.trim().toLowerCase())
|
||||
if (success) {
|
||||
setStep("sent")
|
||||
}
|
||||
// If error contains our specific code, override with friendly message
|
||||
// (the error state is already set by the auth store)
|
||||
```
|
||||
|
||||
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
|
||||
|
||||
The friendly message:
|
||||
|
||||
> **We don't support disposable email addresses**
|
||||
>
|
||||
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
|
||||
>
|
||||
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
|
||||
|
||||
### Step 5: Backend tests
|
||||
|
||||
**File:** `backend/tests/test_email_validation_integration.py` (new)
|
||||
|
||||
Test cases:
|
||||
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
|
||||
2. Magic link with legitimate email proceeds normally
|
||||
3. Magic link with blocked email returns silent success (no actual email sent)
|
||||
4. Magic link with blocked IP returns silent success
|
||||
|
||||
### Step 6: Frontend tests
|
||||
|
||||
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
|
||||
|
||||
Test cases:
|
||||
1. Submitting a disposable email shows the friendly rejection message
|
||||
2. The friendly message contains the key copy (no spam promise, use real email)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
|
||||
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
|
||||
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
|
||||
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
|
||||
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
|
||||
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
|
||||
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
|
||||
@@ -1,236 +0,0 @@
|
||||
# Plan: Combined Lyrics + Instrumental Review Phase
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
|
||||
|
||||
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
|
||||
|
||||
### Key Insight
|
||||
|
||||
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
|
||||
|
||||
### Benefits
|
||||
|
||||
1. **Better UX** - One interaction point instead of two
|
||||
2. **Faster turnaround** - Users complete both reviews in one session
|
||||
3. **Reduced drop-off** - Users less likely to abandon between phases
|
||||
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
|
||||
5. **Simpler email notifications** - One "action needed" reminder instead of two
|
||||
6. **Cleaner codebase** - No conditional paths or legacy state handling
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
|
||||
- [x] Functional: Single email notification for "review needed" instead of two separate ones
|
||||
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
|
||||
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
|
||||
- [x] Functional: Admin "reset to review" action works with combined flow
|
||||
- [x] Non-functional: No regression in existing test coverage
|
||||
- [x] Non-functional: Clean removal of old code paths (no dead code)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Strategy: "Combined Review - Clean Cutover"
|
||||
|
||||
**State Machine - Before:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
**State Machine - After:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
|
||||
|
||||
### Key Changes
|
||||
|
||||
1. **Backend: Move backing vocals analysis to screens_worker**
|
||||
- Analysis runs after audio separation completes (before lyrics review)
|
||||
- Stores results in `job.state_data['backing_vocals_analysis']`
|
||||
- Remove analysis from render_video_worker entirely
|
||||
|
||||
2. **Backend: Replace review completion endpoint**
|
||||
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
|
||||
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
|
||||
- Remove the old endpoint that didn't require instrumental
|
||||
|
||||
3. **Backend: Simplify render_video_worker**
|
||||
- Always expect `instrumental_selection` in state_data
|
||||
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
|
||||
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
|
||||
|
||||
4. **Frontend: Single combined review page**
|
||||
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
|
||||
- Remove separate `/instrumental` route entirely
|
||||
- Single "Complete Review" button submits both
|
||||
|
||||
5. **Backend: Remove instrumental-only endpoints**
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-options`
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
|
||||
- Remove `POST /api/jobs/{job_id}/select-instrumental`
|
||||
- Keep custom instrumental creation for future advanced use
|
||||
|
||||
6. **Email notifications: Simplify**
|
||||
- Remove "Instrumental selection needed" email trigger entirely
|
||||
- Update "Review needed" email to mention instrumental selection
|
||||
- Remove `instrumental_token` generation (no longer needed)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Move backing vocals analysis to screens_worker**
|
||||
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
|
||||
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
|
||||
- Remove analysis code from render_video_worker.py entirely
|
||||
|
||||
2. [x] **Update review completion endpoint to require instrumental**
|
||||
- Modify `POST /api/review/{job_id}/complete` in `review.py`
|
||||
- Add required `instrumental_selection` field to request body
|
||||
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
|
||||
|
||||
3. [x] **Add instrumental data to correction-data endpoint**
|
||||
- Modify `GET /api/review/{job_id}/correction-data` to include:
|
||||
- `instrumental_options`: List of {id, label, audio_url}
|
||||
- `backing_vocals_analysis`: Analysis data from state_data
|
||||
|
||||
4. [x] **Simplify render_video_worker**
|
||||
- Remove conditional check for instrumental pre-selection
|
||||
- Always transition directly to `INSTRUMENTAL_SELECTED`
|
||||
- Always trigger video worker immediately after render
|
||||
|
||||
5. [x] **Update state machine**
|
||||
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
|
||||
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
|
||||
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
|
||||
|
||||
6. [x] **Remove instrumental-only endpoints**
|
||||
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
|
||||
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
|
||||
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
|
||||
- Remove `require_instrumental_auth` dependency
|
||||
- Remove `instrumental_token` generation in job_manager.py
|
||||
|
||||
### Phase 2: Frontend Changes
|
||||
|
||||
7. [x] **Create combined review page**
|
||||
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
|
||||
- Instrumental selection UI appears after preview video section
|
||||
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
|
||||
|
||||
8. [x] **Create InstrumentalSelectorEmbedded component**
|
||||
- Created embedded version of InstrumentalSelector
|
||||
- Accepts options, analysis, value, onChange props
|
||||
- Supports compact mode for modal embedding
|
||||
|
||||
9. [x] **Update review completion flow**
|
||||
- "Complete Review" button requires instrumental selection
|
||||
- Validation error shown if no instrumental selected
|
||||
- Single API call submits both lyrics and instrumental selection
|
||||
|
||||
10. [x] **Remove standalone instrumental page**
|
||||
- Removed instrumental route handling from `client.tsx`
|
||||
- Removed `getExpectedStates` case for "instrumental"
|
||||
- InstrumentalSelector component stubbed out (deprecated)
|
||||
- Kept `selectInstrumental` API for finalise-only jobs
|
||||
|
||||
11. [x] **Update lib/api.ts**
|
||||
- Modified `completeReview()` to require instrumental selection
|
||||
- Kept `selectInstrumental()` for finalise-only jobs
|
||||
- Removed other instrumental-only functions
|
||||
|
||||
### Phase 3: Email & Notification Cleanup
|
||||
|
||||
12. [x] **Remove instrumental notification trigger**
|
||||
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
|
||||
- Updated docstrings to clarify finalise-only vs normal flow
|
||||
- Instrumental reminder scheduling still works for finalise-only jobs
|
||||
|
||||
13. [x] **Update review email template**
|
||||
- Update copy to mention "review lyrics and select instrumental"
|
||||
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
|
||||
|
||||
### Phase 4: Testing & Cleanup
|
||||
|
||||
14. [x] **Update/remove tests for old flow**
|
||||
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
|
||||
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
|
||||
- Updated worker tests to import from screens_worker (where analysis moved)
|
||||
- Fixed conftest.py mocks to return AuthResult objects
|
||||
|
||||
15. [x] **Add tests for combined flow**
|
||||
- Existing tests already cover combined completion endpoint
|
||||
- Test infrastructure validates combined review flow works
|
||||
- 1741 tests pass (only 1 pre-existing unrelated failure)
|
||||
|
||||
16. [x] **Update documentation**
|
||||
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
|
||||
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
|
||||
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
|
||||
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
|
||||
|
||||
17. [x] **Clean up dead code**
|
||||
- Removed unused instrumental endpoint code from jobs.py
|
||||
- Stubbed InstrumentalSelector component (deprecated)
|
||||
- Kept select-instrumental endpoint for finalise-only jobs
|
||||
|
||||
## Files to Modify/Delete
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
|
||||
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
|
||||
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
|
||||
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
|
||||
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
|
||||
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
|
||||
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
|
||||
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
|
||||
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
|
||||
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
|
||||
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
|
||||
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
|
||||
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
|
||||
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
|
||||
|
||||
### Integration Tests
|
||||
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
|
||||
- `test_non_interactive_job`: CLI jobs skip review entirely as before
|
||||
|
||||
### E2E Tests
|
||||
- `test_combined_review_ui`: User completes both reviews in single session
|
||||
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
|
||||
| CLI regression | Test `non_interactive` flag behavior explicitly |
|
||||
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
|
||||
|
||||
## Migration Notes
|
||||
|
||||
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
|
||||
1. Admin resets job to `AWAITING_REVIEW` via admin panel
|
||||
2. User completes combined review
|
||||
3. Job proceeds normally
|
||||
|
||||
This is acceptable because:
|
||||
- Very few jobs should be in this state at any time
|
||||
- Admin reset is already a supported operation
|
||||
- Clean cutover is worth the minor migration friction
|
||||
@@ -1,321 +0,0 @@
|
||||
# Plan: Consolidate YouTube Download Paths
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Status:** Planning
|
||||
|
||||
## Problem Statement
|
||||
|
||||
YouTube downloads currently have **3 different code paths** with different behaviors:
|
||||
|
||||
1. **Audio Search → Select YouTube** (`audio_search.py`)
|
||||
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
|
||||
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
|
||||
|
||||
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
|
||||
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
|
||||
- Runs on Cloud Run, blocked by YouTube bot detection
|
||||
- Never uses remote flacfetch even when configured
|
||||
|
||||
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
|
||||
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
|
||||
- Works fine locally, not relevant to cloud issues
|
||||
|
||||
### Current Issues
|
||||
|
||||
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
|
||||
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
|
||||
3. **Inconsistent error handling** - Different paths handle errors differently
|
||||
4. **Testing gaps** - No integration tests verifying the full flow
|
||||
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
|
||||
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
|
||||
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
|
||||
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
|
||||
5. **Remove dead code** - Clean up redundant implementations
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Entry Points │
|
||||
├──────────────────────┬──────────────────────┬───────────────────────┤
|
||||
│ Audio Search Select │ Direct URL Submit │ File Upload │
|
||||
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
|
||||
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
┌──────────────────────────────────────────────┐ │
|
||||
│ YouTubeDownloadService │ │
|
||||
│ (NEW: backend/services/youtube_service.py) │ │
|
||||
│ │ │
|
||||
│ - download_by_url(url, job_id) → gcs_path │ │
|
||||
│ - download_by_id(video_id, job_id) → gcs_path│ │
|
||||
│ │ │
|
||||
│ Internally: │ │
|
||||
│ 1. Check if remote flacfetch enabled │ │
|
||||
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
|
||||
│ 3. If no: Use local yt_dlp with clear error │ │
|
||||
│ messaging about bot detection risk │ │
|
||||
└──────────────────┬───────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GCS │
|
||||
│ uploads/{job_id}/audio/ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Create YouTubeDownloadService
|
||||
|
||||
**File:** `backend/services/youtube_download_service.py`
|
||||
|
||||
```python
|
||||
class YouTubeDownloadService:
|
||||
"""
|
||||
Single point of entry for all YouTube downloads in the cloud backend.
|
||||
|
||||
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
|
||||
on the flacfetch VM which has YouTube cookies and avoids bot detection.
|
||||
|
||||
When remote is not configured, falls back to local yt_dlp with a warning
|
||||
that downloads may be blocked.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._flacfetch_client = get_flacfetch_client()
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio and upload to GCS.
|
||||
|
||||
Args:
|
||||
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (not gs:// prefix, just the path portion)
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
video_id = self._extract_video_id(url)
|
||||
|
||||
if self._flacfetch_client:
|
||||
return await self._download_remote(video_id, job_id, artist, title)
|
||||
else:
|
||||
logger.warning(
|
||||
"Remote flacfetch not configured. Attempting local download, "
|
||||
"but this may fail due to YouTube bot detection on Cloud Run IPs."
|
||||
)
|
||||
return await self._download_local(url, job_id, artist, title)
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Download by video ID instead of URL."""
|
||||
url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
return await self.download(url, job_id, artist, title)
|
||||
```
|
||||
|
||||
### Phase 2: Refactor Audio Search Path
|
||||
|
||||
**File:** `backend/api/routes/audio_search.py`
|
||||
|
||||
Changes to `_download_and_start_processing()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Complex branching with multiple paths
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Torrent path...
|
||||
elif is_remote_enabled and source_id and source_name:
|
||||
# YouTube remote path (buggy source_id extraction)...
|
||||
else:
|
||||
# Local path with another YouTube branch...
|
||||
|
||||
# AFTER: Clean separation by source type
|
||||
if is_torrent_source:
|
||||
result = await _download_torrent(source_name, source_id, target_file, job_id)
|
||||
elif source_name == 'YouTube':
|
||||
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
|
||||
elif source_name == 'Spotify':
|
||||
result = await _download_spotify(source_id, job_id, artist, title)
|
||||
else:
|
||||
raise DownloadError(f"Unknown source: {source_name}")
|
||||
```
|
||||
|
||||
### Phase 3: Refactor Direct URL Path
|
||||
|
||||
**File:** `backend/api/routes/file_upload.py`
|
||||
|
||||
Changes to `create_job_from_url()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
|
||||
# This always runs on Cloud Run and gets blocked
|
||||
|
||||
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
|
||||
if is_youtube_url(body.url):
|
||||
youtube_service = get_youtube_download_service()
|
||||
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
|
||||
# Update job with audio already downloaded
|
||||
job.input_media_gcs_path = gcs_path
|
||||
# Skip audio download in worker
|
||||
```
|
||||
|
||||
**File:** `backend/workers/audio_worker.py`
|
||||
|
||||
Changes to `download_audio()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Has YouTube download logic with download_from_url()
|
||||
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
|
||||
|
||||
async def download_audio(job_id: str, job: Job) -> str:
|
||||
"""Download audio from GCS (already uploaded by service layer)."""
|
||||
if not job.input_media_gcs_path:
|
||||
raise Exception("No audio path - this shouldn't happen for URL jobs")
|
||||
|
||||
# Download from GCS to local for processing
|
||||
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
|
||||
storage_service.download_file(job.input_media_gcs_path, local_path)
|
||||
return local_path
|
||||
```
|
||||
|
||||
### Phase 4: Clean Up Dead Code
|
||||
|
||||
Remove or consolidate:
|
||||
|
||||
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
|
||||
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
|
||||
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
|
||||
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
|
||||
|
||||
### Phase 5: Comprehensive Testing
|
||||
|
||||
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadService:
|
||||
def test_download_uses_remote_when_configured(self):
|
||||
"""When FLACFETCH_API_URL is set, should use remote client."""
|
||||
|
||||
def test_download_falls_back_to_local_when_remote_disabled(self):
|
||||
"""When remote not configured, should use local yt_dlp."""
|
||||
|
||||
def test_download_extracts_video_id_from_various_url_formats(self):
|
||||
"""Should handle youtube.com, youtu.be, shorts, etc."""
|
||||
|
||||
def test_download_returns_correct_gcs_path_format(self):
|
||||
"""Should return path without gs:// prefix."""
|
||||
|
||||
def test_download_handles_remote_failure_gracefully(self):
|
||||
"""If remote fails, should raise clear error, not fall back silently."""
|
||||
```
|
||||
|
||||
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadFlow:
|
||||
@pytest.mark.integration
|
||||
async def test_audio_search_youtube_selection_downloads_correctly(self):
|
||||
"""Full flow: search → select YouTube → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_direct_url_submission_downloads_correctly(self):
|
||||
"""Full flow: submit YouTube URL → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_youtube_download_with_remote_flacfetch(self):
|
||||
"""Verify remote flacfetch integration works end-to-end."""
|
||||
```
|
||||
|
||||
#### Flacfetch Tests (in flacfetch repo)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadByIdEndpoint:
|
||||
def test_download_by_id_with_youtube_source(self):
|
||||
"""POST /download-by-id with source_name=YouTube works."""
|
||||
|
||||
def test_download_by_id_uploads_to_gcs_when_requested(self):
|
||||
"""When gcs_path provided, should upload and return gs:// path."""
|
||||
|
||||
def test_download_by_id_handles_bot_detection_error(self):
|
||||
"""Should return clear error if YouTube blocks the request."""
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
|
||||
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
|
||||
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
|
||||
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
|
||||
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
|
||||
2. **Phase 3** changes the URL submission flow - may need careful rollout
|
||||
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
|
||||
4. **Phase 5** tests should be written alongside each phase
|
||||
|
||||
## Quick Fix vs Full Refactor
|
||||
|
||||
### Quick Fix (do now)
|
||||
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
|
||||
- This unblocks the "search → select YouTube" flow
|
||||
|
||||
### Full Refactor (this plan)
|
||||
- Consolidate all YouTube downloads into `YouTubeDownloadService`
|
||||
- Fix the "direct URL submission" flow to use remote flacfetch
|
||||
- Remove dead code and simplify branching
|
||||
- Add comprehensive tests
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing flows | Comprehensive tests before deployment |
|
||||
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
|
||||
| YouTube bot detection | Remote flacfetch with cookies is the solution |
|
||||
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **One code path** for YouTube downloads in cloud backend
|
||||
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
|
||||
3. **All tests pass** with >80% coverage on new code
|
||||
4. **Production verified** with real YouTube downloads
|
||||
5. **No dead code** remaining in the codebase
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
|
||||
2. Should direct URL submission be synchronous (wait for download) or async (background)?
|
||||
3. How do we handle YouTube cookies management for remote flacfetch?
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
|
||||
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
|
||||
@@ -1,111 +0,0 @@
|
||||
# Plan: Credit Refund on Job Deletion and Cancellation
|
||||
|
||||
**Created:** 2026-03-04
|
||||
**Branch:** feat/sess-20260304-1308-investigate-user-credits
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Job failures already refund credits (existing behavior, no change needed)
|
||||
- [ ] Job cancellation refunds 1 credit
|
||||
- [ ] Job deletion refunds 1 credit (if not already completed)
|
||||
- [ ] No refund for completed jobs (user got their video)
|
||||
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
|
||||
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
|
||||
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
|
||||
- [ ] Refund failure must not block the cancel/delete operation
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Key Insight: Idempotency via Job Field
|
||||
|
||||
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
|
||||
|
||||
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
|
||||
|
||||
### What Should and Shouldn't Be Refunded
|
||||
|
||||
| Job Status at Time of Action | Delete | Cancel | Notes |
|
||||
|------------------------------|--------|--------|-------|
|
||||
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
|
||||
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
|
||||
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
|
||||
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
|
||||
|
||||
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
|
||||
|
||||
### Refund Helper
|
||||
|
||||
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
|
||||
1. Checks `job.credit_refunded` — skip if already refunded
|
||||
2. Checks `job.user_email` — skip if no user
|
||||
3. Checks if admin — skip if admin job
|
||||
4. Calls `user_service.refund_credit()` with appropriate reason
|
||||
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
|
||||
6. Wraps everything in try/except so failures don't block the parent operation
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
|
||||
- Add `credit_refunded: bool = False` to the Job model
|
||||
- No migration needed — Firestore is schemaless, defaults to False for existing docs
|
||||
|
||||
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
|
||||
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
|
||||
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
|
||||
- Add `credit_refunded` check at the top
|
||||
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
|
||||
|
||||
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
|
||||
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
|
||||
- The job object is already fetched at line 1017
|
||||
|
||||
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
|
||||
- Need to fetch the job (already done when `delete_files=True`, but not always)
|
||||
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
|
||||
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
|
||||
- For FAILED/CANCELLED: skip because refund was already issued during that transition
|
||||
- For COMPLETE/PREP_COMPLETE: skip because user received their output
|
||||
|
||||
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
|
||||
- The helper handles this, but also update the Firestore doc
|
||||
|
||||
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
|
||||
- Test: cancel_job refunds credit for non-admin user
|
||||
- Test: cancel_job skips refund for admin jobs
|
||||
- Test: delete_job refunds credit for non-complete, non-terminal jobs
|
||||
- Test: delete_job does NOT refund for completed jobs
|
||||
- Test: delete_job does NOT double-refund for already-failed jobs
|
||||
- Test: delete_job does NOT double-refund for already-cancelled jobs
|
||||
- Test: refund failure doesn't block cancel/delete
|
||||
- Test: credit_refunded flag prevents double refund
|
||||
|
||||
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
|
||||
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
|
||||
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
|
||||
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
|
||||
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- The `credit_refunded` field is additive (defaults to False for old jobs)
|
||||
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
|
||||
- Any refunds already issued are just bonus credits to users, no harm done
|
||||
@@ -1,204 +0,0 @@
|
||||
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
|
||||
|
||||
**Created:** 2026-01-19
|
||||
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
|
||||
|
||||
This plan covers:
|
||||
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
|
||||
2. Archiving the GitHub repo
|
||||
3. Deprecating the PyPI package (via README update published as final version)
|
||||
4. Reorganizing vendored code to a permanent location in karaoke-gen
|
||||
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
|
||||
- [ ] python-lyrics-transcriber GitHub repo archived
|
||||
- [ ] Final PyPI release with deprecation notice in README
|
||||
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
|
||||
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
|
||||
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
|
||||
- [ ] Tests pass after reorganization
|
||||
- [ ] karaoke-gen pyproject.toml updated for new path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### New Path Structure
|
||||
|
||||
Move the vendored code to `libs/lyrics_transcriber/`:
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── libs/
|
||||
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
|
||||
│ ├── __init__.py
|
||||
│ ├── cli/
|
||||
│ ├── core/
|
||||
│ ├── correction/
|
||||
│ ├── lyrics/
|
||||
│ ├── output/
|
||||
│ ├── review/
|
||||
│ ├── storage/
|
||||
│ ├── transcribers/
|
||||
│ ├── types.py
|
||||
│ └── utils/
|
||||
├── karaoke_gen/
|
||||
├── backend/
|
||||
├── frontend/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Why `libs/lyrics_transcriber/` instead of alternatives?
|
||||
|
||||
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
|
||||
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
|
||||
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
|
||||
|
||||
### pyproject.toml Changes
|
||||
|
||||
```toml
|
||||
packages = [
|
||||
{ include = "karaoke_gen" },
|
||||
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
|
||||
{ include = "backend" }
|
||||
]
|
||||
include = [
|
||||
"karaoke_gen/nextjs_frontend/out",
|
||||
"karaoke_gen/nextjs_frontend/out/**/*",
|
||||
"libs/lyrics_transcriber/output/fonts/*", # Updated path
|
||||
"libs/lyrics_transcriber/output/cdgmaker/images/*",
|
||||
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
|
||||
]
|
||||
```
|
||||
|
||||
### Files to Remove from Vendored Code
|
||||
|
||||
From current `lyrics_transcriber_temp/`:
|
||||
- `pyproject.toml` - Package config for standalone release
|
||||
- `poetry.lock` - Lock file for standalone package
|
||||
- `README.md` - Will be replaced by deprecation notice
|
||||
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
|
||||
- `Dockerfile` - Standalone container build
|
||||
- `.github/` - CI/CD for standalone package
|
||||
- `.cursor/` - Cursor AI config
|
||||
- `.specify/` - Specify config
|
||||
- `.gitignore` - Not needed (karaoke-gen has its own)
|
||||
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
|
||||
- `scripts/` - Development scripts for standalone package
|
||||
- `specs/` - Specification files
|
||||
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
|
||||
|
||||
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update and Archive External Project (Manual Steps)
|
||||
|
||||
1. [ ] Clone python-lyrics-transcriber repo locally
|
||||
2. [ ] Create deprecation notice README:
|
||||
```markdown
|
||||
# python-lyrics-transcriber (DEPRECATED)
|
||||
|
||||
**This project has been deprecated and archived.**
|
||||
|
||||
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
|
||||
|
||||
## Migration
|
||||
|
||||
For karaoke video generation with synchronized lyrics:
|
||||
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
|
||||
- Web app: https://gen.nomadkaraoke.com
|
||||
|
||||
## Historical Context
|
||||
|
||||
This library was originally developed as a standalone tool for creating synchronized
|
||||
lyrics files. It has now been integrated into the karaoke-gen platform which provides
|
||||
a complete end-to-end solution for karaoke video generation.
|
||||
|
||||
The last standalone version was 0.81.0. No further releases will be made to PyPI.
|
||||
```
|
||||
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
|
||||
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
|
||||
|
||||
### Phase 2: Reorganize Vendored Code in karaoke-gen
|
||||
|
||||
5. [ ] Create `libs/` directory
|
||||
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
|
||||
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
|
||||
8. [ ] Update `pyproject.toml` packages and include paths
|
||||
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
|
||||
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
|
||||
|
||||
### Phase 3: Verify and Test
|
||||
|
||||
10. [ ] Run `poetry install` to verify package configuration
|
||||
11. [ ] Run `make test` to ensure all tests pass
|
||||
12. [ ] Verify imports work:
|
||||
```python
|
||||
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
```
|
||||
|
||||
### Phase 4: Handle Tests from lyrics_transcriber
|
||||
|
||||
13. [ ] Evaluate lyrics_transcriber tests:
|
||||
- **74 test files** in `lyrics_transcriber_temp/tests/`
|
||||
- These test the internal lyrics_transcriber library functionality
|
||||
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
|
||||
- Skip integration/performance/manual tests that were for standalone development
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
|
||||
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
|
||||
| `pyproject.toml` | Modify | Update package paths |
|
||||
| `scripts/benchmark_encoding.py` | Modify | Update import path |
|
||||
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Immediate Testing
|
||||
- Run `make test` after reorganization - all existing tests must pass
|
||||
- Verify Poetry can build/install the package correctly
|
||||
|
||||
### Test Migration Decision
|
||||
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
|
||||
- `unit/correction/` - Correction logic tests (important)
|
||||
- `unit/lyrics/` - Lyrics provider tests
|
||||
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
|
||||
- `integration/` - Integration tests
|
||||
- `performance/` - Performance benchmarks
|
||||
|
||||
**Recommendation:** Migrate a subset of critical unit tests, particularly:
|
||||
- `test_corrector.py` - Core correction logic
|
||||
- `test_anchor_sequence*.py` - Anchor sequence matching
|
||||
- Handler tests in `handlers/`
|
||||
|
||||
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
|
||||
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
|
||||
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Git revert the consolidation commits
|
||||
2. `lyrics_transcriber_temp/` can be restored from git history
|
||||
3. The external python-lyrics-transcriber repo can be unarchived if needed
|
||||
|
||||
## Notes
|
||||
|
||||
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
|
||||
- Only the internal project structure changes, not the API
|
||||
- This is primarily a cleanup/organization task with some external repo admin work
|
||||
@@ -1,110 +0,0 @@
|
||||
# Plan: Disable Auto-Correction (Agentic and Heuristic)
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1914-disable-auto-correction
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Disable agentic AI correction (LLM-based via Gemini)
|
||||
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
|
||||
- [x] Preserve the transcription → human review → video generation flow
|
||||
- [x] Minimal code changes (configuration-based if possible)
|
||||
- [x] Easy to re-enable in the future if correction quality improves
|
||||
|
||||
## Technical Approach
|
||||
|
||||
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
|
||||
|
||||
**Current behavior:**
|
||||
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
|
||||
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
|
||||
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
|
||||
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
|
||||
|
||||
**Proposed changes:**
|
||||
|
||||
### Option A: Add a new `skip_correction` config (Recommended)
|
||||
Add a single boolean `skip_correction` setting that:
|
||||
- When `true`: Skips the entire correction process, returning raw transcription
|
||||
- When `false` (future): Falls back to existing agentic/heuristic behavior
|
||||
|
||||
This is cleaner than disabling both systems separately because:
|
||||
1. Single config controls both
|
||||
2. Clear intent ("skip correction" vs "disable two different systems")
|
||||
3. Easy to understand for future maintainers
|
||||
|
||||
### Option B: Just disable both individually
|
||||
- Set `use_agentic_ai` default to `false`
|
||||
- Disable default heuristic handlers by setting `enabled_handlers=[]`
|
||||
|
||||
Option A is cleaner, but Option B requires fewer code changes.
|
||||
|
||||
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Add `skip_correction` config to backend/config.py**
|
||||
- Add `skip_correction: bool` setting, default `true`
|
||||
- Comment explaining this disables both agentic and heuristic correction
|
||||
|
||||
2. [x] **Update lyrics_worker.py to pass skip_correction**
|
||||
- Pass the config value through to LyricsProcessor/LyricsTranscriber
|
||||
|
||||
3. [x] **Update corrector.py to support skip_correction**
|
||||
- When `skip_correction=True`, return immediately with uncorrected segments
|
||||
- No handler processing, no agentic AI
|
||||
- Still need to do anchor/gap finding? No - skip entirely
|
||||
|
||||
4. [x] **Update LyricsProcessor to support skip_correction**
|
||||
- Pass through to LyricsTranscriber
|
||||
|
||||
5. [x] **Update LyricsTranscriber controller to support skip_correction**
|
||||
- Pass through to corrector or skip corrector.run() entirely
|
||||
|
||||
6. [x] **Update docs/README.md**
|
||||
- Note that auto-correction is disabled
|
||||
- Update "Agentic AI correction" status to "Disabled"
|
||||
|
||||
7. [x] **Run tests to verify no regressions**
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
|
||||
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
|
||||
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
|
||||
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
|
||||
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
|
||||
| `docs/README.md` | Modify | Update status to reflect disabled correction |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: No new tests needed - this is a config change
|
||||
- **Existing tests**: Run `make test` to ensure no regressions
|
||||
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the approach is straightforward.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
To re-enable auto-correction:
|
||||
1. Set `SKIP_CORRECTION=false` environment variable, OR
|
||||
2. Change the default back to `false` in `backend/config.py`
|
||||
|
||||
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
|
||||
|
||||
## Alternative Considered
|
||||
|
||||
Instead of adding a new config, we could:
|
||||
1. Set `USE_AGENTIC_AI=false` (disables agentic)
|
||||
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
|
||||
|
||||
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
|
||||
@@ -1,99 +0,0 @@
|
||||
# Plan: Encoding Worker Immutable Deployment Pattern
|
||||
|
||||
## Status: In Progress
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The GCE encoding worker has recurring version mismatch issues because:
|
||||
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
|
||||
2. Wheel selection uses `sort -V` which can fail if image is outdated
|
||||
3. CI only restarts service - doesn't verify version strictly
|
||||
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
|
||||
5. 51 old wheel files accumulate in GCS bucket
|
||||
|
||||
## Solution: Immutable Deployment Pattern
|
||||
|
||||
### Core Principles
|
||||
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
|
||||
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
|
||||
3. **Version Manifest**: CI writes expected version, service MUST match
|
||||
4. **Strict Verification**: CI FAILS if deployed version doesn't match
|
||||
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: GCS Structure & Scripts
|
||||
|
||||
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
|
||||
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
|
||||
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
|
||||
- [x] **Step 1.4**: Create README.md with documentation
|
||||
|
||||
### Phase 2: CI Workflow Updates
|
||||
|
||||
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
|
||||
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
|
||||
- [x] **Step 2.3**: Update CI to write version.txt manifest
|
||||
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
|
||||
|
||||
### Phase 3: Packer Image Updates
|
||||
|
||||
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
|
||||
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
|
||||
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
|
||||
|
||||
### Phase 4: Documentation & Cleanup
|
||||
|
||||
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
|
||||
- [x] **Step 4.2**: Update infrastructure/packer/README.md
|
||||
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
|
||||
|
||||
## New GCS Structure
|
||||
|
||||
```
|
||||
gs://karaoke-gen-storage-nomadkaraoke/
|
||||
├── wheels/
|
||||
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
|
||||
│ └── karaoke_gen-{version}.whl # Versioned for rollback
|
||||
│
|
||||
└── encoding-worker/
|
||||
├── bootstrap.sh # Reference copy
|
||||
├── startup.sh # CI-managed, downloaded on boot
|
||||
└── version.txt # Expected version number
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Purpose |
|
||||
|------|--------|---------|
|
||||
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
|
||||
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
|
||||
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
|
||||
| `.github/workflows/ci.yml` | Modify | New deployment flow |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit test**: Verify startup.sh logic locally
|
||||
2. **Integration test**: After merge, verify:
|
||||
- CI uploads files to correct GCS paths
|
||||
- Service starts with correct version
|
||||
- Health endpoint returns expected version
|
||||
3. **Manual verification**: Check VM logs after first deployment
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
|
||||
2. Previous versioned wheels remain in GCS for manual rollback
|
||||
3. Old Packer images remain for VM rollback
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - design is complete.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bootstrap.sh is ~10 lines and should never need changes
|
||||
- All logic changes go through startup.sh in GCS
|
||||
- Version verification is strict - CI fails on mismatch
|
||||
@@ -1,166 +0,0 @@
|
||||
# Plan: Fix YouTube Remote Download Missing GCS Path
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
|
||||
|
||||
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
|
||||
|
||||
## Root Cause
|
||||
|
||||
In `backend/api/routes/audio_search.py` lines 354-363:
|
||||
|
||||
```python
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
# <-- MISSING: gcs_path parameter
|
||||
)
|
||||
```
|
||||
|
||||
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
|
||||
- [x] Backend should receive a `gs://` path from flacfetch
|
||||
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
|
||||
- [x] Maintain backwards compatibility (local-only deployments should still work)
|
||||
- [x] Add test coverage for this code path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
|
||||
|
||||
The current code structure has two branches:
|
||||
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
|
||||
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
|
||||
|
||||
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
|
||||
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
|
||||
- If remote disabled: download locally and upload to GCS from Cloud Run
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
|
||||
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
|
||||
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
|
||||
4. [x] Update logging to clarify the download path being used
|
||||
5. [x] Bump version in `pyproject.toml`
|
||||
6. [x] Run tests to verify no regressions
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
|
||||
| `pyproject.toml` | Modify | Bump version |
|
||||
|
||||
## Detailed Code Changes
|
||||
|
||||
### audio_search.py (lines 349-393)
|
||||
|
||||
Current structure:
|
||||
```python
|
||||
else:
|
||||
# Local download (YouTube or fallback) <-- WRONG: not always local!
|
||||
temp_dir = tempfile.mkdtemp(...)
|
||||
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
result = audio_search_service.download_by_id(
|
||||
...
|
||||
output_dir=temp_dir, # <-- BUG: unused for remote
|
||||
# Missing gcs_path!
|
||||
)
|
||||
```
|
||||
|
||||
Fixed structure:
|
||||
```python
|
||||
else:
|
||||
# Non-torrent download (YouTube or fallback)
|
||||
|
||||
if is_remote_enabled and source_id and source_name:
|
||||
# Remote download via flacfetch VM - upload directly to GCS
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination, # <-- FIX: Add this!
|
||||
)
|
||||
|
||||
# Handle GCS path response (same logic as torrent branch)
|
||||
if result.filepath.startswith("gs://"):
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: upload manually
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (remote disabled or no source_id)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
# ... existing local download logic ...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
|
||||
|
||||
```python
|
||||
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads via remote flacfetch include GCS path.
|
||||
|
||||
This is the same bug pattern as the torrent download bug fixed in 2025-12:
|
||||
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
|
||||
"""
|
||||
# Setup: YouTube source with remote enabled
|
||||
# Verify: download_by_id called with gcs_path parameter
|
||||
```
|
||||
|
||||
### Manual Testing (post-deploy)
|
||||
1. Create a job and select a YouTube audio source
|
||||
2. Verify job completes successfully
|
||||
3. Check GCS path is properly set in job data
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the fix is straightforward and follows the pattern established for torrent sources.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
This is a minor code change. If issues occur:
|
||||
1. Revert the PR
|
||||
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
@@ -1,407 +0,0 @@
|
||||
# Plan: Add Retry Logic for Flacfetch Download Failures
|
||||
|
||||
**Created:** 2026-02-10
|
||||
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
|
||||
|
||||
Investigation revealed **multiple related issues** when flacfetch network calls fail:
|
||||
|
||||
1. **No retry logic** - A single network hiccup causes immediate job failure
|
||||
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
|
||||
3. **Broken retry button** - Can't retry because no source info was persisted
|
||||
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
|
||||
|
||||
### Root Cause
|
||||
|
||||
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
|
||||
|
||||
### Solution
|
||||
|
||||
This plan fixes all four issues:
|
||||
- Add retry logic to flacfetch client (network resilience)
|
||||
- Persist download info before attempting download (enables retry)
|
||||
- Update retry endpoint to handle audio search jobs
|
||||
- Show error messages in frontend when retry fails
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements - Flacfetch Retry Logic
|
||||
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
|
||||
- [x] Use exponential backoff spread over ~5 minutes
|
||||
- [x] Configurable max retries via environment variable
|
||||
- [x] Log each retry attempt with context
|
||||
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
|
||||
- [x] Do retry on HTTP 5xx errors (server errors - transient)
|
||||
|
||||
### Functional Requirements - Persist Download Info
|
||||
- [x] Save source_id, source_name, target_file, download_url to job document
|
||||
- [x] Save BEFORE attempting download (not after)
|
||||
- [x] Store in top-level fields for easy retry access
|
||||
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
|
||||
|
||||
### Functional Requirements - Retry Endpoint
|
||||
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
|
||||
- [x] Check for saved download params (source_id, source_name)
|
||||
- [x] Re-download using saved params
|
||||
- [x] Clear error state before retrying
|
||||
- [x] Trigger appropriate workers after successful download
|
||||
|
||||
### Functional Requirements - Frontend Error Handling
|
||||
- [x] Show toast notification when retry fails
|
||||
- [x] Display actual error message from API
|
||||
- [x] Don't just console.error - make it visible to user
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [x] Minimal code changes (use existing patterns)
|
||||
- [x] Clear error messages that include retry context
|
||||
- [x] Backwards compatible (no breaking changes)
|
||||
- [x] Should not significantly increase total timeout duration
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decisions
|
||||
|
||||
**Option 1: Tenacity library (Recommended)**
|
||||
- ✅ Industry standard, well-tested
|
||||
- ✅ Declarative retry configuration
|
||||
- ✅ Built-in exponential backoff
|
||||
- ✅ Already used in project (check `poetry.lock`)
|
||||
- ❌ New dependency if not already present
|
||||
|
||||
**Option 2: Manual retry loop**
|
||||
- ✅ No new dependencies
|
||||
- ✅ Full control over logic
|
||||
- ❌ More code to maintain
|
||||
- ❌ Easy to get wrong (edge cases)
|
||||
|
||||
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
```python
|
||||
# Retry on network errors only (RequestError)
|
||||
# Do NOT retry on:
|
||||
# - HTTP 4xx (client errors like 400, 401, 404)
|
||||
# - Already completed downloads
|
||||
# Do retry on:
|
||||
# - httpx.RequestError (connection timeout, refused, DNS failure)
|
||||
# - HTTP 5xx (server errors - service temporarily down)
|
||||
|
||||
Max retries: 8 (total 9 attempts)
|
||||
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
|
||||
- Attempt 1: immediate
|
||||
- Attempt 2: wait 10s
|
||||
- Attempt 3: wait 20s
|
||||
- Attempt 4: wait 40s
|
||||
- Attempt 5: wait 60s (capped)
|
||||
- Attempt 6: wait 60s (capped)
|
||||
- Attempt 7: wait 60s (capped)
|
||||
- Attempt 8: wait 60s (capped)
|
||||
- Attempt 9: wait 60s (capped)
|
||||
Total max time: ~370s (6+ minutes) of backoff + request timeouts
|
||||
|
||||
This gives the flacfetch service time to recover from restarts,
|
||||
network issues, or temporary VM problems.
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to `backend/config.py`:
|
||||
```python
|
||||
# Flacfetch client retry settings
|
||||
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
|
||||
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
|
||||
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
|
||||
```
|
||||
|
||||
### Error Message Improvements
|
||||
|
||||
Change from:
|
||||
```
|
||||
Download by ID request failed:
|
||||
```
|
||||
|
||||
To:
|
||||
```
|
||||
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Check Dependencies
|
||||
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
|
||||
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
|
||||
3. [ ] Run `poetry lock` if dependencies changed
|
||||
|
||||
### Phase 2: Add Configuration
|
||||
4. [ ] Add retry configuration to `backend/config.py`
|
||||
- `flacfetch_retry_max_attempts = 9`
|
||||
- `flacfetch_retry_min_wait = 10.0`
|
||||
- `flacfetch_retry_max_wait = 60.0`
|
||||
|
||||
### Phase 3: Implement Flacfetch Retry Logic
|
||||
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
|
||||
6. [ ] Create retry decorator with:
|
||||
- Retry on `httpx.RequestError`
|
||||
- Retry on `httpx.HTTPStatusError` with 5xx status
|
||||
- Stop on `httpx.HTTPStatusError` with 4xx status
|
||||
- Exponential backoff with cap at 60s
|
||||
- Before-sleep logging callback
|
||||
7. [ ] Apply decorator to `download_by_id()` method
|
||||
8. [ ] Improve error messages to include:
|
||||
- Number of attempts made
|
||||
- Original exception type and message
|
||||
|
||||
### Phase 4: Persist Download Info Before Download
|
||||
9. [ ] Update `audio_search.py:_download_and_start_processing()`
|
||||
10. [ ] After getting source info from selected result (line 307-311)
|
||||
11. [ ] **Save to job document BEFORE download attempt:**
|
||||
```python
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_source_type': 'audio_search',
|
||||
'source_name': source_name,
|
||||
'source_id': source_id,
|
||||
'target_file': target_file,
|
||||
'download_url': download_url,
|
||||
})
|
||||
```
|
||||
12. [ ] Then proceed with download (existing code)
|
||||
13. [ ] Add to Job model if needed (check if fields exist)
|
||||
|
||||
### Phase 5: Update Retry Endpoint for Audio Search
|
||||
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
|
||||
15. [ ] Add new case before "no input audio" error (line ~1088)
|
||||
16. [ ] Check if job has saved download params:
|
||||
```python
|
||||
elif job.audio_source_type == 'audio_search' and job.source_id:
|
||||
# Has audio search download info - retry download
|
||||
```
|
||||
17. [ ] Re-call audio search download logic with saved params
|
||||
18. [ ] Or refactor download into reusable function
|
||||
|
||||
### Phase 6: Frontend Error Handling
|
||||
19. [ ] Update `frontend/components/job/JobActions.tsx`
|
||||
20. [ ] Add toast notification library (check if exists)
|
||||
21. [ ] In `handleRetry()` catch block, show toast:
|
||||
```typescript
|
||||
catch (error) {
|
||||
const message = error.response?.data?.detail || "Failed to retry job"
|
||||
toast.error(message)
|
||||
console.error("Failed to retry job:", error)
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 7: Testing - Flacfetch Retry
|
||||
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
|
||||
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
|
||||
24. [ ] Test no retry on 4xx errors
|
||||
25. [ ] Test retry on 5xx errors
|
||||
26. [ ] Test max retries exhausted
|
||||
27. [ ] Verify logging output includes attempt numbers
|
||||
|
||||
### Phase 8: Testing - End-to-End
|
||||
28. [ ] Test audio search job with flacfetch down
|
||||
29. [ ] Verify source info is saved even when download fails
|
||||
30. [ ] Verify retry button works after source info saved
|
||||
31. [ ] Test retry succeeds when flacfetch comes back up
|
||||
32. [ ] Verify error toast appears when retry fails
|
||||
|
||||
### Phase 9: Documentation
|
||||
33. [ ] Update `backend/services/flacfetch_client.py` docstring
|
||||
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
|
||||
- "Always add retries for external service calls"
|
||||
- "Save operation parameters before attempting operations"
|
||||
- "Persist enough state to enable retry from any failure point"
|
||||
35. [ ] Update `docs/ARCHITECTURE.md` if needed
|
||||
36. [ ] Document the new job fields in API.md
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add retry configuration variables |
|
||||
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
|
||||
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
|
||||
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
|
||||
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
|
||||
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
|
||||
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
|
||||
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
|
||||
| `docs/API.md` | Modify | Document new job fields |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
|
||||
|
||||
1. **Test retry on RequestError**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_network_error():
|
||||
"""Should retry 3 times on connection errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
# Fail 2 times, then succeed
|
||||
mock_post.side_effect = [
|
||||
httpx.RequestError("Connection timeout"),
|
||||
httpx.RequestError("Connection refused"),
|
||||
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
|
||||
]
|
||||
|
||||
result = await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert result == "test123"
|
||||
assert mock_post.call_count == 3
|
||||
```
|
||||
|
||||
2. **Test no retry on 4xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_no_retry_on_client_error():
|
||||
"""Should NOT retry on 400/401/404 errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
response = Mock(status_code=404, text="Not found")
|
||||
response.raise_for_status.side_effect = httpx.HTTPStatusError(
|
||||
"Not found", request=Mock(), response=response
|
||||
)
|
||||
mock_post.return_value = response
|
||||
|
||||
with pytest.raises(FlacfetchServiceError, match="404"):
|
||||
await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert mock_post.call_count == 1 # No retries
|
||||
```
|
||||
|
||||
3. **Test retry on 5xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_server_error():
|
||||
"""Should retry on 500/503 errors."""
|
||||
# Similar to test 1 but with HTTP 503 errors
|
||||
```
|
||||
|
||||
4. **Test max retries exhausted**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_fails_after_max_retries():
|
||||
"""Should fail with clear message after max retries."""
|
||||
# All attempts fail, verify error message includes attempt count
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
5. **Test with actual flacfetch service** (if available in test environment)
|
||||
- Verify retry doesn't break normal operation
|
||||
- Test with deliberate network issues
|
||||
|
||||
### Manual Testing
|
||||
|
||||
6. **Simulate network failure**
|
||||
```bash
|
||||
# Temporarily stop flacfetch VM
|
||||
gcloud compute instances stop flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Submit a test job
|
||||
# Observe logs showing retry attempts
|
||||
|
||||
# Restart VM during retries
|
||||
gcloud compute instances start flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Job should succeed
|
||||
```
|
||||
|
||||
7. **Test with intentional delays**
|
||||
- Add sleep in flacfetch API handler
|
||||
- Verify retries don't cause issues
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
|
||||
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
|
||||
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
|
||||
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
|
||||
- [ ] Should we also add retries to `wait_for_download()`?
|
||||
- Probably NO - that already has its own polling/timeout logic
|
||||
- [ ] Should we add circuit breaker pattern?
|
||||
- Probably NO for now - keep it simple
|
||||
- Could add in future if we see cascading failures
|
||||
- [ ] Should retry logic be applied to other flacfetch methods?
|
||||
- `start_download()` - probably YES (same pattern)
|
||||
- `search()` - maybe YES (search can also have network issues)
|
||||
- `get_download_status()` - probably NO (polling already handles this)
|
||||
- **Decision**: Apply to `start_download()` too, skip `search()` for now
|
||||
- [ ] Which toast library should frontend use?
|
||||
- Check if already exists (react-hot-toast, sonner, etc.)
|
||||
- If not, add lightweight library
|
||||
- [ ] Should we save download params to state_data or top-level fields?
|
||||
- Top-level fields more visible and easier to query
|
||||
- **Decision**: Top-level fields (source_id, source_name, etc.)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If this causes issues:
|
||||
|
||||
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
|
||||
- Disables retries without code change
|
||||
- Can deploy in ~2 minutes
|
||||
|
||||
2. **Full rollback**: Revert the PR
|
||||
- Git revert the merge commit
|
||||
- Deploy previous version
|
||||
|
||||
3. **Monitoring**: Watch for:
|
||||
- Increased request duration to flacfetch
|
||||
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
|
||||
- Duplicate download attempts causing issues
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Jobs with transient flacfetch failures succeed instead of failing
|
||||
- [ ] Error messages are more informative (show attempts and root cause)
|
||||
- [ ] No increase in false positives (jobs that should fail, now hang)
|
||||
- [ ] All tests pass
|
||||
- [ ] No impact on successful jobs (retry overhead is minimal)
|
||||
|
||||
## Related Issues
|
||||
|
||||
### Job 749141f8 - The Incident
|
||||
|
||||
**What happened:**
|
||||
- User searched for audio via audio search
|
||||
- Selected "Elizabeth Moen - Songbird"
|
||||
- Backend tried to download via flacfetch
|
||||
- Flacfetch call failed with network error (connection timeout/refused)
|
||||
- Job immediately failed
|
||||
- No source info saved → Retry button broken
|
||||
- User sees "Retry" button but clicking does nothing (silent 400 error)
|
||||
|
||||
**How this fix helps:**
|
||||
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
|
||||
2. ✅ Persist download info → Even if all retries fail, user can retry later
|
||||
3. ✅ Retry endpoint fix → Retry button will actually work
|
||||
4. ✅ Frontend toast → User sees helpful error if retry still fails
|
||||
|
||||
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
|
||||
|
||||
### Future Work (Not in This PR)
|
||||
|
||||
Similar pattern should be applied to other external service calls:
|
||||
- AudioShake API (lyrics transcription)
|
||||
- Modal API (audio separation)
|
||||
- YouTube direct API calls (not via flacfetch)
|
||||
- Email sending (SendGrid)
|
||||
|
||||
## References
|
||||
|
||||
- [Tenacity documentation](https://tenacity.readthedocs.io/)
|
||||
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
|
||||
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
|
||||
@@ -1,149 +0,0 @@
|
||||
# Plan: Improve Job Observability
|
||||
|
||||
**Created:** 2026-03-07
|
||||
**Branch:** feat/sess-20260307-2216-improve-job-observability
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
|
||||
|
||||
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
### What we found investigating two edited jobs:
|
||||
|
||||
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
|
||||
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
|
||||
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
|
||||
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
|
||||
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
|
||||
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
|
||||
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
|
||||
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
|
||||
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
|
||||
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
|
||||
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
|
||||
- [ ] No breaking changes to existing timeline format (additive only)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Add `metadata` field to `TimelineEvent`
|
||||
|
||||
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
|
||||
|
||||
Key events that gain metadata:
|
||||
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
|
||||
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
|
||||
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
|
||||
|
||||
### 2. Write Firestore job logs from API endpoints
|
||||
|
||||
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
|
||||
|
||||
```python
|
||||
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
|
||||
"""Write a log entry to the job's Firestore log subcollection."""
|
||||
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
|
||||
FirestoreService().append_log_to_subcollection(job_id, entry)
|
||||
```
|
||||
|
||||
### 3. Enhance distribution logging in video_worker_orchestrator
|
||||
|
||||
After each distribution upload succeeds, log the details with structured metadata:
|
||||
- YouTube: video_id, video_url, channel
|
||||
- Dropbox: folder_path, sharing_link
|
||||
- GDrive: file_ids dict
|
||||
- Brand code: code, prefix, allocated vs recycled
|
||||
|
||||
### 4. Snapshot previous outputs on edit
|
||||
|
||||
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
|
||||
|
||||
### 5. Log cleanup results from edit/admin endpoints
|
||||
|
||||
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add `metadata` to `TimelineEvent` model
|
||||
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
|
||||
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
|
||||
|
||||
### Step 2: Create `log_to_job` helper
|
||||
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
|
||||
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
|
||||
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
|
||||
- [ ] Catches and suppresses errors (logging should never break the caller)
|
||||
|
||||
### Step 3: Update `transition_to_state` to accept metadata
|
||||
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
|
||||
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
|
||||
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
|
||||
|
||||
### Step 4: Enhance edit endpoint logging
|
||||
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
|
||||
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
|
||||
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
|
||||
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
|
||||
|
||||
### Step 5: Enhance admin delete-outputs endpoint logging
|
||||
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
|
||||
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
|
||||
|
||||
### Step 6: Enhance distribution logging in video_worker_orchestrator
|
||||
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
|
||||
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
|
||||
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
|
||||
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
|
||||
|
||||
### Step 7: Add output details to `complete` timeline event
|
||||
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
|
||||
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
|
||||
|
||||
### Step 8: Tests
|
||||
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
|
||||
- [ ] Unit test: `log_to_job` writes to subcollection
|
||||
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
|
||||
- [ ] Unit test: `complete` transition includes output metadata
|
||||
- [ ] Verify existing tests still pass
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
|
||||
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
|
||||
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
|
||||
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
|
||||
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
|
||||
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
|
||||
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
|
||||
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
|
||||
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
|
||||
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
|
||||
|
||||
## What This Does NOT Include
|
||||
|
||||
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
|
||||
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
|
||||
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
|
||||
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None — the approach is straightforward and low-risk.
|
||||
@@ -1,218 +0,0 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -1,279 +0,0 @@
|
||||
# Plan: Private (Non-Published) Tracks
|
||||
|
||||
**Created:** 2026-02-24
|
||||
**Branch:** feat/sess-20260224-1149-private-tracks
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Add a `is_private` flag to jobs that changes distribution behavior:
|
||||
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
|
||||
- **YouTube**: Skipped entirely
|
||||
- **Google Drive**: Skipped entirely
|
||||
- **Email**: Completion email includes Dropbox link but no YouTube section
|
||||
- **Discord**: Still notified (for internal tracking)
|
||||
|
||||
Available to all users at job creation and toggleable by admin on existing jobs.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
|
||||
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
|
||||
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
|
||||
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
|
||||
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
|
||||
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
|
||||
- [ ] Discord notifications still fire for private jobs
|
||||
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Core Design: Single Boolean Flag with Runtime Overrides
|
||||
|
||||
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
|
||||
|
||||
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
|
||||
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
|
||||
- **Admin can flip one field** and everything adapts
|
||||
|
||||
### Key Helper Function
|
||||
|
||||
```python
|
||||
# In job_defaults_service.py
|
||||
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
|
||||
"""Return distribution settings, applying private overrides if is_private=True."""
|
||||
if getattr(job, 'is_private', False):
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=settings.default_private_dropbox_path,
|
||||
brand_prefix=settings.default_private_brand_prefix,
|
||||
enable_youtube_upload=False,
|
||||
gdrive_folder_id=None,
|
||||
# Keep other settings from job (discord, youtube_description, etc.)
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=None,
|
||||
)
|
||||
# Non-private: return job's own settings
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=getattr(job, 'youtube_description_template', None),
|
||||
)
|
||||
```
|
||||
|
||||
This helper is used in both distribution paths (orchestrator and native fallback).
|
||||
|
||||
### Distribution Paths That Need Changes
|
||||
|
||||
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
|
||||
|
||||
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
|
||||
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
|
||||
|
||||
### Email Template Change
|
||||
|
||||
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
|
||||
|
||||
```python
|
||||
# Remove YouTube section if no URL provided
|
||||
if not variables.get("youtube_url"):
|
||||
result = re.sub(
|
||||
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["youtube_url"] = ""
|
||||
```
|
||||
|
||||
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
|
||||
|
||||
### Admin Toggle Auto-Delete
|
||||
|
||||
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Model & Config (foundation)
|
||||
|
||||
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
|
||||
- Add `is_private: bool = False` field after `made_for_you` field
|
||||
- Add to `JobCreate` model as well
|
||||
|
||||
2. [ ] **Add private distribution config** (`backend/config.py`)
|
||||
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
|
||||
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
|
||||
|
||||
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
|
||||
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
|
||||
|
||||
### Phase 2: Backend API (job creation + admin)
|
||||
|
||||
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
|
||||
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
|
||||
|
||||
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
|
||||
- Accept and pass through `is_private` field to `JobCreate`
|
||||
|
||||
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
|
||||
- Accept `is_private` Form parameter
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
|
||||
- Accept `is_private` in `AudioSearchRequest`
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
|
||||
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
|
||||
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
|
||||
|
||||
### Phase 3: Distribution Pipeline
|
||||
|
||||
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
|
||||
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
|
||||
|
||||
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
|
||||
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
|
||||
|
||||
### Phase 4: Email Template
|
||||
|
||||
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
|
||||
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
|
||||
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
|
||||
|
||||
### Phase 5: Frontend
|
||||
|
||||
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
|
||||
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
|
||||
- Show for all users (not just admin)
|
||||
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
|
||||
|
||||
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
|
||||
- Add `is_private?: boolean` to all job creation option interfaces
|
||||
- Add `is_private` to `JobUpdateRequest` interface
|
||||
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
|
||||
|
||||
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
|
||||
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
|
||||
- Show current state with visual indicator (e.g., lock icon for private jobs)
|
||||
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
|
||||
|
||||
### Phase 6: Environment Configuration
|
||||
|
||||
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
|
||||
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
|
||||
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
|
||||
- These should be configured in Cloud Run service and CI test environment
|
||||
|
||||
### Phase 7: Testing
|
||||
|
||||
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
|
||||
- Test `get_effective_distribution_for_job()` with private and non-private jobs
|
||||
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
|
||||
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
|
||||
- Test `_upload_to_dropbox()` uses private path and brand prefix
|
||||
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
|
||||
- Test `_handle_native_distribution()` respects private overrides
|
||||
- Test non-private jobs are completely unaffected (regression tests)
|
||||
|
||||
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
|
||||
- Test YouTube section removed from email when youtube_url is None
|
||||
- Test YouTube section present when youtube_url is provided
|
||||
- Test Dropbox section still present for private jobs
|
||||
- Test full email rendering for private job scenario
|
||||
|
||||
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
|
||||
- Test admin can update is_private field
|
||||
- Test toggling is_private to True on completed job triggers output deletion
|
||||
- Test toggling is_private to True on non-completed job does NOT delete outputs
|
||||
- Test toggling is_private to False does nothing special
|
||||
|
||||
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
|
||||
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
|
||||
|
||||
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
|
||||
- Test private checkbox appears in job creation form
|
||||
- Test private toggle in admin dashboard
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
|
||||
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
|
||||
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
|
||||
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
|
||||
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
|
||||
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
|
||||
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
|
||||
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
|
||||
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
|
||||
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
|
||||
- `get_effective_distribution_for_job()` with private=True and False
|
||||
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
|
||||
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
|
||||
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
|
||||
- Native distribution fallback applies same overrides
|
||||
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
|
||||
|
||||
### Email Template Tests (~5 tests)
|
||||
- YouTube section stripped when youtube_url is None
|
||||
- YouTube section preserved when youtube_url has value
|
||||
- Dropbox section always present
|
||||
- Full private job email rendering (Dropbox + no YouTube)
|
||||
- Full non-private job email rendering (both present) - regression
|
||||
|
||||
### Admin Toggle Tests (~5 tests)
|
||||
- is_private editable via PATCH
|
||||
- Toggle to private on completed job deletes outputs
|
||||
- Toggle to private on pending/in-progress job does NOT delete
|
||||
- Toggle to non-private does nothing special
|
||||
|
||||
### Frontend Tests
|
||||
- E2E: Private checkbox visible and functional in job creation
|
||||
- E2E: Admin toggle visible and includes confirmation dialog
|
||||
|
||||
### Manual Verification
|
||||
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
|
||||
- Verify completion email has Dropbox link but no YouTube section
|
||||
- Toggle existing completed job to private, verify outputs deleted
|
||||
|
||||
## Regression Risk Analysis
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
|
||||
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
|
||||
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
|
||||
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
|
||||
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
|
||||
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
|
||||
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
|
||||
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
|
||||
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
|
||||
- [x] Discord? → **Still notify** (useful for internal tracking)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
|
||||
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
|
||||
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
|
||||
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
|
||||
@@ -1,239 +0,0 @@
|
||||
# Plan: Song Lookup & Autocomplete + Community Version Detection
|
||||
|
||||
**Created:** 2026-03-02
|
||||
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
|
||||
**Status:** Implementation Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Add two features to the karaoke-gen job creation flow:
|
||||
|
||||
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
|
||||
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
|
||||
|
||||
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
|
||||
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
|
||||
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
|
||||
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
|
||||
- [ ] Community version check triggers when both artist and title are filled
|
||||
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
|
||||
- [ ] Community check does not block job submission — just informs the user
|
||||
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### How to access karaoke-decide catalog data
|
||||
|
||||
**Decision: Proxy through karaoke-gen backend**
|
||||
|
||||
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
|
||||
|
||||
| Option | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
|
||||
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
|
||||
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
|
||||
|
||||
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
|
||||
|
||||
### How to detect community karaoke versions
|
||||
|
||||
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
|
||||
|
||||
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
|
||||
|
||||
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
|
||||
|
||||
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
|
||||
|
||||
### Autocomplete UI component
|
||||
|
||||
**Decision: Use existing shadcn Command (cmdk) + Popover components**
|
||||
|
||||
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
|
||||
- Keyboard navigation (arrow keys, enter to select)
|
||||
- Debounced search-as-you-type
|
||||
- Clean dropdown with loading states
|
||||
- Works inline within forms (not modal)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Data flow
|
||||
|
||||
```
|
||||
User types in artist field
|
||||
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
|
||||
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
|
||||
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
|
||||
→ user selects "Fleetwood Mac" → field populated with canonical name
|
||||
|
||||
User types in title field
|
||||
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
|
||||
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
|
||||
→ user selects "Dreams" → field populated with canonical title
|
||||
|
||||
Both fields filled → community check fires
|
||||
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
|
||||
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
|
||||
→ if community tracks found with YouTube URLs, return them
|
||||
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
|
||||
```
|
||||
|
||||
### Caching strategy
|
||||
|
||||
| Endpoint | Cache TTL | Scope |
|
||||
|----------|-----------|-------|
|
||||
| Artist search proxy | 5 min | In-memory (backend) |
|
||||
| Track search proxy | 5 min | In-memory (backend) |
|
||||
| Community check | 1 hour | In-memory (backend) |
|
||||
|
||||
### karaoke-decide API endpoints used
|
||||
|
||||
These are all **public, no auth required**:
|
||||
|
||||
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
|
||||
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
|
||||
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
|
||||
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend — Catalog proxy + community check endpoints
|
||||
|
||||
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
|
||||
- HTTP client (httpx async) calling karaoke-decide API
|
||||
- In-memory TTL cache for responses (5 min)
|
||||
- Configurable karaoke-decide base URL
|
||||
- Graceful error handling (return empty results on failure)
|
||||
|
||||
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
|
||||
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
|
||||
- Add async support (httpx instead of requests)
|
||||
- In-memory TTL cache (1 hour)
|
||||
- Return only community tracks with YouTube URLs
|
||||
|
||||
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
|
||||
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
|
||||
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
|
||||
- `POST /api/catalog/community-check` — scrape karaokenerds.com
|
||||
- All endpoints require auth (same as job creation)
|
||||
- Pydantic request/response models
|
||||
|
||||
4. [ ] **Register routes in main.py** — add `catalog.router` to app
|
||||
|
||||
5. [ ] **Add config** — `backend/config.py`
|
||||
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
|
||||
|
||||
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
|
||||
|
||||
### Phase 2: Frontend — Autocomplete component
|
||||
|
||||
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
|
||||
- Wraps an Input field with a Popover + Command dropdown
|
||||
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
|
||||
- Debounced search (300ms) triggered on input change
|
||||
- Shows loading spinner while fetching
|
||||
- Shows "No results" when search returns empty
|
||||
- Keyboard navigation (up/down arrows, enter to select, escape to close)
|
||||
- Clicking outside closes dropdown
|
||||
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
|
||||
- When user types without selecting, uses their raw text (autocomplete is optional)
|
||||
|
||||
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
|
||||
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
|
||||
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
|
||||
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
|
||||
|
||||
### Phase 3: Frontend — Integrate autocomplete into job creation
|
||||
|
||||
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
|
||||
- Fetch suggestions from `api.searchArtists(query)`
|
||||
- On select: populate artist field with canonical name
|
||||
- Show artist name + optional disambiguation in dropdown items
|
||||
|
||||
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
|
||||
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
|
||||
- On select: populate title field with canonical name
|
||||
- Show track name + artist name + duration in dropdown items
|
||||
|
||||
### Phase 4: Frontend — Community version detection
|
||||
|
||||
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
|
||||
- Shows when community karaoke version(s) found
|
||||
- Displays: "A karaoke version of this song already exists!"
|
||||
- Lists community tracks with brand name and YouTube link
|
||||
- "Watch on YouTube" button for top community version
|
||||
- Dismissible (X button or "Continue anyway" link)
|
||||
- Styled with green border (matching kjbox community track styling)
|
||||
|
||||
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
|
||||
- Debounced check (500ms) when both artist AND title fields have values
|
||||
- Only triggers when values change (not on every keystroke)
|
||||
- Shows `CommunityVersionBanner` between the form fields and the submit button
|
||||
- Does NOT block form submission
|
||||
- Clears when artist or title changes
|
||||
|
||||
### Phase 5: Testing
|
||||
|
||||
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
|
||||
- Test proxy endpoints with mocked karaoke-decide responses
|
||||
- Test community check with mocked karaokenerds HTML
|
||||
- Test error handling (karaoke-decide down, karaokenerds down)
|
||||
- Test caching behavior
|
||||
|
||||
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
|
||||
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
|
||||
- Test HTML parsing, community detection, YouTube URL cleanup
|
||||
|
||||
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
|
||||
- Test suggestion fetching and display
|
||||
- Test keyboard navigation
|
||||
- Test selection behavior
|
||||
- Test free-text input (no selection)
|
||||
|
||||
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
|
||||
- Test autocomplete appears on typing
|
||||
- Test selecting from autocomplete populates field
|
||||
- Test community banner appears for known songs
|
||||
- Test form submission works with and without autocomplete selection
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
|
||||
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
|
||||
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
|
||||
| `backend/main.py` | Modify | Register catalog router |
|
||||
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
|
||||
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
|
||||
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
|
||||
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
|
||||
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
|
||||
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
|
||||
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
|
||||
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
|
||||
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
|
||||
|
||||
## Decisions (Resolved)
|
||||
|
||||
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
|
||||
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
|
||||
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
|
||||
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
|
||||
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Feature is entirely additive — no existing functionality is modified
|
||||
- Autocomplete fields fall back to plain text Input if API calls fail
|
||||
- Community check is non-blocking — if it fails, form works normally
|
||||
- All new endpoints are behind the existing auth middleware
|
||||
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
|
||||
@@ -1,329 +0,0 @@
|
||||
# Plan: Unified Combined Review for Local CLI
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
|
||||
|
||||
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
|
||||
- Review code lives inside `correct_lyrics()` method in controller.py
|
||||
- When auto-correction is disabled, `correct_lyrics()` is never called
|
||||
- Users never get a chance to review lyrics
|
||||
|
||||
2. **Standalone instrumental UI still exists for CLI**
|
||||
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
|
||||
- CLI launches this standalone server after lyrics review
|
||||
- The combined review flow we built only works for the web app
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
|
||||
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
|
||||
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
|
||||
4. **Keep it simple** - Minimal changes to achieve the goal
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Current State (Broken)
|
||||
|
||||
```
|
||||
CLI Flow:
|
||||
controller.py::process()
|
||||
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
|
||||
→ ReviewServer starts (lyrics only)
|
||||
→ User reviews
|
||||
→ ReviewServer stops
|
||||
→ gen_cli.py::run_instrumental_review()
|
||||
→ InstrumentalReviewServer starts (instrumental only)
|
||||
→ User selects instrumental
|
||||
→ InstrumentalReviewServer stops
|
||||
|
||||
Cloud Web App Flow:
|
||||
Job enters AWAITING_REVIEW
|
||||
→ User visits /app/jobs/{id}/review
|
||||
→ Combined UI: lyrics + instrumental
|
||||
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
|
||||
```
|
||||
|
||||
### Target State
|
||||
|
||||
```
|
||||
CLI Flow (and Cloud CLI):
|
||||
controller.py::process()
|
||||
→ _run_human_review() [NEW - always called when review enabled]
|
||||
→ ReviewServer starts (combined: lyrics + instrumental)
|
||||
→ User reviews lyrics AND selects instrumental
|
||||
→ ReviewServer stops, returns both results
|
||||
→ Continue with selected instrumental
|
||||
|
||||
Cloud Web App Flow: (unchanged)
|
||||
Same as before
|
||||
```
|
||||
|
||||
## Key Insight
|
||||
|
||||
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
|
||||
|
||||
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
|
||||
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
|
||||
3. Call the review flow from the right place (not inside `correct_lyrics()`)
|
||||
4. Delete `InstrumentalReviewServer` entirely
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Fix ReviewServer to Support Instrumental Selection
|
||||
|
||||
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None, # NEW
|
||||
backing_vocals_analysis: dict | None = None, # NEW
|
||||
...
|
||||
):
|
||||
```
|
||||
|
||||
Update `_correction_data()` endpoint to include these in response:
|
||||
|
||||
```python
|
||||
@app.get("/api/review/{job_id}/correction-data")
|
||||
def _correction_data(job_id: str):
|
||||
return {
|
||||
"lyrics_data": self.correction_result.to_dict(),
|
||||
"instrumental_options": self.instrumental_options, # NEW
|
||||
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
|
||||
}
|
||||
```
|
||||
|
||||
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
|
||||
|
||||
```python
|
||||
@app.post("/api/review/{job_id}/complete-review")
|
||||
def _complete_review(job_id: str, request: ReviewCompleteRequest):
|
||||
# Existing: store corrected lyrics
|
||||
self.corrected_lyrics = request.corrected_lyrics
|
||||
|
||||
# NEW: store instrumental selection
|
||||
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
|
||||
|
||||
self.review_completed = True
|
||||
```
|
||||
|
||||
**1.3. Expose instrumental selection after review completes**
|
||||
|
||||
The calling code needs to get the selection. Update the return value or add an attribute:
|
||||
|
||||
```python
|
||||
# After review loop completes:
|
||||
return {
|
||||
"corrected_lyrics": self.corrected_lyrics,
|
||||
"instrumental_selection": self.instrumental_selection
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Fix controller.py to Call Review at the Right Time
|
||||
|
||||
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Create new method `_run_human_review()` that can be called independently:
|
||||
|
||||
```python
|
||||
def _run_human_review(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None,
|
||||
backing_vocals_analysis: dict | None = None
|
||||
) -> tuple[CorrectionResult, str | None]:
|
||||
"""Run human review server and return corrected result + instrumental selection."""
|
||||
|
||||
if not self.output_config.review_enabled:
|
||||
return correction_result, None
|
||||
|
||||
server = ReviewServer(
|
||||
correction_result=correction_result,
|
||||
instrumental_options=instrumental_options,
|
||||
backing_vocals_analysis=backing_vocals_analysis,
|
||||
...
|
||||
)
|
||||
|
||||
result = server.start() # Blocks until review complete
|
||||
|
||||
return result["corrected_lyrics"], result.get("instrumental_selection")
|
||||
```
|
||||
|
||||
**2.2. Update `process()` to always call review when appropriate**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Current flow (simplified):
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
if self.output_config.run_correction:
|
||||
self.correct_lyrics() # Review is inside here!
|
||||
# ...
|
||||
```
|
||||
|
||||
New flow:
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
|
||||
# Build correction result (with or without auto-correction)
|
||||
if self.output_config.run_correction:
|
||||
correction_result = self._run_auto_correction()
|
||||
else:
|
||||
correction_result = self._create_uncorrected_result()
|
||||
|
||||
# Human review is separate from auto-correction
|
||||
if self.output_config.review_enabled:
|
||||
instrumental_options = self._get_instrumental_options()
|
||||
backing_vocals_analysis = self._analyze_backing_vocals()
|
||||
|
||||
corrected, instrumental_selection = self._run_human_review(
|
||||
correction_result,
|
||||
instrumental_options,
|
||||
backing_vocals_analysis
|
||||
)
|
||||
|
||||
self.results.correction_result = corrected
|
||||
self.results.instrumental_selection = instrumental_selection
|
||||
```
|
||||
|
||||
### Phase 3: Update gen_cli.py to Use Combined Review
|
||||
|
||||
**3.1. Remove standalone instrumental review call**
|
||||
|
||||
File: `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
|
||||
|
||||
Before:
|
||||
```python
|
||||
# After lyrics processing
|
||||
if not args.non_interactive:
|
||||
instrumental_selection = run_instrumental_review(...)
|
||||
```
|
||||
|
||||
After:
|
||||
```python
|
||||
# Instrumental selection comes from controller.results.instrumental_selection
|
||||
instrumental_selection = controller.results.instrumental_selection
|
||||
```
|
||||
|
||||
**3.2. Pass instrumental options to controller**
|
||||
|
||||
The controller needs instrumental options to display in review. These come from the separated audio stems:
|
||||
|
||||
```python
|
||||
controller = LyricsTranscriber(
|
||||
# ... existing params ...
|
||||
)
|
||||
|
||||
# Before calling process(), set up instrumental options
|
||||
# These are typically discovered during audio separation
|
||||
```
|
||||
|
||||
### Phase 4: Delete Standalone InstrumentalReviewServer
|
||||
|
||||
**4.1. Delete files**
|
||||
|
||||
- Delete `karaoke_gen/instrumental_review/server.py`
|
||||
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
|
||||
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
**4.2. Clean up imports**
|
||||
|
||||
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
|
||||
|
||||
### Phase 5: Update Frontend for Local Mode
|
||||
|
||||
**5.1. Verify local mode detection works**
|
||||
|
||||
File: `frontend/lib/api.ts`
|
||||
|
||||
The frontend already has local mode detection based on hostname and port. Verify it works:
|
||||
|
||||
```typescript
|
||||
const isLocalMode = () => {
|
||||
if (typeof window === 'undefined') return false;
|
||||
const hostname = window.location.hostname;
|
||||
const port = parseInt(window.location.port, 10);
|
||||
return hostname === 'localhost' && port >= 8000 && port <= 8770;
|
||||
};
|
||||
```
|
||||
|
||||
**5.2. Verify correction-data endpoint parsing**
|
||||
|
||||
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
|
||||
|
||||
### Phase 6: Testing
|
||||
|
||||
**6.1. Manual testing checklist**
|
||||
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
|
||||
- [ ] Combined UI shows both lyrics editor and instrumental selector
|
||||
- [ ] Selecting instrumental and completing review works
|
||||
- [ ] No standalone instrumental review page appears
|
||||
- [ ] Cloud web app still works (no regression)
|
||||
|
||||
**6.2. Update unit tests**
|
||||
|
||||
- Remove tests for `InstrumentalReviewServer`
|
||||
- Add tests for `ReviewServer` with instrumental options
|
||||
- Add tests for `controller._run_human_review()`
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
|
||||
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
|
||||
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
|
||||
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
|
||||
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
|
||||
| Missing instrumental options in local mode | Fail clearly if options not provided |
|
||||
| Review server port conflicts | Keep existing port allocation logic |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Where does backing vocals analysis run for local CLI?**
|
||||
- Currently runs in `render_video_worker.py` for cloud
|
||||
- For local, need to run it after audio separation, before review
|
||||
- May need to call analysis function from controller
|
||||
|
||||
2. **How are instrumental stems exposed to local review?**
|
||||
- Need to serve audio files from ReviewServer
|
||||
- Or use file:// URLs (may have CORS issues)
|
||||
- Existing approach: ReviewServer already serves static files
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. `SKIP_CORRECTION=true` no longer skips human review
|
||||
2. Single combined review UI appears (no separate instrumental step)
|
||||
3. All platforms (CLI, cloud CLI, web) use same review flow
|
||||
4. No dead code remaining for standalone instrumental review
|
||||
@@ -1,317 +0,0 @@
|
||||
# Plan: White Label B2B Karaoke Portals
|
||||
|
||||
**Created:** 2026-01-08
|
||||
**Branch:** feat/sess-20260108-0358-white-label-portal
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
|
||||
|
||||
**First customer: Vocal Star**
|
||||
- Subdomain: `vocalstar.nomadkaraoke.com`
|
||||
- Workflow: Upload original + instrumental audio (no audio search)
|
||||
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
|
||||
- Distribution: Download only (no Dropbox/GDrive/YouTube)
|
||||
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
|
||||
- [ ] Per-tenant frontend branding (logo, colors, metadata)
|
||||
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
|
||||
- [ ] Per-tenant default theme (always use Vocal Star theme)
|
||||
- [ ] Per-tenant allowed email domains for magic link auth
|
||||
- [ ] Per-tenant fixed API token support (for automation)
|
||||
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
|
||||
- [ ] Users scoped to tenant
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Minimal code duplication - single codebase serves all tenants
|
||||
- [ ] Easy to add new tenants via config, not code changes
|
||||
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
|
||||
- [ ] No performance impact on existing single-tenant flow
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decision: Config-Driven Multi-Tenancy
|
||||
|
||||
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
|
||||
|
||||
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
|
||||
2. **Frontend detects subdomain** and fetches tenant config from API
|
||||
3. **Backend middleware** extracts tenant from request and applies config
|
||||
4. **Firestore collections** get `tenant_id` field for isolation
|
||||
|
||||
This approach:
|
||||
- Requires no database migrations for new tenants
|
||||
- Configs can be managed via simple JSON files
|
||||
- Easy to test locally with query param override (`?tenant=vocalstar`)
|
||||
- Can evolve to admin UI later if needed
|
||||
|
||||
### Tenant Config Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vocalstar",
|
||||
"name": "Vocal Star",
|
||||
"subdomain": "vocalstar.nomadkaraoke.com",
|
||||
"branding": {
|
||||
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
|
||||
"primary_color": "#ffff00",
|
||||
"secondary_color": "#006CF9",
|
||||
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
|
||||
"site_title": "Vocal Star Karaoke Generator"
|
||||
},
|
||||
"features": {
|
||||
"audio_search": false,
|
||||
"file_upload": true,
|
||||
"youtube_upload": false,
|
||||
"dropbox_upload": false,
|
||||
"gdrive_upload": false,
|
||||
"theme_selection": false
|
||||
},
|
||||
"defaults": {
|
||||
"theme_id": "vocalstar",
|
||||
"distribution_mode": "download_only"
|
||||
},
|
||||
"auth": {
|
||||
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
|
||||
"fixed_tokens": ["<will be in Secret Manager>"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### 1. Subdomain Detection (Frontend)
|
||||
- Next.js middleware checks `request.headers.host`
|
||||
- If subdomain matches tenant, store tenant ID in cookie/context
|
||||
- Fetch `/api/tenant/config` to get branding/features
|
||||
- Apply CSS variables dynamically for colors
|
||||
- Swap logo component based on config
|
||||
|
||||
#### 2. Tenant Middleware (Backend)
|
||||
- FastAPI middleware extracts tenant from:
|
||||
- `X-Tenant-ID` header (set by frontend)
|
||||
- Or subdomain from `Host` header
|
||||
- Or query param `?tenant=` (dev only)
|
||||
- Attaches `tenant_id` to request state
|
||||
- Tenant config loaded from GCS (cached 5 min like themes)
|
||||
|
||||
#### 3. Auth Enhancement
|
||||
- `AuthService.validate_token_full()` already returns `user_email`
|
||||
- Add tenant validation: check if user's email domain is in tenant's allowed list
|
||||
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
|
||||
- Magic links include tenant context for proper redirect
|
||||
|
||||
#### 4. Data Isolation
|
||||
- Add `tenant_id: Optional[str]` to User and Job models
|
||||
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
|
||||
- Query filters: `where('tenant_id', '==', tenant_id or None)`
|
||||
- No schema migration needed - new field, optional
|
||||
|
||||
#### 5. Theme Integration
|
||||
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
|
||||
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
|
||||
- Tenant config specifies `default_theme_id: "vocalstar"`
|
||||
- When `theme_selection: false`, UI hides theme picker and uses default
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Tenant Foundation (Day 1-2)
|
||||
|
||||
1. [ ] **Create tenant model and config loader**
|
||||
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
|
||||
- Create `backend/services/tenant_service.py` with GCS config loading and caching
|
||||
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
|
||||
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
|
||||
|
||||
2. [ ] **Add tenant middleware**
|
||||
- Create `backend/api/middleware/tenant.py`
|
||||
- Extract tenant from subdomain or header
|
||||
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
|
||||
- Add `get_tenant()` dependency for routes
|
||||
|
||||
3. [ ] **Update data models**
|
||||
- Add `tenant_id: Optional[str] = None` to `User` model
|
||||
- Add `tenant_id: Optional[str] = None` to `Job` model
|
||||
- Update Firestore indexes in Pulumi for tenant-scoped queries
|
||||
|
||||
4. [ ] **Update auth for tenant awareness**
|
||||
- Modify `AuthService` to validate email domain against tenant config
|
||||
- Add tenant context to magic link generation
|
||||
- Support per-tenant fixed tokens in `auth_tokens` collection
|
||||
|
||||
### Phase 2: Upload Vocal Star Theme (Day 2)
|
||||
|
||||
5. [ ] **Prepare Vocal Star theme for GCS**
|
||||
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
|
||||
- Upload background images to `themes/vocalstar/assets/`
|
||||
- Upload Oswald font to `themes/vocalstar/assets/`
|
||||
- Upload `style_params.json` to `themes/vocalstar/`
|
||||
- Create preview image and upload to `themes/vocalstar/preview.png`
|
||||
- Add Vocal Star to `themes/_metadata.json`
|
||||
|
||||
6. [ ] **Create Vocal Star tenant config**
|
||||
- Create `tenants/vocalstar/config.json` in GCS
|
||||
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
|
||||
- Upload favicon to `tenants/vocalstar/favicon.ico`
|
||||
|
||||
### Phase 3: Frontend Tenant Support (Day 3-4)
|
||||
|
||||
7. [ ] **Add tenant context provider**
|
||||
- Create `frontend/lib/tenant.ts` with tenant config types
|
||||
- Create `frontend/hooks/useTenant.ts` for tenant state
|
||||
- Create `TenantProvider` component that fetches config on mount
|
||||
- Detect subdomain in `_app.tsx` or middleware
|
||||
|
||||
8. [ ] **Dynamic branding**
|
||||
- Create `TenantLogo` component that renders tenant or default logo
|
||||
- Update `globals.css` to support CSS variable overrides
|
||||
- Add `applyTenantColors()` function to set CSS variables from config
|
||||
- Update page metadata (title, favicon) from tenant config
|
||||
|
||||
9. [ ] **Feature-flagged UI**
|
||||
- Wrap audio search in `{tenant.features.audio_search && ...}`
|
||||
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
|
||||
- Wrap distribution options based on tenant features
|
||||
- Show simplified form for Vocal Star (just file upload fields)
|
||||
|
||||
10. [ ] **Update magic link flow**
|
||||
- Pass tenant context when requesting magic link
|
||||
- Magic link redirect goes to tenant subdomain
|
||||
- Verify endpoint extracts tenant from link context
|
||||
|
||||
### Phase 4: Backend Feature Enforcement (Day 4-5)
|
||||
|
||||
11. [ ] **Enforce tenant features in API**
|
||||
- Audio search endpoint: return 403 if `audio_search: false`
|
||||
- File upload: check `file_upload: true`
|
||||
- Distribution settings: ignore/error if disabled features requested
|
||||
- Theme selection: use default if `theme_selection: false`
|
||||
|
||||
12. [ ] **Job creation with tenant context**
|
||||
- `JobCreate` handler sets `tenant_id` from request state
|
||||
- Distribution config respects tenant defaults
|
||||
- Theme defaults to tenant's default theme
|
||||
|
||||
13. [ ] **Query scoping**
|
||||
- Update `JobManager.get_jobs()` to filter by tenant_id
|
||||
- Update `UserService` queries to include tenant_id
|
||||
- Admin endpoints bypass tenant filter (see all tenants)
|
||||
|
||||
### Phase 5: Infrastructure & Deployment (Day 5-6)
|
||||
|
||||
14. [ ] **DNS setup**
|
||||
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
|
||||
- Configure Cloudflare Pages to accept wildcard subdomain
|
||||
- Add SSL certificate for subdomain
|
||||
|
||||
15. [ ] **Backend CORS update**
|
||||
- Add `vocalstar.nomadkaraoke.com` to allowed origins
|
||||
- Or use pattern matching for `*.nomadkaraoke.com`
|
||||
|
||||
16. [ ] **Pulumi updates**
|
||||
- Add Firestore indexes for tenant_id queries
|
||||
- Add any new secrets (tenant fixed tokens)
|
||||
|
||||
### Phase 6: Testing & Documentation (Day 6-7)
|
||||
|
||||
17. [ ] **Testing**
|
||||
- Unit tests for TenantService config loading
|
||||
- Unit tests for tenant middleware
|
||||
- E2E test for Vocal Star flow (upload audio, generate, download)
|
||||
- Test magic link works for @vocal-star.com email
|
||||
|
||||
18. [ ] **Documentation**
|
||||
- Update CLAUDE.md with tenant info
|
||||
- Create `docs/WHITE-LABEL.md` with tenant setup guide
|
||||
- Document how to add new tenants
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/tenant.py` | Create | Tenant config models |
|
||||
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
|
||||
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
|
||||
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
|
||||
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
|
||||
| `backend/models/user.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/models/job.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
|
||||
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
|
||||
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/main.py` | Modify | Add tenant middleware |
|
||||
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
|
||||
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
|
||||
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
|
||||
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
|
||||
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
|
||||
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
|
||||
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
|
||||
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
|
||||
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `TenantService`: Config loading, caching, validation
|
||||
- `TenantMiddleware`: Subdomain extraction, header extraction
|
||||
- `AuthService`: Tenant email domain validation
|
||||
- `JobManager`: Tenant-scoped queries
|
||||
|
||||
### Integration Tests (Emulator)
|
||||
- Create job with tenant context
|
||||
- Query jobs filtered by tenant
|
||||
- Auth with tenant-allowed email domain
|
||||
- Auth rejection for wrong email domain
|
||||
|
||||
### E2E Tests
|
||||
- Vocal Star happy path: Upload audio files, review lyrics, download video
|
||||
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
|
||||
- Feature hiding: Verify audio search not shown on Vocal Star portal
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] **Q: Should tenant configs be in GCS or Firestore?**
|
||||
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
|
||||
|
||||
- [x] **Q: How to handle localhost development for specific tenant?**
|
||||
A: Query param override `?tenant=vocalstar` for local testing
|
||||
|
||||
- [x] **Q: Should admin portal be tenant-aware?**
|
||||
A: No - admins see all tenants. Admin routes bypass tenant filter.
|
||||
|
||||
- [x] **Q: Do we need tenant-specific email sender identity?**
|
||||
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
|
||||
|
||||
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
|
||||
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
|
||||
|
||||
1. Revert frontend to not check tenant
|
||||
2. Revert backend middleware to not extract tenant
|
||||
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
|
||||
4. No schema migration to reverse
|
||||
|
||||
For DNS: Simply remove CNAME record if needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
|
||||
- [ ] Portal shows Vocal Star branding (logo, colors)
|
||||
- [ ] Audio search is hidden; only file upload available
|
||||
- [ ] Theme is locked to Vocal Star style
|
||||
- [ ] Distribution options hidden; download only
|
||||
- [ ] Magic link works for @vocal-star.com emails
|
||||
- [ ] Fixed API token works for automation
|
||||
- [ ] Generated videos use Vocal Star theme
|
||||
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
|
||||
@@ -1,58 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Nomad Karaoke pre-commit hook (karaoke-gen variant)
|
||||
# 1. Gitleaks secret scan (fail fast)
|
||||
# 2. i18n auto-translation when en.json changes
|
||||
# Enable: git config core.hooksPath .githooks
|
||||
set -eu
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Gitleaks secret scan
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
if command -v gitleaks >/dev/null 2>&1; then
|
||||
if ! gitleaks protect --staged --redact --no-banner; then
|
||||
printf '\n\033[31m✗ gitleaks detected a potential secret in staged changes.\033[0m\n' >&2
|
||||
printf ' • If real: remove it, rotate the credential, then re-commit.\n' >&2
|
||||
printf ' • If false positive: add an allowlist entry to .gitleaks.toml.\n' >&2
|
||||
printf ' • Emergency override (not recommended): git commit --no-verify\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
printf '\033[33mwarn:\033[0m gitleaks not installed — skipping secret scan\n' >&2
|
||||
printf ' install: brew install gitleaks\n' >&2
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. i18n auto-translation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
|
||||
|
||||
if [ -z "$STAGED_EN" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "i18n: English translations changed — running translation pipeline..."
|
||||
|
||||
for en_file in $STAGED_EN; do
|
||||
dir=$(dirname "$en_file")
|
||||
|
||||
if [[ "$dir" == frontend/messages ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
elif [[ "$dir" == backend/translations ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
else
|
||||
echo " Warning: Unknown messages dir $dir, skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo " Translating $dir..."
|
||||
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
|
||||
git add "$dir"/*.json
|
||||
echo " Done: $dir"
|
||||
else
|
||||
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
|
||||
fi
|
||||
done
|
||||
|
||||
echo "i18n: Translation complete."
|
||||
@@ -1,115 +0,0 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
+119
-762
File diff suppressed because it is too large.
Load diff
@@ -1,149 +0,0 @@
|
||||
name: DR backup freshness monitor
|
||||
|
||||
# Off-platform monitor: runs in GitHub (not GCP) so it survives a full GCP
|
||||
# compromise. If the nightly backup pipeline dies silently, this catches it.
|
||||
#
|
||||
# Required secrets:
|
||||
# AWS_BACKUP_READONLY_ACCESS_KEY_ID
|
||||
# AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
|
||||
# DR_MONITOR_DISCORD_WEBHOOK
|
||||
#
|
||||
# Setup: see docs/DISASTER-RECOVERY.md § "Backup freshness monitor".
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 14:00 UTC (~9h after the 1am ET backup window).
|
||||
- cron: "0 14 * * *"
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-freshness:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BACKUP_READONLY_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BACKUP_READONLY_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: us-east-1
|
||||
BUCKET: nomadkaraoke-backup
|
||||
DISCORD_WEBHOOK: ${{ secrets.DR_MONITOR_DISCORD_WEBHOOK }}
|
||||
|
||||
steps:
|
||||
- name: Verify required secrets are set
|
||||
run: |
|
||||
missing=0
|
||||
for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY DISCORD_WEBHOOK; do
|
||||
if [ -z "${!var}" ]; then
|
||||
echo "::error::Required secret $var is empty"
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
[ "$missing" = "0" ] || exit 1
|
||||
|
||||
- name: Install boto3
|
||||
run: pip install --quiet boto3
|
||||
|
||||
- name: Check freshness of each tracked prefix
|
||||
id: check
|
||||
shell: bash
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys, datetime, boto3
|
||||
|
||||
BUCKET = os.environ["BUCKET"]
|
||||
# (label, prefix, max_age_hours)
|
||||
TARGETS = [
|
||||
("Firestore", "firestore/", 36),
|
||||
("GCS job files", "gcs/job-files/", 36),
|
||||
("Encrypted secrets", "secrets/", 36),
|
||||
("BigQuery weekly", "bigquery/daily-refresh/", 192),
|
||||
]
|
||||
|
||||
s3 = boto3.client("s3")
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
stale, ok = [], []
|
||||
|
||||
# Paginated scan — JMESPath --query on `aws s3api list-objects-v2`
|
||||
# operates per-page, which silently breaks "newest across all pages"
|
||||
# queries on prefixes with >1000 objects (e.g. firestore/ has ~22k).
|
||||
paginator = s3.get_paginator("list_objects_v2")
|
||||
for label, prefix, max_hours in TARGETS:
|
||||
latest = None
|
||||
for page in paginator.paginate(Bucket=BUCKET, Prefix=prefix):
|
||||
for obj in page.get("Contents", []):
|
||||
if latest is None or obj["LastModified"] > latest:
|
||||
latest = obj["LastModified"]
|
||||
|
||||
if latest is None:
|
||||
stale.append(f"{label}: NO OBJECTS in s3://{BUCKET}/{prefix}")
|
||||
continue
|
||||
age = (now - latest).total_seconds()
|
||||
age_h = int(age // 3600)
|
||||
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
|
||||
(stale if age > max_hours * 3600 else ok).append(line)
|
||||
|
||||
summary = ["## DR backup freshness", ""]
|
||||
if stale:
|
||||
summary.append("### Stale")
|
||||
summary.extend(f"- {s}" for s in stale)
|
||||
summary.append("")
|
||||
summary.append("### Fresh")
|
||||
summary.extend(f"- {o}" for o in ok)
|
||||
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
|
||||
f.write("\n".join(summary) + "\n")
|
||||
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
|
||||
f.write(f"ok_count={len(ok)}\n")
|
||||
f.write(f"stale_count={len(stale)}\n")
|
||||
if stale:
|
||||
f.write("STALE_LIST<<EOF\n")
|
||||
f.write("\n".join(stale) + "\n")
|
||||
f.write("EOF\n")
|
||||
|
||||
for o in ok:
|
||||
print("OK:", o)
|
||||
if stale:
|
||||
for s in stale:
|
||||
print("STALE:", s, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PY
|
||||
|
||||
- name: Alert Discord on stale backup
|
||||
if: failure() && steps.check.outputs.stale_count != ''
|
||||
env:
|
||||
STALE: ${{ steps.check.outputs.STALE_LIST }}
|
||||
run: |
|
||||
payload=$(jq -n \
|
||||
--arg title "🚨 DR backup is stale" \
|
||||
--arg desc "$STALE" \
|
||||
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
|
||||
'{
|
||||
embeds: [{
|
||||
title: $title,
|
||||
description: $desc,
|
||||
color: 15158332,
|
||||
fields: [{name: "Workflow run", value: $url}]
|
||||
}]
|
||||
}')
|
||||
curl -fsS -X POST -H "Content-Type: application/json" \
|
||||
-d "$payload" "$DISCORD_WEBHOOK"
|
||||
|
||||
- name: Alert Discord on workflow failure (catch-all)
|
||||
if: failure() && steps.check.outputs.stale_count == ''
|
||||
run: |
|
||||
payload=$(jq -n \
|
||||
--arg title "⚠️ DR freshness monitor failed to run" \
|
||||
--arg desc "Could not list S3 backup bucket — credentials, network, or bucket policy may be broken." \
|
||||
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
|
||||
'{
|
||||
embeds: [{
|
||||
title: $title,
|
||||
description: $desc,
|
||||
color: 15844367,
|
||||
fields: [{name: "Workflow run", value: $url}]
|
||||
}]
|
||||
}')
|
||||
curl -fsS -X POST -H "Content-Type: application/json" \
|
||||
-d "$payload" "$DISCORD_WEBHOOK"
|
||||
@@ -1,337 +0,0 @@
|
||||
name: E2E Daily Test (Payment + Happy Path)
|
||||
|
||||
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
|
||||
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
skip_payment:
|
||||
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
use_test_token:
|
||||
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# =========================================================================
|
||||
# Stage 1: Purchase credits via real Stripe Checkout
|
||||
# =========================================================================
|
||||
e2e-credit-purchase:
|
||||
name: "Stage 1: Credit Purchase"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
|
||||
|
||||
outputs:
|
||||
session_token: ${{ steps.extract_token.outputs.token }}
|
||||
result: ${{ steps.test_result.outputs.result }}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Run Credit Purchase Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
|
||||
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
|
||||
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
|
||||
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 1: Credit Purchase via Real Stripe ==="
|
||||
echo "Time: $(date)"
|
||||
npx playwright test credit-purchase-real.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=600000 \
|
||||
2>&1 | tee test-output-payment.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "CREDIT PURCHASE TEST PASSED" test-output-payment.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Extract session token
|
||||
id: extract_token
|
||||
if: steps.test_result.outputs.result == 'success'
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if [ -f test-results/e2e-session-token.txt ]; then
|
||||
TOKEN=$(cat test-results/e2e-session-token.txt)
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
echo "Session token extracted for Stage 2"
|
||||
else
|
||||
echo "No session token file found"
|
||||
echo "token=" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Upload artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-payment-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/test-output-payment.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase\", \"description\": \"Payment flow is broken. Customers may not be receiving credits.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Stage 2: Happy Path (uses credit from Stage 1)
|
||||
# =========================================================================
|
||||
e2e-happy-path:
|
||||
name: "Stage 2: Happy Path"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
needs: e2e-credit-purchase
|
||||
if: |
|
||||
always() && (
|
||||
needs.e2e-credit-purchase.result == 'success' ||
|
||||
github.event.inputs.skip_payment == 'true' ||
|
||||
github.event.inputs.use_test_token == 'true'
|
||||
)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Refresh E2E session token (when use_test_token enabled)
|
||||
id: refresh_token
|
||||
if: github.event.inputs.use_test_token == 'true'
|
||||
run: |
|
||||
RESPONSE=$(curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json")
|
||||
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Run Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 2: Happy Path ==="
|
||||
echo "Time: $(date)"
|
||||
TOKEN_SOURCE="none"
|
||||
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
|
||||
TOKEN_SOURCE="use_test_token (impersonation)"
|
||||
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
|
||||
TOKEN_SOURCE="Stage 1 (credit purchase)"
|
||||
fi
|
||||
echo "Token source: $TOKEN_SOURCE"
|
||||
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output-happy-path.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output-happy-path.log && ! grep -q "failed" test-output-happy-path.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output-happy-path.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
|
||||
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Final: Send notifications
|
||||
# =========================================================================
|
||||
notify:
|
||||
name: "Notify"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [e2e-credit-purchase, e2e-happy-path]
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Discord notification (success)
|
||||
if: needs.e2e-happy-path.result == 'success'
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Send email notification
|
||||
if: github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Daily Test Results
|
||||
======================
|
||||
|
||||
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}
|
||||
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
|
||||
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
@@ -0,0 +1,173 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
@@ -1,10 +0,0 @@
|
||||
name: Security
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
gitleaks:
|
||||
uses: nomadkaraoke/.github/.github/workflows/gitleaks.yml@main
|
||||
+2
-6
@@ -3,7 +3,6 @@
|
||||
.vscode
|
||||
*.csv
|
||||
*.mp3
|
||||
ABBA*
|
||||
|
||||
# Andrew functional adds
|
||||
/audio/
|
||||
@@ -13,9 +12,6 @@ tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Local karaoke-gen CLI test output dirs (created during manual validation)
|
||||
/test-output-*/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
@@ -62,7 +58,8 @@ frontend/.eslintcache
|
||||
# Storybook
|
||||
frontend/storybook-static/
|
||||
|
||||
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
|
||||
# PWA files
|
||||
frontend/**/public/sw.js
|
||||
frontend/**/public/workbox-*.js
|
||||
frontend/**/public/swe-worker-*.js
|
||||
|
||||
@@ -235,4 +232,3 @@ flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
.superpowers/
|
||||
@@ -1,60 +0,0 @@
|
||||
# Nomad Karaoke shared gitleaks config.
|
||||
# Extends the default ruleset with one custom rule for Discord webhooks
|
||||
# and an allowlist covering known false-positive paths.
|
||||
|
||||
title = "Nomad Karaoke"
|
||||
|
||||
[extend]
|
||||
useDefault = true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Custom rules
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
[[rules]]
|
||||
id = "discord-webhook-url"
|
||||
description = "Discord webhook URL (token embedded in URL)"
|
||||
regex = '''https://(?:discord|discordapp)\.com/api/webhooks/\d{17,20}/[A-Za-z0-9_-]{60,}'''
|
||||
tags = ["discord", "webhook", "secret"]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Global allowlist
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
[allowlist]
|
||||
description = "Paths and patterns that are known false positives"
|
||||
|
||||
paths = [
|
||||
'''(?i)\.(jpg|jpeg|png|gif|bmp|svg|webp|pdf|ico|mp3|mp4|wav|flac|m4a)$''',
|
||||
'''(^|/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|poetry\.lock|Podfile\.lock|Cargo\.lock|go\.sum|composer\.lock|Gemfile\.lock|uv\.lock)$''',
|
||||
'''(^|/)node_modules/''',
|
||||
'''(^|/)vendor/''',
|
||||
'''(^|/)wp-content/''',
|
||||
'''(^|/)\.dart_tool/''',
|
||||
'''(^|/)\.venv/''',
|
||||
'''(^|/)\.next/''',
|
||||
'''(^|/)build/''',
|
||||
'''(^|/)dist/''',
|
||||
'''(^|/)\.flutter-plugins''',
|
||||
'''(^|/)test[-_]?fixtures?/''',
|
||||
'''(^|/)tests?/.*conftest\.py$''',
|
||||
'''(^|/)tests?/.*test_[^/]+\.py$''',
|
||||
'''(^|/)tests?/.*_test\.py$''',
|
||||
'''(^|/).*\.(test|spec)\.(js|mjs|cjs|ts|tsx|jsx)$''',
|
||||
'''(^|/).*\.test\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
|
||||
'''(^|/).*\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
|
||||
]
|
||||
|
||||
regexes = [
|
||||
# Explicit placeholder patterns
|
||||
'''YOUR_[A-Z_]{2,}''',
|
||||
'''EXAMPLE_[A-Z_]{2,}''',
|
||||
'''PLACEHOLDER_[A-Z_]{2,}''',
|
||||
'''CHANGE[_-]?ME''',
|
||||
'''<[A-Z_]{2,}>''',
|
||||
'''\$\{[A-Z_]{2,}\}''',
|
||||
|
||||
# Firebase / GCP client API keys — public by design (identify the project,
|
||||
# access is gated by Firebase Security Rules).
|
||||
'''AIzaSy[A-Za-z0-9_-]{33}''',
|
||||
]
|
||||
@@ -1 +0,0 @@
|
||||
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -1 +0,0 @@
|
||||
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -16,19 +16,14 @@
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
|
||||
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
@@ -39,78 +34,24 @@
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required)
|
||||
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
|
||||
- Which test types are needed (unit, integration, E2E)
|
||||
- Where tests should be placed
|
||||
- What mocking approach to use
|
||||
- Coverage expectations
|
||||
- **Production E2E tests for user-facing features** - see below
|
||||
|
||||
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
|
||||
|
||||
**Before committing**, run all tests:
|
||||
### Testing (Required Before Commit)
|
||||
```bash
|
||||
# Backend (10 min timeout, limit output)
|
||||
make test 2>&1 | tail -n 500
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run test:all 2>&1 | tail -n 200
|
||||
```
|
||||
|
||||
This single command:
|
||||
- Installs dependencies automatically (backend + frontend)
|
||||
- Runs all backend tests (unit, integration, emulator)
|
||||
- Runs all frontend tests (Jest unit + Playwright E2E)
|
||||
- Takes ~5-10 minutes
|
||||
|
||||
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
|
||||
|
||||
### Testing in Production (for Agents)
|
||||
|
||||
When asked to "test it yourself in prod" or verify a fix in production:
|
||||
|
||||
```bash
|
||||
# Get admin token
|
||||
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
|
||||
|
||||
# Option 1: Use the debug template directly
|
||||
cd frontend && node e2e/helpers/debug-prod-template.mjs
|
||||
|
||||
# Option 2: Copy template for customization (gitignored)
|
||||
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
|
||||
# Edit the script, then run it
|
||||
node test-my-issue.local.mjs
|
||||
|
||||
# Option 3: Run existing production E2E tests
|
||||
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
|
||||
```
|
||||
|
||||
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
|
||||
|
||||
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- **Always run `pulumi up` locally BEFORE merging PRs** - CI deploys run on spot VMs that can be preempted mid-apply, causing partial state. Apply locally first, then merge (CI re-runs as no-op)
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- All GCP changes via Pulumi in `infrastructure/`
|
||||
- `gcloud` CLI for reading/debugging only
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
### Internationalization (i18n)
|
||||
|
||||
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
|
||||
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
|
||||
- Components use `useTranslations('namespace')` from next-intl
|
||||
- Pages are under `frontend/src/app/[locale]/` — locale-aware routing
|
||||
- Admin pages (under `/admin/`, `/app/`) use DefaultIntlProvider (English-only, excluded from i18n)
|
||||
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
|
||||
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
|
||||
- Translation uses Gemini 3.1 Pro via Vertex AI with GCS cache for efficiency
|
||||
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
|
||||
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
|
||||
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
@@ -144,16 +85,14 @@ EOF
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# Check health (includes wheel_version)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
|
||||
# Restart service (pick up new wheel after a deploy)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-a --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
|
||||
@@ -1,77 +1,34 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Per-stage timeout in seconds (default: 10 minutes)
|
||||
# Prevents hangs from stale dev servers, broken emulators, etc.
|
||||
# Override with: make test TEST_TIMEOUT=300
|
||||
TEST_TIMEOUT ?= 600
|
||||
.PHONY: help test test-unit test-backend test-integration test-e2e test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend unit tests (Jest)"
|
||||
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make build-frontend - Build frontend and copy to Python package"
|
||||
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
|
||||
@echo " make test - Run all tests (unit + backend + emulator-based)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen)"
|
||||
@echo " make test-backend - Run backend unit tests only"
|
||||
@echo " make test-e2e - Run E2E tests with emulators (starts/stops automatically)"
|
||||
@echo " make test-all - Alias for 'make test'"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Install dependencies (only if needed)
|
||||
install-backend:
|
||||
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
|
||||
echo "=== Installing backend dependencies ==="; \
|
||||
poetry install; \
|
||||
fi
|
||||
|
||||
install-frontend:
|
||||
@if [ ! -d "frontend/node_modules" ]; then \
|
||||
echo "=== Installing frontend dependencies ==="; \
|
||||
cd frontend && npm install; \
|
||||
fi
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Build frontend and copy to Python package
|
||||
build-frontend: install-frontend
|
||||
@echo "=== Building frontend ==="
|
||||
cd frontend && npm run build
|
||||
@echo "=== Copying build to Python package ==="
|
||||
rm -rf karaoke_gen/nextjs_frontend/out
|
||||
mkdir -p karaoke_gen/nextjs_frontend/out
|
||||
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
|
||||
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
|
||||
|
||||
# Build frontend + install package in editable mode for local CLI testing
|
||||
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
|
||||
dev-install: build-frontend
|
||||
@echo "=== Installing package in editable mode ==="
|
||||
pip install -e .
|
||||
@echo ""
|
||||
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
|
||||
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit: install-backend
|
||||
test-unit:
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
# Run backend unit tests
|
||||
test-backend:
|
||||
@echo "=== Running backend unit tests ==="
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e: install-backend
|
||||
test-e2e:
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
@@ -80,28 +37,13 @@ test-e2e: install-backend
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all backend tests (unit + emulator)
|
||||
test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend unit tests (Jest only - E2E requires manual setup)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend unit tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
|
||||
|
||||
# Run frontend E2E tests (requires dev server - run manually, not in make test)
|
||||
test-frontend-e2e: install-frontend
|
||||
@echo "=== Running frontend E2E tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
# Run all tests (use this before committing!)
|
||||
test: test-unit test-backend test-e2e
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
|
||||
@@ -159,9 +159,6 @@ karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# Skip video encoding (CDG/TXT only, faster)
|
||||
karaoke-gen --no-video --enable_cdg "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
@@ -406,8 +403,26 @@ gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
|
||||
See `.github/workflows/ci.yml` for the full deployment workflow.
|
||||
```bash
|
||||
# Build and deploy
|
||||
gcloud builds submit --config=cloudbuild.yaml
|
||||
|
||||
# Get outputs from Pulumi
|
||||
SA_EMAIL=$(pulumi stack output service_account_email)
|
||||
BUCKET_NAME=$(pulumi stack output bucket_name)
|
||||
|
||||
# Deploy Cloud Run service
|
||||
gcloud run deploy karaoke-backend \
|
||||
--image us-central1-docker.pkg.dev/YOUR-PROJECT/karaoke-repo/karaoke-backend:latest \
|
||||
--platform managed \
|
||||
--region us-central1 \
|
||||
--allow-unauthenticated \
|
||||
--service-account $SA_EMAIL \
|
||||
--memory 2Gi \
|
||||
--cpu 2 \
|
||||
--timeout 600 \
|
||||
--set-env-vars="GOOGLE_CLOUD_PROJECT=YOUR-PROJECT,GCS_BUCKET_NAME=$BUCKET_NAME"
|
||||
```
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
|
||||
@@ -29,8 +29,6 @@ tmp/
|
||||
# Google Cloud
|
||||
.gcloud/
|
||||
*.json
|
||||
# Allow i18n translation files
|
||||
!translations/*.json
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
|
||||
+4
-11
@@ -22,19 +22,12 @@ WORKDIR /app
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
|
||||
# This ensures new modules (like style_loader.py) are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is vendored in lyrics_transcriber_temp/ and bundled via pyproject.toml
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
|
||||
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
|
||||
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
|
||||
# CPU images: normal install resolves all deps from pyproject.toml.
|
||||
RUN if [ -n "$MODEL_DIR" ]; then \
|
||||
pip install --no-cache-dir --no-deps -e .; \
|
||||
else \
|
||||
pip install --no-cache-dir -e .; \
|
||||
fi
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
+7
-27
@@ -1,15 +1,8 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock or this Dockerfile changes
|
||||
#
|
||||
# This image is rebuilt only when poetry.lock changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 3. pip install (the slow part - cached unless pyproject.toml changes)
|
||||
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 5. spaCy model download (cached unless pip install layer changes)
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
@@ -23,7 +16,6 @@ RUN apt-get update && apt-get install -y \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
chromium \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
@@ -58,26 +50,14 @@ RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-stati
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
# Copy dependency files and source code needed for pip install
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds, so partial
|
||||
# dependency updates only download changed packages instead of everything
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
|
||||
@@ -1,136 +0,0 @@
|
||||
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
|
||||
# This image is rebuilt only when deps or models change (infrequently).
|
||||
#
|
||||
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
||||
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. CUDA library path configuration
|
||||
# 3. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
|
||||
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 6. spaCy model download (cached unless pip install layer changes)
|
||||
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
|
||||
|
||||
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
|
||||
|
||||
# Prevent interactive prompts during package installation
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python 3.11 from deadsnakes PPA and system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
software-properties-common \
|
||||
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends \
|
||||
# Python 3.11
|
||||
python3.11 \
|
||||
python3.11-dev \
|
||||
python3.11-venv \
|
||||
# Audio libraries
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
# Build tools
|
||||
build-essential \
|
||||
# Utilities
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& python3.11 --version
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install FFmpeg from Ubuntu package
|
||||
# The audio worker only uses FFmpeg for format conversion and padding, not
|
||||
# encoding-intensive tasks, so the Debian package is sufficient. The static
|
||||
# build from johnvansickle.com is used for the encoding worker instead.
|
||||
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set Python 3.11 as default and install pip
|
||||
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
|
||||
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
|
||||
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
|
||||
|
||||
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
|
||||
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
|
||||
|
||||
# Environment configuration for CUDA and Python
|
||||
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
|
||||
PATH=/usr/local/cuda/bin:$PATH \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
MODEL_DIR=/models
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all karaoke-gen Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
|
||||
# This is separate from pip install -e . to keep layer caching granular
|
||||
#
|
||||
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
|
||||
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
|
||||
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
||||
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
||||
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
||||
pip install "audio-separator[gpu]"
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# Must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Create models directory and pre-download ensemble preset models
|
||||
# Baking models into the image eliminates cold-start model download latency.
|
||||
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
|
||||
RUN mkdir -p /models
|
||||
COPY backend/scripts/download_models.py /tmp/download_models.py
|
||||
RUN python /tmp/download_models.py /models && \
|
||||
rm /tmp/download_models.py && \
|
||||
ls -lh /models/
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
|
||||
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
+114
-14
@@ -113,19 +113,6 @@ async def require_auth(
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# If token has a tenant_id, override the hostname-based tenant detection.
|
||||
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
|
||||
# We must also load tenant_config so downstream handlers (e.g. job creation)
|
||||
# that rely on tenant_config for setting tenant_id on jobs work correctly.
|
||||
if auth_result.tenant_id:
|
||||
request.state.tenant_id = auth_result.tenant_id
|
||||
if not getattr(request.state, "tenant_config", None):
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(auth_result.tenant_id)
|
||||
if tenant_config and tenant_config.is_active:
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
@@ -135,7 +122,6 @@ async def require_auth(
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"tenant_id": auth_result.tenant_id,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
@@ -355,3 +341,117 @@ def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
def require_instrumental_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create an instrumental review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific instrumental token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates instrumental review access
|
||||
"""
|
||||
async def require_instrumental_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
instrumental_token: Optional[str] = Query(None, description="Job-specific instrumental review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate instrumental review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "instrumental_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Instrumental access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's instrumental review"
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try instrumental token
|
||||
if instrumental_token:
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate instrumental token matches
|
||||
if job.instrumental_token and secrets.compare_digest(job.instrumental_token, instrumental_token):
|
||||
# Check expiry if set
|
||||
if job.instrumental_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.instrumental_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Instrumental token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Instrumental review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Instrumental access granted via instrumental_token for job {job_id}")
|
||||
return job_id, "instrumental_token"
|
||||
else:
|
||||
logger.warning(f"Invalid instrumental token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid instrumental_token for this job."
|
||||
)
|
||||
|
||||
return require_instrumental_auth
|
||||
|
||||
|
||||
# Default instance for most instrumental review endpoints
|
||||
require_instrumental_auth = require_instrumental_auth_factory()
|
||||
+97
-3468
File diff suppressed because it is too large.
Load diff
+222
-538
File diff suppressed because it is too large.
Load diff
+35
-41
@@ -9,7 +9,7 @@ Provides endpoints for:
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
@@ -18,7 +18,6 @@ from backend.services.credential_manager import (
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
@@ -126,16 +125,15 @@ async def get_credentials_status():
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
async def get_service_credential_status(service: str, request: Request):
|
||||
async def get_service_credential_status(service: str):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
@@ -143,7 +141,7 @@ async def get_service_credential_status(service: str, request: Request):
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
|
||||
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
@@ -155,43 +153,42 @@ async def get_service_credential_status(service: str, request: Request):
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
async def validate_credentials(request: CredentialValidationRequest, http_request: Request):
|
||||
async def validate_credentials(request: CredentialValidationRequest):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
|
||||
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
|
||||
)
|
||||
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message=t(locale, "auth.validCredentials")
|
||||
message="All requested credentials are valid"
|
||||
)
|
||||
|
||||
|
||||
@@ -200,41 +197,40 @@ async def validate_credentials(request: CredentialValidationRequest, http_reques
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
@@ -263,36 +259,35 @@ async def poll_youtube_device_auth(device_code: str):
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
@@ -316,20 +311,19 @@ async def poll_gdrive_device_auth(device_code: str):
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=t(locale, "auth.discordAlertNotConfigured")
|
||||
detail="Discord alert webhook not configured"
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
@@ -1,163 +0,0 @@
|
||||
"""
|
||||
Catalog API routes for song/artist lookup and community version detection.
|
||||
|
||||
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
|
||||
and checks karaokenerds.com for existing community karaoke versions.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/catalog", tags=["catalog"])
|
||||
|
||||
# Simple per-user rate limiting: 20 requests/minute
|
||||
# { user_email: [timestamp, timestamp, ...] }
|
||||
_rate_limit_windows: dict[str, list[float]] = {}
|
||||
RATE_LIMIT_MAX_REQUESTS = 20
|
||||
RATE_LIMIT_WINDOW_SECONDS = 60
|
||||
|
||||
|
||||
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
|
||||
"""Check per-user rate limit. Raises 429 if exceeded."""
|
||||
now = time.monotonic()
|
||||
window_start = now - RATE_LIMIT_WINDOW_SECONDS
|
||||
|
||||
# Get or create window for this user
|
||||
if user_email not in _rate_limit_windows:
|
||||
_rate_limit_windows[user_email] = []
|
||||
|
||||
# Remove expired entries
|
||||
timestamps = _rate_limit_windows[user_email]
|
||||
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
|
||||
|
||||
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
|
||||
)
|
||||
|
||||
_rate_limit_windows[user_email].append(now)
|
||||
|
||||
|
||||
# --- Response Models ---
|
||||
|
||||
|
||||
class ArtistResult(BaseModel):
|
||||
"""Artist search result from MusicBrainz/Spotify catalog."""
|
||||
name: str
|
||||
mbid: Optional[str] = None
|
||||
disambiguation: Optional[str] = None
|
||||
artist_type: Optional[str] = None
|
||||
spotify_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
genres: Optional[list[str]] = None
|
||||
tags: Optional[list[str]] = None
|
||||
|
||||
|
||||
class TrackResult(BaseModel):
|
||||
"""Track search result from Spotify catalog."""
|
||||
track_name: str
|
||||
artist_name: str
|
||||
track_id: Optional[str] = None
|
||||
artist_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
duration_ms: Optional[int] = None
|
||||
explicit: Optional[bool] = None
|
||||
|
||||
|
||||
class CommunityTrack(BaseModel):
|
||||
"""A community-approved karaoke track from karaokenerds."""
|
||||
brand_name: str
|
||||
brand_code: str
|
||||
youtube_url: str
|
||||
is_community: bool = True
|
||||
|
||||
|
||||
class CommunityCheckSong(BaseModel):
|
||||
"""A song with community karaoke tracks."""
|
||||
title: str
|
||||
artist: str
|
||||
community_tracks: list[CommunityTrack]
|
||||
|
||||
|
||||
class CommunityCheckRequest(BaseModel):
|
||||
"""Request body for community version check."""
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
|
||||
|
||||
class CommunityCheckResponse(BaseModel):
|
||||
"""Response for community version check."""
|
||||
has_community: bool
|
||||
songs: list[CommunityCheckSong]
|
||||
best_youtube_url: Optional[str] = None
|
||||
|
||||
|
||||
# --- Routes ---
|
||||
|
||||
|
||||
@router.get("/artists", response_model=list[ArtistResult])
|
||||
async def search_artists(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Artist name search query"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
|
||||
for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_artists
|
||||
results = await search_artists(q, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/tracks", response_model=list[TrackResult])
|
||||
async def search_tracks(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Track name search query"),
|
||||
artist: Optional[str] = Query(None, description="Filter by artist name"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for tracks by name, optionally filtered by artist.
|
||||
Returns canonical Spotify track data for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_tracks
|
||||
results = await search_tracks(q, artist, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.post("/community-check", response_model=CommunityCheckResponse)
|
||||
async def check_community_versions(
|
||||
request: Request,
|
||||
body: CommunityCheckRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Check if a song already has community-approved karaoke versions on YouTube
|
||||
via karaokenerds.com. Non-blocking — just informs the user.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.karaokenerds_service import check_community_versions
|
||||
result = await check_community_versions(body.artist, body.title)
|
||||
return result
|
||||
@@ -1,92 +0,0 @@
|
||||
"""POST /api/client-errors — ingest frontend crash reports.
|
||||
|
||||
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
|
||||
via the existing error-monitor pipeline — see
|
||||
``backend/services/error_monitor``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Request, HTTPException
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
|
||||
from backend.services.error_monitor.frontend_ingestion import (
|
||||
FrontendErrorReport,
|
||||
RateLimiter,
|
||||
build_pattern_data,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
|
||||
|
||||
_limiter = RateLimiter(max_per_minute=60)
|
||||
_adapter_singleton: ErrorPatternsAdapter | None = None
|
||||
|
||||
|
||||
def _get_adapter() -> ErrorPatternsAdapter:
|
||||
global _adapter_singleton
|
||||
if _adapter_singleton is None:
|
||||
_adapter_singleton = ErrorPatternsAdapter()
|
||||
return _adapter_singleton
|
||||
|
||||
|
||||
class ClientErrorPayload(BaseModel):
|
||||
message: str = Field(..., min_length=1, max_length=4000)
|
||||
stack: Optional[str] = Field(None, max_length=1_000_000)
|
||||
url: str = Field("", max_length=2048)
|
||||
user_agent: str = Field("", max_length=1024)
|
||||
release: str = Field("", max_length=64)
|
||||
user_email: Optional[str] = Field(None, max_length=320)
|
||||
viewport: Optional[dict] = None
|
||||
locale: str = Field("en", max_length=10)
|
||||
source: str = Field("unknown", max_length=64)
|
||||
# optional free-form extra for debugging. Capped server-side.
|
||||
extra: Optional[dict] = None
|
||||
|
||||
@field_validator("message", "url", "user_agent", "locale", "source")
|
||||
@classmethod
|
||||
def strip(cls, v: str) -> str:
|
||||
return (v or "").strip()
|
||||
|
||||
|
||||
class ClientErrorResponse(BaseModel):
|
||||
pattern_id: str
|
||||
is_new: bool
|
||||
|
||||
|
||||
@router.post("", response_model=ClientErrorResponse, status_code=202)
|
||||
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
if not _limiter.allow(client_ip, time.monotonic()):
|
||||
raise HTTPException(status_code=429, detail="too many reports")
|
||||
|
||||
report = FrontendErrorReport(
|
||||
message=payload.message,
|
||||
stack=payload.stack,
|
||||
url=payload.url,
|
||||
user_agent=payload.user_agent,
|
||||
release=payload.release,
|
||||
user_email=payload.user_email,
|
||||
viewport=payload.viewport,
|
||||
locale=payload.locale,
|
||||
extra=payload.extra,
|
||||
)
|
||||
pattern_data = build_pattern_data(report)
|
||||
|
||||
try:
|
||||
result = _get_adapter().upsert_pattern(pattern_data)
|
||||
except Exception: # pragma: no cover - Firestore transient errors
|
||||
logger.exception("Failed to upsert frontend error pattern")
|
||||
raise HTTPException(status_code=503, detail="storage unavailable")
|
||||
|
||||
logger.info(
|
||||
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
|
||||
result.pattern_id,
|
||||
result.is_new,
|
||||
payload.source,
|
||||
)
|
||||
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
|
||||
@@ -1,65 +0,0 @@
|
||||
"""
|
||||
Encoding worker lifecycle endpoints.
|
||||
|
||||
Provides warmup and heartbeat endpoints for the blue-green
|
||||
encoding worker VMs. Called by the frontend to ensure the
|
||||
primary VM is running before encoding requests.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from fastapi import APIRouter, Depends
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
|
||||
|
||||
# Lazy-loaded singleton
|
||||
_manager = None
|
||||
|
||||
|
||||
def get_worker_manager():
|
||||
global _manager
|
||||
if _manager is None:
|
||||
from google.cloud import compute_v1, firestore
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
db = firestore.Client(project=settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
_manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=settings.google_cloud_project,
|
||||
)
|
||||
return _manager
|
||||
|
||||
|
||||
@router.post("/warmup")
|
||||
async def warmup_encoding_worker(
|
||||
_admin=Depends(require_admin),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Start the primary encoding worker VM if it's stopped."""
|
||||
try:
|
||||
result = manager.ensure_primary_running()
|
||||
if result["started"]:
|
||||
logger.info(f"Started encoding worker VM: {result['vm_name']}")
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to warm up encoding worker: {e}")
|
||||
return {"started": False, "error": str(e)}
|
||||
|
||||
|
||||
@router.post("/heartbeat")
|
||||
async def heartbeat_encoding_worker(
|
||||
_admin=Depends(require_admin),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Update activity timestamp to prevent idle shutdown."""
|
||||
try:
|
||||
manager.update_activity()
|
||||
return {"status": "ok"}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update encoding worker heartbeat: {e}")
|
||||
return {"status": "error", "error": str(e)}
|
||||
+208
-686
File diff suppressed because it is too large.
Load diff
@@ -5,22 +5,14 @@ import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter, Depends
|
||||
from typing import Dict, Any, Optional, Tuple
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
|
||||
from backend.api.dependencies import optional_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
from backend.services.job_health_service import check_job_consistency_detailed
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -235,7 +227,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
@@ -246,239 +237,6 @@ async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/flacfetch")
|
||||
async def flacfetch_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check flacfetch service status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_flacfetch_service_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("version"),
|
||||
}
|
||||
|
||||
|
||||
def check_audio_separator_status() -> Dict[str, Any]:
|
||||
"""Check audio separator availability without calling the remote GPU service."""
|
||||
from backend.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.audio_separator_api_url:
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
pkg_version = version("audio-separator")
|
||||
except Exception:
|
||||
pkg_version = None
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": pkg_version,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/audio-separator")
|
||||
async def audio_separator_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Return the installed audio-separator package version.
|
||||
|
||||
Does NOT call the remote GPU service — that would cold-start an expensive
|
||||
Cloud Run GPU instance (~$0.07 per wake). Instead reports the locally
|
||||
installed package version, which matches what the audio worker uses.
|
||||
|
||||
No authentication required.
|
||||
"""
|
||||
from backend.config import get_settings
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.audio_separator_api_url:
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
pkg_version = version("audio-separator")
|
||||
except Exception:
|
||||
pkg_version = None
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": pkg_version,
|
||||
}
|
||||
|
||||
|
||||
def _get_encoding_worker_manager():
|
||||
"""Get the EncodingWorkerManager instance. Separated for testability."""
|
||||
try:
|
||||
encoding_service = get_encoding_service()
|
||||
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
|
||||
return encoding_service._worker_manager
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
@router.get("/health/system-status")
|
||||
async def system_status(
|
||||
auth: Optional[Tuple] = Depends(optional_auth),
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Aggregated system status for all services.
|
||||
|
||||
Returns basic info for everyone. Admin users get additional
|
||||
blue-green deployment details for the encoder.
|
||||
"""
|
||||
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
|
||||
|
||||
# Fetch all health statuses concurrently
|
||||
import asyncio
|
||||
encoding_task = check_encoding_worker_status()
|
||||
flacfetch_task = check_flacfetch_service_status()
|
||||
encoding_status, flacfetch_status = await asyncio.gather(
|
||||
encoding_task, flacfetch_task
|
||||
)
|
||||
separator_status = check_audio_separator_status()
|
||||
|
||||
# Build frontend service info (from env vars baked at build time — returned as-is)
|
||||
frontend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
|
||||
}
|
||||
|
||||
# Build backend service info
|
||||
backend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION,
|
||||
"deployed_at": STARTUP_TIME,
|
||||
}
|
||||
if COMMIT_SHA:
|
||||
backend_svc["commit_sha"] = COMMIT_SHA
|
||||
if PR_NUMBER:
|
||||
backend_svc["pr_number"] = PR_NUMBER
|
||||
if PR_TITLE:
|
||||
backend_svc["pr_title"] = PR_TITLE
|
||||
|
||||
# Build encoder service info
|
||||
encoder_svc = {
|
||||
"status": "ok" if encoding_status.get("available") else "offline",
|
||||
"version": encoding_status.get("wheel_version"),
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
}
|
||||
|
||||
# Build flacfetch service info
|
||||
flacfetch_svc = {
|
||||
"status": "ok" if flacfetch_status.get("available") else "offline",
|
||||
"version": flacfetch_status.get("version"),
|
||||
}
|
||||
|
||||
# Build separator service info
|
||||
separator_svc = {
|
||||
"status": "ok" if separator_status.get("available") else "offline",
|
||||
"version": separator_status.get("version"),
|
||||
}
|
||||
|
||||
# Admin-only: blue-green encoder details from Firestore
|
||||
if is_admin:
|
||||
manager = _get_encoding_worker_manager()
|
||||
if manager:
|
||||
try:
|
||||
config = manager.get_config()
|
||||
encoder_svc["admin_details"] = {
|
||||
"primary_vm": config.primary_vm,
|
||||
"primary_ip": config.primary_ip,
|
||||
"primary_version": config.primary_version,
|
||||
"primary_deployed_at": config.primary_deployed_at,
|
||||
"secondary_vm": config.secondary_vm,
|
||||
"secondary_ip": config.secondary_ip,
|
||||
"secondary_version": config.secondary_version,
|
||||
"secondary_deployed_at": config.secondary_deployed_at,
|
||||
"last_swap_at": config.last_swap_at,
|
||||
"deploy_in_progress": config.deploy_in_progress,
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
"queue_length": encoding_status.get("queue_length", 0),
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get blue-green config: {e}")
|
||||
|
||||
# Add error details for offline services
|
||||
if not flacfetch_status.get("available"):
|
||||
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
|
||||
if not separator_status.get("available"):
|
||||
separator_svc["admin_details"] = {"error": separator_status.get("error")}
|
||||
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
|
||||
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
|
||||
|
||||
return {
|
||||
"services": {
|
||||
"frontend": frontend_svc,
|
||||
"backend": backend_svc,
|
||||
"encoder": encoder_svc,
|
||||
"flacfetch": flacfetch_svc,
|
||||
"separator": separator_svc,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -542,131 +300,6 @@ async def detailed_health_check() -> Dict[str, Any]:
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
async def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/job-consistency")
|
||||
async def job_consistency_check(
|
||||
status: Optional[str] = None,
|
||||
limit: int = 50
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Check job consistency across the system.
|
||||
|
||||
This endpoint identifies jobs with state inconsistencies that may indicate
|
||||
bugs in the state machine or trigger logic. Use this to find stuck jobs
|
||||
or jobs in invalid states.
|
||||
|
||||
Args:
|
||||
status: Optional filter by job status (e.g., "pending", "downloading")
|
||||
limit: Maximum number of jobs to check (default 50)
|
||||
|
||||
Returns:
|
||||
Dictionary with:
|
||||
- total_checked: Number of jobs checked
|
||||
- inconsistent_count: Number of jobs with issues
|
||||
- inconsistent_jobs: List of job details with their issues
|
||||
- summary: Summary of issue types found
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
|
||||
# Query jobs to check
|
||||
query_params = {}
|
||||
if status:
|
||||
try:
|
||||
job_status = JobStatus(status)
|
||||
query_params['status'] = job_status
|
||||
except ValueError:
|
||||
return {
|
||||
"error": f"Invalid status: {status}",
|
||||
"valid_statuses": [s.value for s in JobStatus]
|
||||
}
|
||||
|
||||
# Get jobs from Firestore
|
||||
jobs = job_manager.list_jobs(limit=limit, **query_params)
|
||||
|
||||
# Check each job for consistency
|
||||
inconsistent_jobs = []
|
||||
issue_summary = {}
|
||||
|
||||
for job in jobs:
|
||||
result = check_job_consistency_detailed(job)
|
||||
if not result['is_healthy']:
|
||||
inconsistent_jobs.append(result)
|
||||
# Count issue types
|
||||
for issue in result['issues']:
|
||||
issue_type = issue.split(':')[0] if ':' in issue else issue
|
||||
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total_checked": len(jobs),
|
||||
"inconsistent_count": len(inconsistent_jobs),
|
||||
"inconsistent_jobs": inconsistent_jobs,
|
||||
"summary": issue_summary,
|
||||
"status_filter": status,
|
||||
"limit": limit,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
|
||||
@@ -25,7 +25,7 @@ from backend.workers.screens_worker import generate_screens
|
||||
from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult, UserType
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
@@ -76,18 +76,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Reject workers for jobs in terminal states (failed, cancelled, complete)
|
||||
# This prevents Cloud Tasks retries from triggering workers on dead jobs
|
||||
terminal_statuses = {'failed', 'cancelled', 'complete', 'prep_complete', 'error'}
|
||||
job_status = job.status.value if hasattr(job.status, 'value') else str(job.status)
|
||||
if job_status in terminal_statuses:
|
||||
logger.info(f"[job:{job_id}] Job in terminal state '{job_status}', skipping {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="skipped",
|
||||
job_id=job_id,
|
||||
message=f"Job is in terminal state '{job_status}', {worker_name} worker not needed"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
@@ -114,31 +102,6 @@ def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerR
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/sync-disposable-domains", response_model=WorkerResponse)
|
||||
async def sync_disposable_domains_endpoint(
|
||||
request: Request,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Sync external disposable domain blocklist. Called by Cloud Scheduler daily."""
|
||||
import asyncio
|
||||
from backend.services.disposable_domain_sync_service import (
|
||||
fetch_external_blocklist, sync_disposable_domains
|
||||
)
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import EmailValidationService
|
||||
|
||||
domains = await fetch_external_blocklist()
|
||||
db = get_firestore_client()
|
||||
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
return WorkerResponse(
|
||||
status="completed",
|
||||
job_id="sync-disposable-domains",
|
||||
message=f"Synced {result['external_count']} domains"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
@@ -325,7 +288,7 @@ async def trigger_render_video_worker(
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
Output: with_vocals.mkv in GCS
|
||||
Next state: INSTRUMENTAL_SELECTED (instrumental is now selected during review)
|
||||
Next state: AWAITING_INSTRUMENTAL_SELECTION
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
@@ -353,490 +316,15 @@ async def trigger_render_video_worker(
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters AWAITING_REVIEW state.
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
# Note: AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs only
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
|
||||
if getattr(job, 'made_for_you', False):
|
||||
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
|
||||
add_span_event("made_for_you_skip")
|
||||
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.post("/youtube-queue/process")
|
||||
async def process_youtube_upload_queue(
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Process queued YouTube uploads.
|
||||
|
||||
Called by Cloud Scheduler (hourly) to retry uploads that were deferred
|
||||
due to YouTube API quota exhaustion. Can also be triggered manually
|
||||
from the admin dashboard.
|
||||
|
||||
Processes queued uploads one at a time, stopping if quota is exhausted.
|
||||
"""
|
||||
from backend.workers.youtube_queue_processor import process_youtube_upload_queue as process_queue
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("YOUTUBE_QUEUE_PROCESS starting")
|
||||
add_span_attribute("operation", "youtube_queue_process")
|
||||
|
||||
# Run in background so the HTTP response returns quickly
|
||||
# (Cloud Scheduler has a 30min timeout but we don't want to block)
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_queue()
|
||||
logger.info(f"YOUTUBE_QUEUE_PROCESS complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"YOUTUBE_QUEUE_PROCESS failed: {e}")
|
||||
|
||||
background_tasks.add_task(_process)
|
||||
|
||||
add_span_event("queue_processing_started")
|
||||
return {
|
||||
"status": "started",
|
||||
"message": "YouTube upload queue processing started in background"
|
||||
}
|
||||
|
||||
|
||||
@router.post("/process-stale-reviews")
|
||||
async def process_stale_reviews_endpoint(
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Process stale review jobs — send reminders and expire old ones.
|
||||
|
||||
Called by Cloud Scheduler (hourly). Can also be triggered manually
|
||||
from the admin dashboard.
|
||||
|
||||
- Jobs in review for >= 24h get a reminder email
|
||||
- Jobs in review for >= 48h are auto-cancelled with credit refund
|
||||
"""
|
||||
from backend.workers.stale_review_processor import process_stale_reviews
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("STALE_REVIEW_PROCESS starting")
|
||||
add_span_attribute("operation", "stale_review_process")
|
||||
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_stale_reviews()
|
||||
logger.info(f"STALE_REVIEW_PROCESS complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"STALE_REVIEW_PROCESS failed: {e}")
|
||||
|
||||
background_tasks.add_task(_process)
|
||||
|
||||
add_span_event("stale_review_processing_started")
|
||||
return {
|
||||
"status": "started",
|
||||
"message": "Stale review processing started in background"
|
||||
}
|
||||
|
||||
|
||||
@router.post("/trigger-gdrive-validation")
|
||||
async def trigger_gdrive_validation_endpoint(
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger GDrive validation via the Cloud Function.
|
||||
|
||||
Called by a Cloud Tasks delayed task after a job completes and uploads
|
||||
to the public GDrive share. The 5-minute delay ensures E2E test cleanup
|
||||
finishes before the validator checks for sequence gaps.
|
||||
"""
|
||||
from backend.services.gdrive_validator_client import trigger_gdrive_validation
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info("GDRIVE_VALIDATION_TRIGGER starting (delayed post-job)")
|
||||
add_span_attribute("operation", "gdrive_validation_trigger")
|
||||
|
||||
try:
|
||||
result = trigger_gdrive_validation()
|
||||
if result is None:
|
||||
return {"status": "skipped", "message": "GDRIVE_VALIDATOR_URL not configured"}
|
||||
|
||||
status = result.get("status", "unknown")
|
||||
add_span_event("validation_complete", {"result_status": status})
|
||||
return {"status": status, "message": "GDrive validation completed", "result": result}
|
||||
except Exception as e:
|
||||
logger.exception(f"GDrive validation trigger failed: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "message": str(e)}
|
||||
|
||||
|
||||
@router.post("/recover-stuck-jobs")
|
||||
async def recover_stuck_jobs(
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Detect and recover jobs stuck in downloading_audio status.
|
||||
|
||||
Called by Cloud Scheduler (every 5 minutes) or manually from admin.
|
||||
Finds jobs stuck in DOWNLOADING_AUDIO for >10 minutes and fails them
|
||||
so they can be retried.
|
||||
"""
|
||||
from backend.services.job_health_service import check_job_consistency
|
||||
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
logger.info("RECOVER_STUCK_JOBS starting")
|
||||
add_span_attribute("operation", "recover_stuck_jobs")
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Query jobs in downloading_audio status
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
jobs_ref = job_manager.firestore.db.collection("jobs")
|
||||
stuck_query = jobs_ref.where(
|
||||
filter=FieldFilter("status", "==", "downloading_audio")
|
||||
).stream()
|
||||
|
||||
recovered = []
|
||||
for doc in stuck_query:
|
||||
job_data = doc.to_dict()
|
||||
job_id = job_data.get("job_id", doc.id)
|
||||
|
||||
# Use the health service to check if actually stuck
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
continue
|
||||
|
||||
issues = check_job_consistency(job)
|
||||
stuck_issues = [i for i in issues if "downloading_audio_stuck" in i]
|
||||
|
||||
if stuck_issues:
|
||||
logger.warning(f"[job:{job_id}] Recovering stuck download: {stuck_issues[0]}")
|
||||
job_manager.fail_job(
|
||||
job_id,
|
||||
"Audio download timed out (stuck >10 minutes). Use retry to re-attempt."
|
||||
)
|
||||
recovered.append(job_id)
|
||||
|
||||
logger.info(f"RECOVER_STUCK_JOBS complete: recovered={len(recovered)} jobs")
|
||||
add_span_event("recovery_complete", {"recovered_count": len(recovered)})
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"recovered_jobs": recovered,
|
||||
"recovered_count": len(recovered),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
return {"status": "healthy", "service": "karaoke-backend-internal"}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Test Webhook Endpoint (for E2E testing)
|
||||
# =============================================================================
|
||||
|
||||
class TestWebhookRequest(BaseModel):
|
||||
"""
|
||||
Request to simulate a Stripe webhook event for E2E testing.
|
||||
|
||||
This allows E2E tests to trigger payment flow logic without requiring
|
||||
actual Stripe checkout sessions or valid webhook signatures.
|
||||
"""
|
||||
event_type: str # e.g., "checkout.session.completed"
|
||||
session_id: str # Must start with "e2e-test-" prefix
|
||||
customer_email: str
|
||||
metadata: dict # order_type, package_id, credits, artist, title, etc.
|
||||
|
||||
|
||||
class TestWebhookResponse(BaseModel):
|
||||
"""Response from test webhook processing."""
|
||||
status: str # "processed", "already_processed", "error"
|
||||
job_id: Optional[str] = None # For made-for-you orders
|
||||
credits_added: Optional[int] = None # For credit purchases
|
||||
new_balance: Optional[int] = None # For credit purchases
|
||||
message: str
|
||||
|
||||
|
||||
@router.post("/test-webhook", response_model=TestWebhookResponse)
|
||||
async def test_webhook(
|
||||
request: TestWebhookRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Test endpoint that simulates Stripe webhook events for E2E testing.
|
||||
|
||||
SECURITY:
|
||||
- Protected by admin authentication (X-Admin-Token header)
|
||||
- Session IDs must start with "e2e-test-" prefix to prevent collision
|
||||
with real Stripe sessions
|
||||
- Only for E2E testing - bypasses Stripe signature verification
|
||||
|
||||
This endpoint reuses the same handler logic as the real webhook endpoint,
|
||||
ensuring E2E tests validate actual business logic.
|
||||
|
||||
Supported event types:
|
||||
- checkout.session.completed: Handles credit purchases and made-for-you orders
|
||||
|
||||
For credit purchases, metadata must include:
|
||||
- package_id: e.g., "1_credit"
|
||||
- credits: e.g., "1"
|
||||
- user_email: Email of user to credit
|
||||
|
||||
For made-for-you orders, metadata must include:
|
||||
- order_type: "made_for_you"
|
||||
- customer_email: Customer email for delivery
|
||||
- artist: Song artist
|
||||
- title: Song title
|
||||
- source_type: "search" or "youtube"
|
||||
- youtube_url: (optional) If source_type is "youtube"
|
||||
- notes: (optional) Customer notes
|
||||
"""
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.api.routes.users import _handle_made_for_you_order
|
||||
|
||||
# Validate session_id prefix for safety
|
||||
if not request.session_id.startswith("e2e-test-"):
|
||||
logger.warning(f"Test webhook rejected: session_id '{request.session_id}' missing required prefix")
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Session ID must start with 'e2e-test-' prefix for test webhooks"
|
||||
)
|
||||
|
||||
logger.info(f"TEST_WEBHOOK event_type={request.event_type} session_id={request.session_id}")
|
||||
add_span_attribute("event_type", request.event_type)
|
||||
add_span_attribute("session_id", request.session_id)
|
||||
add_span_attribute("is_test_webhook", True)
|
||||
|
||||
user_service = get_user_service()
|
||||
email_service = get_email_service()
|
||||
stripe_service = get_stripe_service()
|
||||
|
||||
if request.event_type == "checkout.session.completed":
|
||||
session_id = request.session_id
|
||||
metadata = request.metadata
|
||||
|
||||
# Idempotency check: Skip if this session was already processed
|
||||
if user_service.is_stripe_session_processed(session_id):
|
||||
logger.info(f"Test webhook: session {session_id} already processed")
|
||||
return TestWebhookResponse(
|
||||
status="already_processed",
|
||||
message=f"Session {session_id} was already processed"
|
||||
)
|
||||
|
||||
# Check if this is a made-for-you order
|
||||
if metadata.get("order_type") == "made_for_you":
|
||||
try:
|
||||
# Call the same handler used by the real webhook
|
||||
await _handle_made_for_you_order(
|
||||
session_id=session_id,
|
||||
metadata=metadata,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
)
|
||||
|
||||
# Get the job ID from the most recent job for this customer
|
||||
# The handler creates a job, so we need to find it
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
db = user_service.db
|
||||
# Look for the job by session_id pattern in state_data or by customer_email
|
||||
# Since the job was just created, query by customer_email and made_for_you flag
|
||||
customer_email = metadata.get("customer_email", "")
|
||||
jobs_query = db.collection("jobs").where(
|
||||
filter=FieldFilter("customer_email", "==", customer_email)
|
||||
).where(
|
||||
filter=FieldFilter("made_for_you", "==", True)
|
||||
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(1)
|
||||
|
||||
jobs = list(jobs_query.stream())
|
||||
job_id = jobs[0].to_dict().get("job_id") if jobs else None
|
||||
|
||||
logger.info(f"Test webhook: made-for-you order processed, job_id={job_id}")
|
||||
return TestWebhookResponse(
|
||||
status="processed",
|
||||
job_id=job_id,
|
||||
message=f"Made-for-you order created successfully"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception(f"Test webhook: error processing made-for-you order: {e}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Error processing made-for-you order: {str(e)}"
|
||||
)
|
||||
else:
|
||||
# Handle regular credit purchase
|
||||
# Build a synthetic session object that matches Stripe's format
|
||||
synthetic_session = {
|
||||
"id": session_id,
|
||||
"customer_email": request.customer_email,
|
||||
"metadata": metadata,
|
||||
}
|
||||
|
||||
success, user_email, credits, msg = stripe_service.handle_checkout_completed(
|
||||
synthetic_session
|
||||
)
|
||||
|
||||
if not success:
|
||||
logger.warning(f"Test webhook: credit purchase validation failed: {msg}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=msg
|
||||
)
|
||||
|
||||
if user_email and credits > 0:
|
||||
# Add credits to user account
|
||||
ok, new_balance, credit_msg = user_service.add_credits(
|
||||
email=user_email,
|
||||
amount=credits,
|
||||
reason="stripe_purchase",
|
||||
stripe_session_id=session_id,
|
||||
)
|
||||
|
||||
if ok:
|
||||
# Send confirmation email (same as real webhook)
|
||||
email_service.send_credits_added(user_email, credits, new_balance)
|
||||
logger.info(f"Test webhook: added {credits} credits to {user_email}, new balance: {new_balance}")
|
||||
return TestWebhookResponse(
|
||||
status="processed",
|
||||
credits_added=credits,
|
||||
new_balance=new_balance,
|
||||
message=f"Added {credits} credits to {user_email}"
|
||||
)
|
||||
else:
|
||||
logger.error(f"Test webhook: failed to add credits: {credit_msg}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Failed to add credits: {credit_msg}"
|
||||
)
|
||||
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message="Invalid credit purchase data"
|
||||
)
|
||||
else:
|
||||
# Unsupported event type
|
||||
logger.warning(f"Test webhook: unsupported event type '{request.event_type}'")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Unsupported event type: {request.event_type}"
|
||||
)
|
||||
|
||||
+528
-1412
File diff suppressed because it is too large.
Load diff
@@ -1,263 +0,0 @@
|
||||
"""
|
||||
Push Notification API routes.
|
||||
|
||||
Provides endpoints for managing Web Push notification subscriptions:
|
||||
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
|
||||
- POST /api/push/subscribe: Register a push subscription
|
||||
- POST /api/push/unsubscribe: Remove a push subscription
|
||||
- GET /api/push/subscriptions: List user's subscriptions
|
||||
- POST /api/push/test: Send a test notification (admin only)
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/push", tags=["push"])
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
|
||||
class VapidPublicKeyResponse(BaseModel):
|
||||
"""Response containing VAPID public key."""
|
||||
enabled: bool
|
||||
vapid_public_key: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeRequest(BaseModel):
|
||||
"""Request to subscribe to push notifications."""
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
tenant_id: Optional[str] = None # Tenant scope for notification isolation
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
"""Response after subscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnsubscribeRequest(BaseModel):
|
||||
"""Request to unsubscribe from push notifications."""
|
||||
endpoint: str
|
||||
|
||||
|
||||
class UnsubscribeResponse(BaseModel):
|
||||
"""Response after unsubscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class SubscriptionInfo(BaseModel):
|
||||
"""Information about a push subscription."""
|
||||
endpoint: str
|
||||
device_name: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_used_at: Optional[str] = None
|
||||
|
||||
|
||||
class SubscriptionsListResponse(BaseModel):
|
||||
"""Response containing user's subscriptions."""
|
||||
subscriptions: List[SubscriptionInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
"""Request to send a test notification."""
|
||||
title: Optional[str] = "Test Notification"
|
||||
body: Optional[str] = "This is a test push notification from Karaoke Generator"
|
||||
|
||||
|
||||
class TestNotificationResponse(BaseModel):
|
||||
"""Response after sending test notification."""
|
||||
status: str
|
||||
sent_count: int
|
||||
message: str
|
||||
|
||||
|
||||
# Routes
|
||||
|
||||
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
|
||||
async def get_vapid_public_key():
|
||||
"""
|
||||
Get the VAPID public key for push subscription.
|
||||
|
||||
This endpoint is public - no authentication required.
|
||||
Returns the public key needed for client-side PushManager.subscribe().
|
||||
"""
|
||||
settings = get_settings()
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
if not settings.enable_push_notifications:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
public_key = push_service.get_public_key()
|
||||
if not public_key:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
return VapidPublicKeyResponse(
|
||||
enabled=True,
|
||||
vapid_public_key=public_key
|
||||
)
|
||||
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
subscribe_request: SubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Register a push notification subscription for the current user.
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
|
||||
Subscriptions are scoped by tenant_id to prevent cross-tenant
|
||||
notification leakage. The tenant is resolved from:
|
||||
1. Request body tenant_id field (explicit from frontend)
|
||||
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
|
||||
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
|
||||
|
||||
# Resolve tenant_id: prefer explicit from body, fall back to middleware
|
||||
tenant_id = subscribe_request.tenant_id
|
||||
if tenant_id is None:
|
||||
tenant_id = getattr(http_request.state, "tenant_id", None)
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=subscribe_request.endpoint,
|
||||
keys=subscribe_request.keys,
|
||||
device_name=subscribe_request.device_name,
|
||||
tenant_id=tenant_id or None,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionRegistered")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Remove a push notification subscription.
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
success = await push_service.remove_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint
|
||||
)
|
||||
|
||||
if not success:
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionNotFound")
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionRemoved")
|
||||
)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
List all push notification subscriptions for the current user.
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
|
||||
|
||||
return SubscriptionsListResponse(
|
||||
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
|
||||
count=len(subscriptions)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Send a test push notification to the current user's devices.
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
sent_count = await push_service.send_push(
|
||||
user_email=auth_result.user_email,
|
||||
title=request.title or "Test Notification",
|
||||
body=request.body or "This is a test push notification",
|
||||
url="/app/",
|
||||
tag="test"
|
||||
)
|
||||
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
|
||||
)
|
||||
@@ -1,346 +0,0 @@
|
||||
"""
|
||||
Admin API routes for blocklist and YouTube queue management.
|
||||
|
||||
Handles:
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- YouTube upload queue management
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class YouTubeQueueEntry(BaseModel):
|
||||
"""A queued YouTube upload entry."""
|
||||
job_id: str
|
||||
status: str
|
||||
reason: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
brand_code: Optional[str] = None
|
||||
queued_at: Optional[datetime] = None
|
||||
attempts: int = 0
|
||||
max_attempts: int = 5
|
||||
last_error: Optional[str] = None
|
||||
youtube_url: Optional[str] = None
|
||||
notification_sent: bool = False
|
||||
|
||||
|
||||
class YouTubeQueueListResponse(BaseModel):
|
||||
"""List of YouTube upload queue entries."""
|
||||
entries: List[YouTubeQueueEntry]
|
||||
stats: dict
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data with domain source separation."""
|
||||
external_domains: List[str]
|
||||
manual_domains: List[str]
|
||||
allowlisted_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
last_sync_at: Optional[datetime] = None
|
||||
last_sync_count: Optional[int] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
updated_by: Optional[str] = None
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
"""Request to add/remove a domain."""
|
||||
domain: str
|
||||
|
||||
|
||||
class EmailRequest(BaseModel):
|
||||
"""Request to add/remove an email."""
|
||||
email: str
|
||||
|
||||
|
||||
class IPRequest(BaseModel):
|
||||
"""Request to add/remove an IP."""
|
||||
ip_address: str
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/blocklists", response_model=BlocklistsResponse)
|
||||
async def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all blocklist data with domain source separation."""
|
||||
email_validation = get_email_validation_service()
|
||||
raw_data = email_validation.get_blocklist_raw_data()
|
||||
return BlocklistsResponse(**raw_data)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
async def add_disposable_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.domainAddedToBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_disposable_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.domainRemovedFromBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
|
||||
async def add_allowlisted_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the allowlist (overrides external blocklist)."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_allowlisted_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the allowlist."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.post("/blocklists/sync", response_model=SuccessResponse)
|
||||
async def trigger_sync(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Manually trigger a sync of the external disposable domain blocklist."""
|
||||
import asyncio
|
||||
from backend.services.disposable_domain_sync_service import (
|
||||
fetch_external_blocklist, sync_disposable_domains
|
||||
)
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
|
||||
domains = await fetch_external_blocklist()
|
||||
db = get_firestore_client()
|
||||
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
|
||||
async def add_blocked_email(
|
||||
request: EmailRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an email to the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.emailAddedToBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
|
||||
async def remove_blocked_email(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.emailRemovedFromBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
|
||||
async def add_blocked_ip(
|
||||
request: IPRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
|
||||
async def remove_blocked_ip(
|
||||
ip_address: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# YouTube Upload Queue Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
|
||||
async def get_youtube_queue(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get YouTube upload queue entries and stats.
|
||||
|
||||
Returns all queue entries (any status) and aggregate statistics.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
entries = queue_service.get_all_queue_entries(limit=50)
|
||||
stats = queue_service.get_queue_stats()
|
||||
|
||||
return YouTubeQueueListResponse(
|
||||
entries=[YouTubeQueueEntry(**entry) for entry in entries],
|
||||
stats=stats,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
|
||||
async def retry_youtube_upload(
|
||||
job_id: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Manually retry a failed YouTube upload.
|
||||
|
||||
Resets the entry back to 'queued' status with 0 attempts.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
|
||||
if not queue_service.retry_upload(job_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/youtube-queue/process", response_model=SuccessResponse)
|
||||
async def trigger_youtube_queue_processing(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Manually trigger YouTube upload queue processing.
|
||||
|
||||
Same as the scheduled processor but triggered from the admin dashboard.
|
||||
Runs in the background and returns immediately.
|
||||
"""
|
||||
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
|
||||
import asyncio
|
||||
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_youtube_upload_queue()
|
||||
logger.info(f"Manual YouTube queue processing complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"Manual YouTube queue processing failed: {e}")
|
||||
|
||||
# Fire and forget
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(_process())
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.youtubeQueueProcessingStarted")
|
||||
)
|
||||
@@ -1,374 +0,0 @@
|
||||
"""
|
||||
Referral API routes.
|
||||
|
||||
Handles:
|
||||
- Public referral interstitial (click tracking)
|
||||
- Authenticated referral dashboard
|
||||
- Stripe Connect onboarding
|
||||
- Admin vanity link creation and link management
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.models.referral import (
|
||||
CreateReferralLinkRequest,
|
||||
UpdateReferralLinkRequest,
|
||||
ReferralInterstitialResponse,
|
||||
)
|
||||
from backend.services.flyer_service import get_flyer_service, FlyerError
|
||||
from backend.services.referral_service import get_referral_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/referrals", tags=["referrals"])
|
||||
|
||||
|
||||
class GenerateFlyerRequest(BaseModel):
|
||||
theme: str = Field(..., pattern="^(light|dark)$")
|
||||
qr_data_url: str = Field(..., max_length=500_000)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Public endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
|
||||
async def get_referral_interstitial(code: str, no_track: bool = False):
|
||||
"""
|
||||
Public endpoint: get interstitial data for a referral link.
|
||||
|
||||
Increments click count unless no_track=true. Returns valid=False if code not found.
|
||||
"""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
return ReferralInterstitialResponse(valid=False)
|
||||
|
||||
# Track the click (skip when fetching for display purposes)
|
||||
if not no_track:
|
||||
try:
|
||||
service.increment_clicks(code)
|
||||
except Exception:
|
||||
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
|
||||
|
||||
return ReferralInterstitialResponse(
|
||||
valid=True,
|
||||
referral_code=link.code,
|
||||
referrer_display_name=link.display_name,
|
||||
discount_percent=link.discount_percent,
|
||||
discount_duration_days=link.discount_duration_days,
|
||||
custom_message=link.custom_message,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Authenticated user endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
async def get_my_dashboard(auth=Depends(require_auth)):
|
||||
"""Get referral dashboard data for the authenticated user."""
|
||||
service = get_referral_service()
|
||||
return service.get_dashboard_data(auth.user_email)
|
||||
|
||||
|
||||
@router.put("/me")
|
||||
async def update_my_link(
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Update display_name and custom_message on the user's own referral link."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
update_fields = {}
|
||||
if updates.display_name is not None:
|
||||
update_fields["display_name"] = updates.display_name
|
||||
if updates.custom_message is not None:
|
||||
update_fields["custom_message"] = updates.custom_message
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(link.code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/me/connect")
|
||||
async def start_stripe_connect(auth=Depends(require_auth)):
|
||||
"""Start Stripe Connect onboarding for the authenticated user."""
|
||||
# Check if user already has a Connect account
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(auth.user_email)
|
||||
if user and user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
|
||||
|
||||
service = get_referral_service()
|
||||
account_id, onboarding_url = service.create_connect_account(auth.user_email)
|
||||
|
||||
if not account_id:
|
||||
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
|
||||
|
||||
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
|
||||
|
||||
return {"account_id": account_id, "onboarding_url": onboarding_url}
|
||||
|
||||
|
||||
@router.post("/me/connect/dashboard-link")
|
||||
async def get_connect_dashboard_link(auth=Depends(require_auth)):
|
||||
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_login_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
@router.post("/me/connect/update-link")
|
||||
async def get_connect_update_link(auth=Depends(require_auth)):
|
||||
"""Get a link to update Stripe Connect account info (re-onboarding)."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_update_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create update link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
class VanityRequest(BaseModel):
|
||||
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
|
||||
|
||||
|
||||
@router.post("/me/vanity-request")
|
||||
async def request_vanity_url(
|
||||
request: VanityRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Request a custom vanity referral URL. Sends email to admin for review."""
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
# Check if code is already taken
|
||||
existing = service.get_link_by_code(request.desired_code)
|
||||
if existing:
|
||||
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
|
||||
|
||||
email_service = get_email_service()
|
||||
email_service.provider.send_email(
|
||||
to_email="andrew@nomadkaraoke.com",
|
||||
subject=f"Vanity referral URL request: {request.desired_code}",
|
||||
html_content=f"""
|
||||
<h2>Vanity Referral URL Request</h2>
|
||||
<p><strong>User:</strong> {auth.user_email}</p>
|
||||
<p><strong>Current code:</strong> {link.code}</p>
|
||||
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
|
||||
<p>To approve, create the vanity link in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
|
||||
""",
|
||||
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
|
||||
)
|
||||
|
||||
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
|
||||
return {"ok": True, "message": "Request sent"}
|
||||
|
||||
|
||||
@router.post("/me/flyer")
|
||||
async def generate_flyer(
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Generate a personalized referral flyer PDF."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Admin endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.post("/admin/vanity")
|
||||
async def create_vanity_link(
|
||||
request: CreateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: create a vanity referral link."""
|
||||
if not request.vanity_code:
|
||||
raise HTTPException(status_code=400, detail="vanity_code is required")
|
||||
|
||||
service = get_referral_service()
|
||||
success, link, message = service.create_vanity_link(
|
||||
code=request.vanity_code,
|
||||
owner_email=request.owner_email,
|
||||
display_name=request.display_name,
|
||||
custom_message=request.custom_message,
|
||||
discount_percent=request.discount_percent,
|
||||
kickback_percent=request.kickback_percent,
|
||||
discount_duration_days=request.discount_duration_days,
|
||||
earning_duration_days=request.earning_duration_days,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "code": link.code, "message": message}
|
||||
|
||||
|
||||
@router.get("/admin/links")
|
||||
async def list_links(
|
||||
limit: int = Query(50, ge=1, le=200),
|
||||
offset: int = Query(0, ge=0),
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: list all referral links."""
|
||||
service = get_referral_service()
|
||||
links = service.list_links(limit=limit, offset=offset)
|
||||
return {
|
||||
"links": [link.model_dump(mode="json") for link in links],
|
||||
"count": len(links),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/admin/apply-discount")
|
||||
async def admin_apply_discount(
|
||||
request: Request,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: apply a referral discount to any user, even without a referral link."""
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
body = await request.json()
|
||||
user_email = body.get("email", "").lower()
|
||||
discount_percent = body.get("discount_percent", 10)
|
||||
duration_days = body.get("duration_days", 30)
|
||||
referral_code = body.get("referral_code", "admin-grant")
|
||||
|
||||
if not user_email:
|
||||
raise HTTPException(status_code=400, detail="Email is required")
|
||||
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(user_email)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
now = datetime.utcnow()
|
||||
|
||||
user_service.update_user(
|
||||
user_email,
|
||||
referred_by_code=referral_code,
|
||||
referred_at=now,
|
||||
referral_discount_expires_at=now + timedelta(days=duration_days),
|
||||
)
|
||||
|
||||
# Create/ensure the Stripe coupon exists for this discount percent
|
||||
service = get_referral_service()
|
||||
service.get_or_create_stripe_coupon(discount_percent)
|
||||
|
||||
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"message": f"Applied {discount_percent}% discount for {duration_days} days",
|
||||
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.put("/admin/links/{code}")
|
||||
async def update_link(
|
||||
code: str,
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: update any referral link's settings."""
|
||||
service = get_referral_service()
|
||||
|
||||
update_fields = {
|
||||
k: v for k, v in updates.model_dump().items() if v is not None
|
||||
}
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=404, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/admin/links/{code}/flyer")
|
||||
async def admin_generate_flyer(
|
||||
code: str,
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: generate a personalized referral flyer PDF for any code."""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
|
||||
},
|
||||
)
|
||||
+161
-1580
File diff suppressed because it is too large.
Load diff
@@ -1,121 +0,0 @@
|
||||
"""
|
||||
Tenant API routes for white-label portal configuration.
|
||||
|
||||
These endpoints are public (no auth required) since the frontend needs
|
||||
to fetch tenant branding before the user logs in.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
|
||||
|
||||
|
||||
@router.get("/config", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config(
|
||||
request: Request,
|
||||
tenant: Optional[str] = Query(
|
||||
None,
|
||||
description="Tenant ID override (for development). If not provided, detected from subdomain.",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Get tenant configuration for the current portal.
|
||||
|
||||
This endpoint detects the tenant from:
|
||||
1. Query parameter `tenant` (for development/testing)
|
||||
2. X-Tenant-ID header (set by frontend)
|
||||
3. Host header subdomain
|
||||
|
||||
Returns the public tenant configuration (branding, features, defaults)
|
||||
or indicates this is the default Nomad Karaoke portal.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
|
||||
# Priority 1: Query parameter (for dev/testing)
|
||||
tenant_id = tenant
|
||||
|
||||
# Priority 2: X-Tenant-ID header
|
||||
if not tenant_id:
|
||||
tenant_id = request.headers.get("X-Tenant-ID")
|
||||
|
||||
# Priority 3: Detect from Host header
|
||||
if not tenant_id:
|
||||
host = request.headers.get("Host", "")
|
||||
# Check if this is a tenant subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
if host and "nomadkaraoke.com" in host.lower():
|
||||
parts = host.lower().split(".")
|
||||
# Skip known non-tenant subdomains
|
||||
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
|
||||
potential_tenant = parts[0]
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
tenant_id = potential_tenant
|
||||
|
||||
# If no tenant detected, return default config
|
||||
if not tenant_id:
|
||||
logger.debug("No tenant detected, returning default config")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Load tenant config
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
logger.warning(f"Tenant not found: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
logger.info(f"Returning config for tenant: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config_by_id(tenant_id: str):
|
||||
"""
|
||||
Get tenant configuration by explicit tenant ID.
|
||||
|
||||
This is useful for admin tools or debugging.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Check if tenant is active
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/asset/{tenant_id}/{asset_name}")
|
||||
async def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
This redirects to the signed GCS URL for the asset.
|
||||
"""
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
tenant_service = get_tenant_service()
|
||||
url = tenant_service.get_asset_url(tenant_id, asset_name)
|
||||
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
@@ -9,11 +9,10 @@ before job creation.
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -21,7 +20,7 @@ router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes(request: Request) -> ThemesListResponse:
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
@@ -30,7 +29,6 @@ async def list_themes(request: Request) -> ThemesListResponse:
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
@@ -39,12 +37,12 @@ async def list_themes(request: Request) -> ThemesListResponse:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=t(locale, "themes.themesLoadError")
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
@@ -60,7 +58,6 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
@@ -68,7 +65,7 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
@@ -78,12 +75,12 @@ async def get_theme(theme_id: str, request: Request) -> ThemeDetailResponse:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=t(locale, "themes.themeLoadError")
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str, request: Request) -> dict:
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
@@ -99,21 +96,20 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t(locale, "themes.themeNotFound", theme_id=theme_id)
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t(locale, "themes.previewNotAvailable", theme_id=theme_id)
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
@@ -123,12 +119,12 @@ async def get_theme_preview(theme_id: str, request: Request) -> dict:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=t(locale, "themes.previewLoadError")
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str, request: Request) -> dict:
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
@@ -145,14 +141,13 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t(locale, "themes.youtubeDescriptionNotFound", theme_id=theme_id)
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
@@ -163,5 +158,5 @@ async def get_theme_youtube_description(theme_id: str, request: Request) -> dict
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=t(locale, "themes.youtubeDescriptionLoadError")
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
+368
-1181
File diff suppressed because it is too large.
Load diff
@@ -1,35 +0,0 @@
|
||||
# Cloud Build config for GPU base image
|
||||
#
|
||||
# Builds the CUDA + PyTorch + models base image used by the audio worker.
|
||||
# This image is large (~10GB) so builds take 15-30 minutes.
|
||||
#
|
||||
# Trigger manually:
|
||||
# gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
||||
#
|
||||
# The app-layer Dockerfile uses this as its BASE_IMAGE for audio worker deploys.
|
||||
|
||||
steps:
|
||||
- name: 'gcr.io/cloud-builders/docker'
|
||||
args:
|
||||
- 'build'
|
||||
- '-f'
|
||||
- 'backend/Dockerfile.gpu-base'
|
||||
- '-t'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
|
||||
- '-t'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
|
||||
- '--build-arg'
|
||||
- 'BUILD_DATE=$_BUILD_DATE'
|
||||
- '--build-arg'
|
||||
- 'BUILD_ID=$BUILD_ID'
|
||||
- '.'
|
||||
|
||||
images:
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:latest'
|
||||
- 'us-east4-docker.pkg.dev/nomadkaraoke/karaoke-backend-gpu/karaoke-backend-gpu-base:$BUILD_ID'
|
||||
|
||||
timeout: '3600s'
|
||||
|
||||
options:
|
||||
machineType: 'E2_HIGHCPU_32'
|
||||
diskSizeGb: 100
|
||||
+2
-63
@@ -32,10 +32,6 @@ class Settings(BaseSettings):
|
||||
|
||||
# Authentication
|
||||
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
|
||||
scheduler_service_account: str = os.getenv(
|
||||
"SCHEDULER_SERVICE_ACCOUNT",
|
||||
"karaoke-backend@nomadkaraoke.iam.gserviceaccount.com"
|
||||
)
|
||||
|
||||
# Application
|
||||
environment: str = os.getenv("ENVIRONMENT", "development")
|
||||
@@ -45,15 +41,7 @@ class Settings(BaseSettings):
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Lyrics Correction Settings
|
||||
# Skip all auto-correction (both agentic AI and heuristic handlers).
|
||||
# When true, raw transcription goes directly to human review without any automatic fixes.
|
||||
# This is the default because auto-correction currently creates more work for reviewers.
|
||||
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
|
||||
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# Only used when skip_correction=false
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
@@ -61,12 +49,6 @@ class Settings(BaseSettings):
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# AI Credit Evaluation (anti-abuse gating for free credits)
|
||||
# When enabled, welcome and feedback credit grants are evaluated by Gemini
|
||||
# before granting. Fail-open: if evaluation fails, credits are granted anyway.
|
||||
credit_eval_enabled: bool = os.getenv("CREDIT_EVAL_ENABLED", "true").lower() in ("true", "1", "yes")
|
||||
credit_eval_model: str = os.getenv("CREDIT_EVAL_MODEL", "gemini-3.1-pro-preview")
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
@@ -86,16 +68,7 @@ class Settings(BaseSettings):
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Karaoke-Decide catalog API (for song/artist autocomplete)
|
||||
karaoke_decide_api_url: str = os.getenv("KARAOKE_DECIDE_API_URL", "https://decide.nomadkaraoke.com")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
@@ -110,13 +83,7 @@ class Settings(BaseSettings):
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
# Flacfetch client retry settings
|
||||
# Retries help handle transient network failures when calling the flacfetch service
|
||||
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
|
||||
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
|
||||
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
|
||||
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
@@ -128,41 +95,13 @@ class Settings(BaseSettings):
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
|
||||
# Private (non-published) track distribution settings
|
||||
default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")
|
||||
default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")
|
||||
|
||||
# YouTube Data API v3 Quota Configuration
|
||||
# Daily quota limit (YouTube grants 10,000 units/day by default)
|
||||
youtube_quota_daily_limit: int = int(os.getenv("YOUTUBE_QUOTA_DAILY_LIMIT", "10000"))
|
||||
# Estimated units consumed per upload (videos.insert=100 + search.list=100 + thumbnails.set=50 + buffer)
|
||||
youtube_quota_upload_cost: int = int(os.getenv("YOUTUBE_QUOTA_UPLOAD_COST", "300"))
|
||||
# Safety margin to reserve (avoid hitting exact limit)
|
||||
youtube_quota_safety_margin: int = int(os.getenv("YOUTUBE_QUOTA_SAFETY_MARGIN", "500"))
|
||||
# E2E test bypass key for rate limiting (set via secret in production)
|
||||
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
# Default CDG/TXT generation settings
|
||||
# When True, CDG and TXT packages are generated by default (when a theme is set)
|
||||
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
|
||||
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
|
||||
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Push Notifications Configuration
|
||||
# When enabled, users can subscribe to push notifications for job status updates
|
||||
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
|
||||
# Maximum number of push subscriptions per user (oldest removed when exceeded)
|
||||
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
|
||||
# VAPID subject (email or URL for push service to contact)
|
||||
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
|
||||
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
|
||||
@@ -1,94 +0,0 @@
|
||||
"""
|
||||
Custom exceptions for the karaoke generation backend.
|
||||
|
||||
These exceptions are used for structured error handling across the application,
|
||||
particularly for validation errors and credit checks.
|
||||
"""
|
||||
|
||||
|
||||
class EmailValidationError(Exception):
|
||||
"""
|
||||
Raised when email validation fails.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, reason: str = "invalid"):
|
||||
self.message = message
|
||||
self.reason = reason
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class IPBlockedError(Exception):
|
||||
"""
|
||||
Raised when a request comes from a blocked IP address.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
ip_address: The blocked IP address (may be partially masked)
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, ip_address: str = ""):
|
||||
self.message = message
|
||||
self.ip_address = ip_address
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class InsufficientCreditsError(Exception):
|
||||
"""
|
||||
Raised when a user attempts to create a job without sufficient credits.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
credits_available: Number of credits the user currently has
|
||||
credits_required: Number of credits required (always 1 for now)
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
credits_available: int = 0,
|
||||
credits_required: int = 1
|
||||
):
|
||||
self.message = message
|
||||
self.credits_available = credits_available
|
||||
self.credits_required = credits_required
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class InvalidStateTransitionError(Exception):
|
||||
"""
|
||||
Raised when an invalid job state transition is attempted.
|
||||
|
||||
This exception is raised by JobManager.transition_to_state() when the
|
||||
requested transition is not allowed by the state machine defined in
|
||||
STATE_TRANSITIONS.
|
||||
|
||||
Example: Trying to transition from PENDING to GENERATING_SCREENS is invalid
|
||||
because PENDING can only transition to DOWNLOADING, SEARCHING_AUDIO, FAILED,
|
||||
or CANCELLED.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
job_id: The job ID that failed to transition
|
||||
from_status: Current status of the job
|
||||
to_status: Attempted target status
|
||||
valid_transitions: List of valid transitions from current status
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
job_id: str = "",
|
||||
from_status: str = "",
|
||||
to_status: str = "",
|
||||
valid_transitions: list = None
|
||||
):
|
||||
self.message = message
|
||||
self.job_id = job_id
|
||||
self.from_status = from_status
|
||||
self.to_status = to_status
|
||||
self.valid_transitions = valid_transitions or []
|
||||
super().__init__(message)
|
||||
@@ -1,99 +0,0 @@
|
||||
"""
|
||||
Lightweight i18n utility for backend translations.
|
||||
|
||||
Usage:
|
||||
from backend.i18n import t, get_locale_from_request, SUPPORTED_LOCALES
|
||||
|
||||
# In a route handler:
|
||||
locale = get_locale_from_request(request)
|
||||
message = t(locale, "auth.userNotFound")
|
||||
message = t(locale, "jobs.notFound", job_id="abc123")
|
||||
"""
|
||||
|
||||
import json
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from starlette.requests import Request
|
||||
|
||||
SUPPORTED_LOCALES = ("en", "es", "de")
|
||||
DEFAULT_LOCALE = "en"
|
||||
TRANSLATIONS_DIR = Path(__file__).parent / "translations"
|
||||
|
||||
|
||||
@lru_cache(maxsize=len(SUPPORTED_LOCALES))
|
||||
def _load_translations(locale: str) -> dict[str, Any]:
|
||||
"""Load and cache translation file for a locale."""
|
||||
path = TRANSLATIONS_DIR / f"{locale}.json"
|
||||
if not path.exists():
|
||||
if locale != DEFAULT_LOCALE:
|
||||
return _load_translations(DEFAULT_LOCALE)
|
||||
return {}
|
||||
with open(path, encoding="utf-8") as f:
|
||||
result: dict[str, Any] = json.load(f)
|
||||
return result
|
||||
|
||||
|
||||
def t(locale: str, key: str, **kwargs: Any) -> str:
|
||||
"""Get a translated string by dotted key path.
|
||||
|
||||
Args:
|
||||
locale: Language code (en, es, de)
|
||||
key: Dotted path like "auth.userNotFound" or "emails.magicLink.subject"
|
||||
**kwargs: Interpolation variables (e.g., job_id="abc123")
|
||||
|
||||
Returns:
|
||||
Translated string with variables interpolated, or the key itself if not found.
|
||||
"""
|
||||
if locale not in SUPPORTED_LOCALES:
|
||||
locale = DEFAULT_LOCALE
|
||||
|
||||
translations = _load_translations(locale)
|
||||
|
||||
# Walk the dotted path
|
||||
value: Any = translations
|
||||
for part in key.split("."):
|
||||
if isinstance(value, dict) and part in value:
|
||||
value = value[part]
|
||||
else:
|
||||
# Key not found — fall back to English
|
||||
if locale != DEFAULT_LOCALE:
|
||||
return t(DEFAULT_LOCALE, key, **kwargs)
|
||||
return key # Return the key itself as last resort
|
||||
|
||||
if not isinstance(value, str):
|
||||
return key
|
||||
|
||||
result: str = value
|
||||
# Interpolate variables
|
||||
if kwargs:
|
||||
try:
|
||||
result = result.format(**kwargs)
|
||||
except KeyError:
|
||||
pass # Missing variable — return template as-is
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def get_locale_from_request(request: Request) -> str:
|
||||
"""Extract locale from Accept-Language header.
|
||||
|
||||
Returns the first supported locale found, or DEFAULT_LOCALE.
|
||||
"""
|
||||
accept_lang = request.headers.get("accept-language", "")
|
||||
|
||||
# Simple parsing — Accept-Language can be "de", "de-DE", "en-US,en;q=0.9,de;q=0.8"
|
||||
for part in accept_lang.split(","):
|
||||
lang = part.split(";")[0].strip().split("-")[0].lower()
|
||||
if lang in SUPPORTED_LOCALES:
|
||||
return lang
|
||||
|
||||
return DEFAULT_LOCALE
|
||||
|
||||
|
||||
def get_locale_prefix(locale: str) -> str:
|
||||
"""Get URL locale prefix for frontend URLs (e.g., '/en', '/es')."""
|
||||
if locale not in SUPPORTED_LOCALES:
|
||||
locale = DEFAULT_LOCALE
|
||||
return f"/{locale}"
|
||||
+6
-80
@@ -7,15 +7,10 @@ from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push, catalog, encoding_worker, client_errors
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
from backend.services.nltk_preloader import preload_all_nltk_resources
|
||||
from backend.services.langfuse_preloader import preload_langfuse_handler
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware
|
||||
from backend.workers.registry import worker_registry
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
@@ -72,28 +67,7 @@ async def lifespan(app: FastAPI):
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
# Preload NLP models and resources to avoid cold start delays
|
||||
# See docs/archive/2026-01-08-performance-investigation.md for background
|
||||
|
||||
# 1. SpaCy model (60+ second delay without preload)
|
||||
try:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
except Exception as e:
|
||||
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
||||
|
||||
# 2. NLTK cmudict (50-100+ second delay without preload)
|
||||
try:
|
||||
preload_all_nltk_resources()
|
||||
except Exception as e:
|
||||
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
||||
|
||||
# 3. Langfuse callback handler (200+ second delay without preload)
|
||||
try:
|
||||
preload_langfuse_handler()
|
||||
except Exception as e:
|
||||
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
||||
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
@@ -101,23 +75,8 @@ async def lifespan(app: FastAPI):
|
||||
logger.error(f"Credential validation failed: {e}")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown - wait for any active workers to complete before terminating
|
||||
# This prevents Cloud Run from killing workers mid-processing
|
||||
logger.info("Shutdown requested, checking for active workers...")
|
||||
if worker_registry.has_active_workers():
|
||||
active = worker_registry.get_active_workers()
|
||||
logger.info(f"Active workers found: {active}")
|
||||
logger.info("Waiting for workers to complete (timeout: 600s)...")
|
||||
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
|
||||
if not completed:
|
||||
logger.error(
|
||||
"Shutdown timeout - some workers may not have completed cleanly. "
|
||||
f"Remaining workers: {worker_registry.get_active_workers()}"
|
||||
)
|
||||
else:
|
||||
logger.info("No active workers, proceeding with shutdown")
|
||||
|
||||
|
||||
# Shutdown
|
||||
logger.info("Shutting down karaoke generation backend")
|
||||
|
||||
|
||||
@@ -136,8 +95,8 @@ if tracing_enabled:
|
||||
# Configure CORS
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"],
|
||||
allow_credentials=False,
|
||||
allow_origins=["*"], # Configure this properly for production
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
@@ -145,50 +104,17 @@ app.add_middleware(
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Add tenant detection middleware (extracts tenant from subdomain/headers)
|
||||
app.add_middleware(TenantMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
||||
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(encoding_worker.router, prefix="/api") # Encoding worker lifecycle
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
from backend.api.routes import referrals
|
||||
app.include_router(referrals.router, prefix="/api") # Referral links and payouts
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
||||
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
||||
app.include_router(catalog.router, prefix="/api") # Catalog proxy for song/artist autocomplete
|
||||
app.include_router(client_errors.router, prefix="/api") # Frontend crash reports
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
|
||||
|
||||
# Exception handlers
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from backend.exceptions import InsufficientCreditsError
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
|
||||
@app.exception_handler(InsufficientCreditsError)
|
||||
async def insufficient_credits_exception_handler(request: Request, exc: InsufficientCreditsError):
|
||||
"""Handle insufficient credits errors with 402 Payment Required status."""
|
||||
locale = get_locale_from_request(request)
|
||||
return JSONResponse(
|
||||
status_code=402,
|
||||
content={
|
||||
"detail": t(locale, "errors.insufficientCredits"),
|
||||
"credits_available": exc.credits_available,
|
||||
"credits_required": exc.credits_required,
|
||||
"buy_url": t(locale, "errors.insufficientCreditsBuyUrl"),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
|
||||
@@ -1,11 +1,5 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
|
||||
|
||||
__all__ = [
|
||||
"AuditLoggingMiddleware",
|
||||
"TenantMiddleware",
|
||||
"get_tenant_from_request",
|
||||
"get_tenant_config_from_request",
|
||||
]
|
||||
__all__ = ["AuditLoggingMiddleware"]
|
||||
@@ -1,192 +0,0 @@
|
||||
"""
|
||||
Tenant detection middleware for white-label B2B portals.
|
||||
|
||||
This middleware extracts the tenant ID from incoming requests and attaches
|
||||
the tenant configuration to the request state for use by downstream handlers.
|
||||
|
||||
Tenant detection priority:
|
||||
1. X-Tenant-ID header (explicitly set by frontend)
|
||||
2. `tenant` query parameter (for development/testing)
|
||||
3. Host header subdomain detection (production flow)
|
||||
|
||||
The middleware is non-blocking - if no tenant is detected, the request
|
||||
proceeds as a default Nomad Karaoke request (tenant_id = None).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.models.tenant import TenantConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Only allow query param tenant override in non-production environments
|
||||
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
|
||||
os.environ.get("ENVIRONMENT", "").lower() == "production"
|
||||
|
||||
|
||||
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
|
||||
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
|
||||
|
||||
|
||||
class TenantMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that detects tenant from request and attaches config to state.
|
||||
|
||||
After this middleware runs, routes can access:
|
||||
- request.state.tenant_id: str or None
|
||||
- request.state.tenant_config: TenantConfig or None
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = getattr(request.state, "tenant_id", None)
|
||||
tenant_config = getattr(request.state, "tenant_config", None)
|
||||
```
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Extract tenant ID from request
|
||||
tenant_id = self._extract_tenant_id(request)
|
||||
|
||||
# Load tenant config if tenant detected
|
||||
tenant_config: Optional[TenantConfig] = None
|
||||
if tenant_id:
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(tenant_id)
|
||||
|
||||
if tenant_config and not tenant_config.is_active:
|
||||
# Tenant exists but is inactive - treat as default
|
||||
logger.warning(f"Inactive tenant requested: {tenant_id}")
|
||||
tenant_id = None
|
||||
tenant_config = None
|
||||
|
||||
# Attach to request state
|
||||
request.state.tenant_id = tenant_id
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log tenant detection for debugging
|
||||
if tenant_id:
|
||||
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
|
||||
|
||||
# Process request
|
||||
response = await call_next(request)
|
||||
|
||||
# Add tenant ID to response headers for debugging
|
||||
if tenant_id:
|
||||
response.headers["X-Tenant-ID"] = tenant_id
|
||||
|
||||
return response
|
||||
|
||||
def _extract_tenant_id(self, request: Request) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from request using priority-based detection.
|
||||
|
||||
Returns:
|
||||
Tenant ID if detected, None for default Nomad Karaoke
|
||||
"""
|
||||
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
|
||||
header_tenant = request.headers.get("X-Tenant-ID")
|
||||
if header_tenant:
|
||||
return header_tenant.lower().strip()
|
||||
|
||||
# Priority 2: Query parameter (for development/testing only)
|
||||
# Disabled in production to prevent tenant spoofing
|
||||
if not IS_PRODUCTION:
|
||||
query_tenant = request.query_params.get("tenant")
|
||||
if query_tenant:
|
||||
return query_tenant.lower().strip()
|
||||
|
||||
# Priority 3: Host header subdomain detection
|
||||
host = request.headers.get("Host", "")
|
||||
return self._extract_tenant_from_host(host)
|
||||
|
||||
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from Host header subdomain.
|
||||
|
||||
Patterns handled:
|
||||
- vocalstar.nomadkaraoke.com -> vocalstar
|
||||
- vocalstar.gen.nomadkaraoke.com -> vocalstar
|
||||
- localhost:3000 -> None (local dev)
|
||||
- gen.nomadkaraoke.com -> None (main app)
|
||||
- api.nomadkaraoke.com -> None (API)
|
||||
|
||||
Returns:
|
||||
Tenant ID if subdomain matches tenant, None otherwise
|
||||
"""
|
||||
if not host:
|
||||
return None
|
||||
|
||||
# Normalize
|
||||
host_lower = host.lower()
|
||||
|
||||
# Remove port if present
|
||||
if ":" in host_lower:
|
||||
host_lower = host_lower.split(":")[0]
|
||||
|
||||
# Check if this is a nomadkaraoke.com domain
|
||||
if "nomadkaraoke.com" not in host_lower:
|
||||
return None
|
||||
|
||||
# Split into parts
|
||||
parts = host_lower.split(".")
|
||||
|
||||
# Need at least 3 parts: subdomain.nomadkaraoke.com
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
|
||||
# First part is the potential tenant ID
|
||||
potential_tenant = parts[0]
|
||||
|
||||
# Skip known non-tenant subdomains
|
||||
if potential_tenant in NON_TENANT_SUBDOMAINS:
|
||||
return None
|
||||
|
||||
# Verify tenant exists in GCS
|
||||
tenant_service = get_tenant_service()
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
return potential_tenant
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_tenant_from_request(request: Request) -> Optional[str]:
|
||||
"""
|
||||
Helper function to get tenant ID from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_id", None)
|
||||
|
||||
|
||||
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Helper function to get tenant config from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
config = get_tenant_config_from_request(request)
|
||||
if config and not config.features.audio_search:
|
||||
raise HTTPException(403, "Audio search not available")
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_config", None)
|
||||
@@ -1,110 +0,0 @@
|
||||
"""
|
||||
Audio edit session model for persisting input audio editing progress.
|
||||
|
||||
Stored as a Firestore subcollection: jobs/{job_id}/audio_edit_sessions/{session_id}
|
||||
Full edit data is stored in GCS to avoid Firestore's 1MB document limit.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Dict, Any, List
|
||||
import uuid
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioEditSessionSummary:
|
||||
"""Compact summary of operations in an audio edit session, for list views."""
|
||||
total_operations: int = 0
|
||||
operations_breakdown: Dict[str, int] = field(default_factory=dict)
|
||||
duration_change_seconds: float = 0.0
|
||||
net_duration_seconds: float = 0.0
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"total_operations": self.total_operations,
|
||||
"operations_breakdown": self.operations_breakdown,
|
||||
"duration_change_seconds": self.duration_change_seconds,
|
||||
"net_duration_seconds": self.net_duration_seconds,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSessionSummary":
|
||||
return cls(
|
||||
total_operations=data.get("total_operations", 0),
|
||||
operations_breakdown=data.get("operations_breakdown", {}),
|
||||
duration_change_seconds=data.get("duration_change_seconds", 0.0),
|
||||
net_duration_seconds=data.get("net_duration_seconds", 0.0),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioEditSession:
|
||||
"""
|
||||
A saved snapshot of an audio editing session.
|
||||
|
||||
Stored at: jobs/{job_id}/audio_edit_sessions/{session_id}
|
||||
Edit data stored at: jobs/{job_id}/audio_edit_sessions/{session_id}.json (GCS)
|
||||
"""
|
||||
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = ""
|
||||
user_email: str = ""
|
||||
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
edit_count: int = 0
|
||||
trigger: str = "auto" # "auto" | "manual" | "submit"
|
||||
audio_duration_seconds: Optional[float] = None
|
||||
original_duration_seconds: Optional[float] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
summary: AudioEditSessionSummary = field(default_factory=AudioEditSessionSummary)
|
||||
edit_data_gcs_path: str = ""
|
||||
|
||||
# Hash of the edit data JSON, used to skip duplicate saves
|
||||
data_hash: str = ""
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"session_id": self.session_id,
|
||||
"job_id": self.job_id,
|
||||
"user_email": self.user_email,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
"edit_count": self.edit_count,
|
||||
"trigger": self.trigger,
|
||||
"audio_duration_seconds": self.audio_duration_seconds,
|
||||
"original_duration_seconds": self.original_duration_seconds,
|
||||
"artist": self.artist,
|
||||
"title": self.title,
|
||||
"summary": self.summary.to_dict(),
|
||||
"edit_data_gcs_path": self.edit_data_gcs_path,
|
||||
"data_hash": self.data_hash,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "AudioEditSession":
|
||||
created_at = data.get("created_at")
|
||||
if isinstance(created_at, str):
|
||||
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
updated_at = data.get("updated_at")
|
||||
if isinstance(updated_at, str):
|
||||
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
summary_data = data.get("summary", {})
|
||||
summary = AudioEditSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else AudioEditSessionSummary()
|
||||
|
||||
return cls(
|
||||
session_id=data.get("session_id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
user_email=data.get("user_email", ""),
|
||||
created_at=created_at or datetime.now(timezone.utc),
|
||||
updated_at=updated_at or datetime.now(timezone.utc),
|
||||
edit_count=data.get("edit_count", 0),
|
||||
trigger=data.get("trigger", "auto"),
|
||||
audio_duration_seconds=data.get("audio_duration_seconds"),
|
||||
original_duration_seconds=data.get("original_duration_seconds"),
|
||||
artist=data.get("artist"),
|
||||
title=data.get("title"),
|
||||
summary=summary,
|
||||
edit_data_gcs_path=data.get("edit_data_gcs_path", ""),
|
||||
data_hash=data.get("data_hash", ""),
|
||||
)
|
||||
+56
-179
@@ -9,70 +9,58 @@ from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from karaoke_gen.utils import normalize_text
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
Job status enumeration - Complete state machine.
|
||||
|
||||
The workflow has 8 main stages with up to 3 human interaction points:
|
||||
|
||||
The workflow has 8 main stages with 3 human interaction points:
|
||||
1. Input & Setup (may include audio source selection)
|
||||
1b. Optional Audio Editing (BLOCKING) - trim/cut/mute/join input audio
|
||||
2. Parallel Processing (audio + lyrics)
|
||||
3. Title/End Screen Generation + Backing Vocals Analysis
|
||||
3. Title/End Screen Generation
|
||||
4. Countdown Padding Synchronization
|
||||
5. Combined Human Review (BLOCKING) - lyrics review + instrumental selection
|
||||
6. Video Finalization
|
||||
7. Distribution
|
||||
|
||||
Note: As of 2026-01, lyrics review and instrumental selection are combined
|
||||
into a single human review step. The user selects their instrumental
|
||||
during the lyrics review phase, before the video is rendered.
|
||||
5. Human Review (BLOCKING)
|
||||
6. Instrumental Selection (BLOCKING)
|
||||
7. Video Finalization
|
||||
8. Distribution
|
||||
"""
|
||||
# Initial states
|
||||
PENDING = "pending" # Job created, queued for processing
|
||||
|
||||
|
||||
# Audio search states (for artist+title search mode)
|
||||
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
|
||||
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
|
||||
|
||||
|
||||
DOWNLOADING = "downloading" # Downloading from URL or processing upload
|
||||
|
||||
# Optional: Audio editing (before processing begins)
|
||||
AWAITING_AUDIO_EDIT = "awaiting_audio_edit" # ⚠️ WAITING FOR USER - edit input audio
|
||||
IN_AUDIO_EDIT = "in_audio_edit" # User is actively editing audio
|
||||
AUDIO_EDIT_COMPLETE = "audio_edit_complete" # User submitted edited audio
|
||||
|
||||
|
||||
# Stage 2a: Audio separation (parallel track 1)
|
||||
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
|
||||
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
|
||||
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
|
||||
|
||||
|
||||
# Stage 2b: Lyrics processing (parallel track 2)
|
||||
TRANSCRIBING = "transcribing" # AudioShake API transcription
|
||||
CORRECTING = "correcting" # Automatic lyrics correction
|
||||
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
|
||||
|
||||
# Stage 3: Title/End screens + Backing Vocals Analysis
|
||||
|
||||
# Stage 3: Title/End screens
|
||||
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
|
||||
|
||||
|
||||
# Stage 4: Countdown padding (automatic)
|
||||
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
|
||||
|
||||
# Stage 5: Combined Human Review (BLOCKING) - lyrics + instrumental selection
|
||||
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - combined review needed
|
||||
IN_REVIEW = "in_review" # User is actively reviewing
|
||||
REVIEW_COMPLETE = "review_complete" # User submitted lyrics + instrumental selection
|
||||
|
||||
|
||||
# Stage 5: Human review (BLOCKING)
|
||||
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - lyrics review needed
|
||||
IN_REVIEW = "in_review" # User is actively reviewing lyrics
|
||||
REVIEW_COMPLETE = "review_complete" # User submitted corrected lyrics
|
||||
|
||||
# Stage 5.5: Render video with corrected lyrics (post-review)
|
||||
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
|
||||
|
||||
# Stage 6: Instrumental already selected (during combined review)
|
||||
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # LEGACY - no longer used
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected" # Instrumental was selected during combined review
|
||||
|
||||
# Stage 6: Instrumental selection (BLOCKING)
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # ⚠️ WAITING FOR USER
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected" # User made selection
|
||||
|
||||
# Stage 7: Video generation and finalization
|
||||
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
|
||||
@@ -99,83 +87,71 @@ class JobStatus(str, Enum):
|
||||
|
||||
# Valid state transitions
|
||||
STATE_TRANSITIONS = {
|
||||
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search),
|
||||
# or DOWNLOADING_AUDIO (create-from-search where audio is pre-selected)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# PENDING can go to DOWNLOADING (file upload), SEARCHING_AUDIO (artist+title search), or AWAITING_INSTRUMENTAL_SELECTION (finalise-only)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# Audio search flow (for artist+title search mode)
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
|
||||
|
||||
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
|
||||
# Can also enter optional audio edit phase if requires_audio_edit flag is set
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.AWAITING_AUDIO_EDIT, JobStatus.FAILED],
|
||||
|
||||
# Optional audio edit phase (trim, cut, mute, join input audio before processing)
|
||||
JobStatus.AWAITING_AUDIO_EDIT: [JobStatus.IN_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_AUDIO_EDIT: [JobStatus.AWAITING_AUDIO_EDIT, JobStatus.AUDIO_EDIT_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.AUDIO_EDIT_COMPLETE: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Audio separation flow
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
|
||||
# Lyrics flow
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Post-parallel processing (screens + backing vocals analysis)
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
|
||||
# Post-parallel processing
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
|
||||
# Combined human review flow (lyrics + instrumental selection)
|
||||
|
||||
# Human review flow
|
||||
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
|
||||
|
||||
# Video rendering (post-review) - instrumental was already selected during combined review
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
|
||||
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
|
||||
|
||||
# Video rendering (post-review)
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Instrumental selection flow
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
|
||||
|
||||
# Video generation flow
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
|
||||
# Distribution flow
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Terminal states - COMPLETE allows edit (reopen review) and visibility change
|
||||
|
||||
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
|
||||
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
|
||||
# PREP_COMPLETE allows continuation from combined review
|
||||
JobStatus.COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.LYRICS_COMPLETE], # Edit track or visibility change
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
|
||||
# PREP_COMPLETE allows finalise-only continuation
|
||||
JobStatus.COMPLETE: [],
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_INSTRUMENTAL_SELECTION, JobStatus.FAILED], # Finalise-only continues from here
|
||||
JobStatus.FAILED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
JobStatus.AWAITING_REVIEW, # Retry from combined review
|
||||
],
|
||||
JobStatus.CANCELLED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.DOWNLOADING_AUDIO, # Retry audio download (audio search flow)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
JobStatus.AWAITING_REVIEW, # Retry from combined review
|
||||
],
|
||||
|
||||
|
||||
# Legacy states (for backward compatibility)
|
||||
JobStatus.QUEUED: [JobStatus.PENDING],
|
||||
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
|
||||
@@ -191,7 +167,6 @@ class TimelineEvent(BaseModel):
|
||||
timestamp: str
|
||||
progress: Optional[int] = None
|
||||
message: Optional[str] = None
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class LogEntry(BaseModel):
|
||||
@@ -221,14 +196,7 @@ class Job(BaseModel):
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
|
||||
|
||||
# Audio source tracking (for retry capability)
|
||||
audio_source_type: Optional[str] = None # Source type: 'audio_search', 'youtube_url', 'file_upload'
|
||||
source_name: Optional[str] = None # Provider name (RED, OPS, YouTube, Spotify) - for audio_search
|
||||
source_id: Optional[str] = None # Source-specific ID (torrent ID, video ID, track ID) - for audio_search
|
||||
target_file: Optional[str] = None # For torrents, specific file to extract - for audio_search
|
||||
download_url: Optional[str] = None # Direct download URL if available - for audio_search
|
||||
|
||||
|
||||
# User preferences
|
||||
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
|
||||
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
|
||||
@@ -237,9 +205,6 @@ class Job(BaseModel):
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Multi-tenant support ("" = consumer portal, "vocalstar"/etc = tenant portal)
|
||||
tenant_id: str = "" # Tenant ID for white-label portal scoping
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
@@ -309,31 +274,7 @@ class Job(BaseModel):
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Private (non-published) track mode
|
||||
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
|
||||
|
||||
# Creation-time decision params (for observability — what did the user actually choose?)
|
||||
creation_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
# Anti-abuse: IP address at job creation time (for cross-account correlation)
|
||||
creation_ip: Optional[str] = None
|
||||
|
||||
# Output deletion tracking (for admin cleanup without deleting job)
|
||||
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
|
||||
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
|
||||
|
||||
# Credit refund tracking (prevents double-refund on cancel→delete or fail→delete)
|
||||
credit_refunded: bool = False
|
||||
|
||||
# Edit tracking (how many times a completed track has been re-edited)
|
||||
edit_count: int = 0
|
||||
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
@@ -424,43 +365,6 @@ class Job(BaseModel):
|
||||
}
|
||||
"""
|
||||
|
||||
# Processing metadata (immutable provenance data written by workers during processing)
|
||||
processing_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Immutable processing provenance data. Written once by each worker during processing.
|
||||
Never modified after initial write. Used for analytics, debugging, and data export.
|
||||
|
||||
Structure:
|
||||
{
|
||||
"transcription": {
|
||||
"provider": "audioshake",
|
||||
"audioshake_task_id": "...",
|
||||
"audioshake_asset_id": "...",
|
||||
"language_detected": "en",
|
||||
"word_count": 358,
|
||||
"segment_count": 42,
|
||||
},
|
||||
"correction": {
|
||||
"handlers_applied": [...],
|
||||
"corrections_made": 12,
|
||||
"correction_ratio": 0.034,
|
||||
"reference_sources_found": ["genius", "spotify"],
|
||||
"agentic_routing": "rule-based",
|
||||
"duration_seconds": 33.2,
|
||||
},
|
||||
"separation": {
|
||||
"provider": "modal",
|
||||
"clean_model": "...",
|
||||
"duration_seconds": 420.5,
|
||||
},
|
||||
"timing": {
|
||||
"lyrics_worker_seconds": 125.3,
|
||||
"audio_worker_seconds": 420.5,
|
||||
...
|
||||
},
|
||||
}
|
||||
"""
|
||||
|
||||
# Request metadata (captured at job creation for tracking and filtering)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -568,15 +472,7 @@ class JobCreate(BaseModel):
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Private (non-published) track mode
|
||||
is_private: bool = False # Private tracks: Dropbox only (Tracks-NonPublished/NOMADNP), no YouTube/GDrive
|
||||
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
@@ -588,33 +484,14 @@ class JobCreate(BaseModel):
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
# Tenant scoping for white-label portals ("" = consumer, "vocalstar"/etc = tenant)
|
||||
tenant_id: str = "" # Tenant ID for job scoping
|
||||
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
"""Validate URL is not empty."""
|
||||
|
||||
@validator('url', 'artist', 'title')
|
||||
def validate_inputs(cls, v):
|
||||
"""Validate string inputs are not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_artist_title(cls, v):
|
||||
"""Normalize artist/title text to standardize Unicode characters.
|
||||
|
||||
This ensures consistent data storage by converting:
|
||||
- Curly quotes -> straight quotes
|
||||
- Various dashes -> hyphen
|
||||
- Unusual whitespace -> regular space
|
||||
"""
|
||||
if v is not None and isinstance(v, str):
|
||||
if not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
# normalize_text handles stripping and Unicode normalization
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
|
||||
@@ -1,178 +0,0 @@
|
||||
"""
|
||||
Referral system models.
|
||||
|
||||
Supports:
|
||||
- Referral link creation and management
|
||||
- Earnings tracking from referred purchases
|
||||
- Payout processing via Stripe Connect
|
||||
- API request/response shapes for referral endpoints
|
||||
"""
|
||||
from datetime import datetime
|
||||
from typing import Optional, List
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Firestore Document Models
|
||||
# ============================================================================
|
||||
|
||||
class ReferralLinkStats(BaseModel):
|
||||
"""Aggregated statistics for a referral link."""
|
||||
clicks: int = 0
|
||||
signups: int = 0
|
||||
purchases: int = 0
|
||||
total_earned_cents: int = 0
|
||||
|
||||
|
||||
class ReferralLink(BaseModel):
|
||||
"""
|
||||
Referral link stored in Firestore.
|
||||
|
||||
Each user can have one referral link. Vanity links use a custom code
|
||||
(e.g., "karaoke-king") while standard links use a generated code.
|
||||
"""
|
||||
code: str # Unique referral code (URL slug)
|
||||
owner_email: str # Email of the referrer
|
||||
display_name: Optional[str] = None # Public name shown on interstitial
|
||||
custom_message: Optional[str] = None # Optional message on interstitial
|
||||
|
||||
# Reward configuration
|
||||
discount_percent: int = 10 # Discount for referred user
|
||||
kickback_percent: int = 20 # Earnings for referrer
|
||||
discount_duration_days: int = 30 # How long the discount lasts
|
||||
earning_duration_days: int = 365 # How long referrer earns from this link
|
||||
|
||||
# Stripe
|
||||
stripe_coupon_id: Optional[str] = None # Stripe coupon for this link
|
||||
|
||||
# Flags
|
||||
is_vanity: bool = False # Custom vanity code vs generated
|
||||
enabled: bool = True # Can be disabled without deleting
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
# Aggregated stats
|
||||
stats: ReferralLinkStats = Field(default_factory=ReferralLinkStats)
|
||||
|
||||
|
||||
class ReferralEarning(BaseModel):
|
||||
"""
|
||||
Record of a single earning from a referred purchase.
|
||||
|
||||
Created when a referred user completes a Stripe checkout.
|
||||
"""
|
||||
id: str
|
||||
referrer_email: str
|
||||
referred_email: str
|
||||
referral_code: str
|
||||
stripe_session_id: str
|
||||
purchase_amount_cents: int
|
||||
earning_amount_cents: int
|
||||
status: str = "pending" # pending, paid, cancelled
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
paid_at: Optional[datetime] = None
|
||||
payout_id: Optional[str] = None
|
||||
|
||||
|
||||
class ReferralPayout(BaseModel):
|
||||
"""
|
||||
Record of a payout to a referrer via Stripe Connect.
|
||||
|
||||
Groups multiple earnings into a single transfer.
|
||||
"""
|
||||
id: str
|
||||
referrer_email: str
|
||||
stripe_transfer_id: str
|
||||
amount_cents: int
|
||||
earnings_included: List[str] # List of ReferralEarning IDs
|
||||
status: str = "processing" # processing, completed, failed
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# API Request Models
|
||||
# ============================================================================
|
||||
|
||||
class CreateReferralLinkRequest(BaseModel):
|
||||
"""Admin request to create a vanity referral link."""
|
||||
vanity_code: Optional[str] = None
|
||||
owner_email: str
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: int = 10
|
||||
kickback_percent: int = 20
|
||||
discount_duration_days: int = 30
|
||||
earning_duration_days: int = 365
|
||||
|
||||
|
||||
class UpdateReferralLinkRequest(BaseModel):
|
||||
"""Request to update a referral link (user: name/message only, admin: all fields)."""
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: Optional[int] = None
|
||||
kickback_percent: Optional[int] = None
|
||||
discount_duration_days: Optional[int] = None
|
||||
earning_duration_days: Optional[int] = None
|
||||
enabled: Optional[bool] = None
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# API Response Models
|
||||
# ============================================================================
|
||||
|
||||
class ReferralLinkResponse(BaseModel):
|
||||
"""Public referral link information."""
|
||||
code: str
|
||||
owner_email: str
|
||||
display_name: Optional[str] = None
|
||||
custom_message: Optional[str] = None
|
||||
discount_percent: int
|
||||
kickback_percent: int
|
||||
is_vanity: bool
|
||||
enabled: bool
|
||||
stats: ReferralLinkStats
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ReferralEarningResponse(BaseModel):
|
||||
"""Public earning information (for referrer dashboard)."""
|
||||
id: str
|
||||
referred_email: str
|
||||
referral_code: str
|
||||
purchase_amount_cents: int
|
||||
earning_amount_cents: int
|
||||
status: str
|
||||
created_at: datetime
|
||||
paid_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ReferralPayoutResponse(BaseModel):
|
||||
"""Public payout information (for referrer dashboard)."""
|
||||
id: str
|
||||
amount_cents: int
|
||||
earnings_count: int
|
||||
status: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ReferralDashboardResponse(BaseModel):
|
||||
"""Full referral dashboard data for the referrer."""
|
||||
referral_link: ReferralLinkResponse
|
||||
recent_earnings: List[ReferralEarningResponse]
|
||||
recent_payouts: List[ReferralPayoutResponse]
|
||||
total_earned_cents: int
|
||||
total_paid_cents: int
|
||||
pending_balance_cents: int
|
||||
stripe_connect_onboarded: bool
|
||||
|
||||
|
||||
class ReferralInterstitialResponse(BaseModel):
|
||||
"""Data for the referral landing/interstitial page."""
|
||||
valid: bool = True
|
||||
referral_code: str = ""
|
||||
referrer_display_name: Optional[str] = None
|
||||
discount_percent: int = 0
|
||||
discount_duration_days: int = 0
|
||||
custom_message: Optional[str] = None
|
||||
@@ -10,15 +10,14 @@ class URLSubmissionRequest(BaseModel):
|
||||
url: HttpUrl
|
||||
artist: Optional[str] = None # Auto-detected if not provided
|
||||
title: Optional[str] = None # Auto-detected if not provided
|
||||
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
is_private: Optional[bool] = None # Private (non-published) track mode
|
||||
|
||||
|
||||
class UploadSubmissionRequest(BaseModel):
|
||||
@@ -29,7 +28,7 @@ class UploadSubmissionRequest(BaseModel):
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
@@ -38,13 +37,12 @@ class UploadSubmissionRequest(BaseModel):
|
||||
class CorrectionsSubmission(BaseModel):
|
||||
"""
|
||||
Request to submit corrected lyrics after human review.
|
||||
|
||||
|
||||
This is the critical human-in-the-loop interaction point.
|
||||
The corrections data comes from the lyrics-transcriber review interface.
|
||||
"""
|
||||
corrections: Dict[str, Any] # Full corrections JSON from frontend
|
||||
user_notes: Optional[str] = None # Optional notes from reviewer
|
||||
is_duet: Optional[bool] = None # When present, persists duet flag to state_data
|
||||
|
||||
@validator('corrections')
|
||||
def validate_corrections_format(cls, v):
|
||||
@@ -59,13 +57,13 @@ class CorrectionsSubmission(BaseModel):
|
||||
class InstrumentalSelection(BaseModel):
|
||||
"""
|
||||
Request to select instrumental audio option.
|
||||
|
||||
|
||||
This is the second critical human-in-the-loop interaction point.
|
||||
User chooses between clean instrumental, instrumental with backing vocals,
|
||||
or a custom instrumental (created via create-custom-instrumental endpoint).
|
||||
"""
|
||||
selection: str # "clean", "with_backing", or "custom"
|
||||
|
||||
|
||||
@validator('selection')
|
||||
def validate_selection(cls, v):
|
||||
"""Validate selection is a valid option."""
|
||||
@@ -75,26 +73,6 @@ class InstrumentalSelection(BaseModel):
|
||||
return v
|
||||
|
||||
|
||||
class CompleteReviewRequest(BaseModel):
|
||||
"""
|
||||
Request to complete the review with optional instrumental selection.
|
||||
|
||||
For the combined review flow, this includes the instrumental selection.
|
||||
For legacy flows, the instrumental_selection can be omitted.
|
||||
"""
|
||||
instrumental_selection: Optional[str] = None # "clean", "with_backing", or "custom"
|
||||
|
||||
@validator('instrumental_selection')
|
||||
def validate_instrumental_selection(cls, v):
|
||||
"""Validate instrumental selection if provided."""
|
||||
if v is None:
|
||||
return v
|
||||
valid_options = ['clean', 'with_backing', 'custom']
|
||||
if v not in valid_options:
|
||||
raise ValueError(f"instrumental_selection must be one of: {valid_options}")
|
||||
return v
|
||||
|
||||
|
||||
class MuteRegionRequest(BaseModel):
|
||||
"""A region to mute in the backing vocals."""
|
||||
start_seconds: float
|
||||
@@ -143,24 +121,3 @@ class RetryJobRequest(BaseModel):
|
||||
"""Request to retry a failed job."""
|
||||
from_stage: Optional[str] = None # Optional: restart from specific stage
|
||||
|
||||
|
||||
class ChangeVisibilityRequest(BaseModel):
|
||||
"""Request to change job visibility (public/private)."""
|
||||
target_visibility: str # "public" or "private"
|
||||
|
||||
@validator('target_visibility')
|
||||
def validate_target(cls, v):
|
||||
if v not in ('public', 'private'):
|
||||
raise ValueError("target_visibility must be 'public' or 'private'")
|
||||
return v
|
||||
|
||||
|
||||
class ChangeVisibilityResponse(BaseModel):
|
||||
"""Response from change visibility endpoint."""
|
||||
status: str # "success", "processing", "error"
|
||||
job_id: str
|
||||
message: str
|
||||
previous_visibility: str # "public" or "private"
|
||||
new_visibility: str # "public" or "private"
|
||||
reprocessing_required: bool
|
||||
|
||||
@@ -1,107 +0,0 @@
|
||||
"""
|
||||
Review session model for persisting lyrics review progress.
|
||||
|
||||
Stored as a Firestore subcollection: jobs/{job_id}/review_sessions/{session_id}
|
||||
Full correction data is stored in GCS to avoid Firestore's 1MB document limit.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Dict, Any, List
|
||||
import uuid
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewSessionSummary:
|
||||
"""Compact summary of edits in a review session, for list views."""
|
||||
total_segments: int = 0
|
||||
total_words: int = 0
|
||||
corrections_made: int = 0
|
||||
changed_words: List[Dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"total_segments": self.total_segments,
|
||||
"total_words": self.total_words,
|
||||
"corrections_made": self.corrections_made,
|
||||
"changed_words": self.changed_words,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSessionSummary":
|
||||
return cls(
|
||||
total_segments=data.get("total_segments", 0),
|
||||
total_words=data.get("total_words", 0),
|
||||
corrections_made=data.get("corrections_made", 0),
|
||||
changed_words=data.get("changed_words", []),
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ReviewSession:
|
||||
"""
|
||||
A saved snapshot of a lyrics review editing session.
|
||||
|
||||
Stored at: jobs/{job_id}/review_sessions/{session_id}
|
||||
Correction data stored at: jobs/{job_id}/review_sessions/{session_id}.json (GCS)
|
||||
"""
|
||||
session_id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = ""
|
||||
user_email: str = ""
|
||||
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
edit_count: int = 0
|
||||
trigger: str = "auto" # "auto" | "preview" | "manual"
|
||||
audio_duration_seconds: Optional[float] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
summary: ReviewSessionSummary = field(default_factory=ReviewSessionSummary)
|
||||
correction_data_gcs_path: str = ""
|
||||
|
||||
# Hash of the correction data JSON, used to skip duplicate saves
|
||||
data_hash: str = ""
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"session_id": self.session_id,
|
||||
"job_id": self.job_id,
|
||||
"user_email": self.user_email,
|
||||
"created_at": self.created_at,
|
||||
"updated_at": self.updated_at,
|
||||
"edit_count": self.edit_count,
|
||||
"trigger": self.trigger,
|
||||
"audio_duration_seconds": self.audio_duration_seconds,
|
||||
"artist": self.artist,
|
||||
"title": self.title,
|
||||
"summary": self.summary.to_dict(),
|
||||
"correction_data_gcs_path": self.correction_data_gcs_path,
|
||||
"data_hash": self.data_hash,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "ReviewSession":
|
||||
created_at = data.get("created_at")
|
||||
if isinstance(created_at, str):
|
||||
created_at = datetime.fromisoformat(created_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
updated_at = data.get("updated_at")
|
||||
if isinstance(updated_at, str):
|
||||
updated_at = datetime.fromisoformat(updated_at.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
summary_data = data.get("summary", {})
|
||||
summary = ReviewSessionSummary.from_dict(summary_data) if isinstance(summary_data, dict) else ReviewSessionSummary()
|
||||
|
||||
return cls(
|
||||
session_id=data.get("session_id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
user_email=data.get("user_email", ""),
|
||||
created_at=created_at or datetime.now(timezone.utc),
|
||||
updated_at=updated_at or datetime.now(timezone.utc),
|
||||
edit_count=data.get("edit_count", 0),
|
||||
trigger=data.get("trigger", "auto"),
|
||||
audio_duration_seconds=data.get("audio_duration_seconds"),
|
||||
artist=data.get("artist"),
|
||||
title=data.get("title"),
|
||||
summary=summary,
|
||||
correction_data_gcs_path=data.get("correction_data_gcs_path", ""),
|
||||
data_hash=data.get("data_hash", ""),
|
||||
)
|
||||
@@ -1,218 +0,0 @@
|
||||
"""
|
||||
Tenant data models for white-label B2B portals.
|
||||
|
||||
Tenants are B2B customers who get their own branded portal with custom
|
||||
configuration. Each tenant has:
|
||||
- Custom branding (logo, colors, site title)
|
||||
- Feature flags (enable/disable audio search, distribution, etc.)
|
||||
- Default settings (theme, distribution mode)
|
||||
- Auth configuration (allowed email domains, fixed tokens)
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class TenantBranding(BaseModel):
|
||||
"""Branding configuration for a tenant's portal."""
|
||||
|
||||
logo_url: Optional[str] = Field(
|
||||
None, description="URL to tenant's logo image (PNG with transparency preferred)"
|
||||
)
|
||||
logo_height: int = Field(40, description="Logo height in pixels for header display")
|
||||
primary_color: str = Field(
|
||||
"#ff5bb8", description="Primary brand color (hex format)"
|
||||
)
|
||||
secondary_color: str = Field(
|
||||
"#8b5cf6", description="Secondary brand color (hex format)"
|
||||
)
|
||||
accent_color: Optional[str] = Field(
|
||||
None, description="Accent color for highlights (hex format)"
|
||||
)
|
||||
background_color: Optional[str] = Field(
|
||||
None, description="Custom background color (defaults to dark theme)"
|
||||
)
|
||||
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
|
||||
site_title: str = Field(
|
||||
"Karaoke Generator", description="Browser tab title"
|
||||
)
|
||||
tagline: Optional[str] = Field(
|
||||
None, description="Optional tagline shown below logo"
|
||||
)
|
||||
|
||||
|
||||
class TenantFeatures(BaseModel):
|
||||
"""Feature flags controlling what's available in the tenant's portal."""
|
||||
|
||||
# Audio input methods
|
||||
audio_search: bool = Field(
|
||||
True, description="Enable audio search (flacfetch integration)"
|
||||
)
|
||||
file_upload: bool = Field(True, description="Enable direct file upload")
|
||||
youtube_url: bool = Field(True, description="Enable YouTube URL input")
|
||||
|
||||
# Distribution options
|
||||
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
|
||||
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
|
||||
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
|
||||
|
||||
# Customization options
|
||||
theme_selection: bool = Field(
|
||||
True, description="Allow user to select theme (false = use default)"
|
||||
)
|
||||
color_overrides: bool = Field(
|
||||
True, description="Allow user to customize colors"
|
||||
)
|
||||
|
||||
# Output formats
|
||||
enable_cdg: bool = Field(True, description="Generate CDG format output")
|
||||
enable_4k: bool = Field(True, description="Generate 4K video output")
|
||||
|
||||
# Advanced features
|
||||
admin_access: bool = Field(
|
||||
False, description="Allow access to admin dashboard"
|
||||
)
|
||||
|
||||
|
||||
class TenantDefaults(BaseModel):
|
||||
"""Default settings applied to all jobs for this tenant."""
|
||||
|
||||
theme_id: Optional[str] = Field(
|
||||
None, description="Default theme ID (required if theme_selection=false)"
|
||||
)
|
||||
locked_theme: Optional[str] = Field(
|
||||
None,
|
||||
description="If set, users cannot change theme - always uses this theme ID"
|
||||
)
|
||||
distribution_mode: str = Field(
|
||||
"all",
|
||||
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
|
||||
)
|
||||
brand_prefix: Optional[str] = Field(
|
||||
None, description="Prefix for output filenames (e.g., 'VSTAR')"
|
||||
)
|
||||
dropbox_path: Optional[str] = Field(
|
||||
None, description="Tenant-specific Dropbox folder path (e.g., '/Karaoke/Vocal-Star')"
|
||||
)
|
||||
gdrive_folder_id: Optional[str] = Field(
|
||||
None, description="Tenant-specific Google Drive folder ID for output uploads"
|
||||
)
|
||||
youtube_description_template: Optional[str] = Field(
|
||||
None, description="Default YouTube description template"
|
||||
)
|
||||
|
||||
|
||||
class TenantAuth(BaseModel):
|
||||
"""Authentication configuration for a tenant."""
|
||||
|
||||
allowed_email_domains: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
|
||||
)
|
||||
require_email_domain: bool = Field(
|
||||
True,
|
||||
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
|
||||
)
|
||||
fixed_token_ids: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
|
||||
)
|
||||
sender_email: Optional[str] = Field(
|
||||
None,
|
||||
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
|
||||
)
|
||||
|
||||
|
||||
class TenantConfig(BaseModel):
|
||||
"""Complete configuration for a white-label tenant."""
|
||||
|
||||
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
|
||||
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
|
||||
subdomain: str = Field(
|
||||
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
|
||||
)
|
||||
is_active: bool = Field(True, description="Whether tenant portal is active")
|
||||
|
||||
branding: TenantBranding = Field(default_factory=TenantBranding)
|
||||
features: TenantFeatures = Field(default_factory=TenantFeatures)
|
||||
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
|
||||
auth: TenantAuth = Field(default_factory=TenantAuth)
|
||||
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
def get_frontend_url(self) -> str:
|
||||
"""Get the frontend URL for this tenant's portal."""
|
||||
return f"https://{self.subdomain}"
|
||||
|
||||
def get_sender_email(self) -> str:
|
||||
"""Get the email sender address for this tenant."""
|
||||
if self.auth.sender_email:
|
||||
return self.auth.sender_email
|
||||
# Default pattern: {tenant_id}@nomadkaraoke.com
|
||||
return f"{self.id}@nomadkaraoke.com"
|
||||
|
||||
def is_email_allowed(self, email: str) -> bool:
|
||||
"""Check if an email address is allowed for this tenant."""
|
||||
if not self.auth.allowed_email_domains:
|
||||
# No domain restrictions
|
||||
return True
|
||||
|
||||
email_lower = email.lower()
|
||||
for domain in self.auth.allowed_email_domains:
|
||||
if email_lower.endswith(f"@{domain.lower()}"):
|
||||
return True
|
||||
|
||||
return not self.auth.require_email_domain
|
||||
|
||||
|
||||
class TenantPublicConfig(BaseModel):
|
||||
"""
|
||||
Public tenant configuration returned to frontend.
|
||||
|
||||
This excludes sensitive auth details like token IDs.
|
||||
"""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
subdomain: str
|
||||
is_active: bool
|
||||
branding: TenantBranding
|
||||
features: TenantFeatures
|
||||
defaults: TenantDefaults
|
||||
# Auth info limited to what frontend needs
|
||||
allowed_email_domains: List[str] = Field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
|
||||
"""Create public config from full config."""
|
||||
return cls(
|
||||
id=config.id,
|
||||
name=config.name,
|
||||
subdomain=config.subdomain,
|
||||
is_active=config.is_active,
|
||||
branding=config.branding,
|
||||
features=config.features,
|
||||
defaults=TenantDefaults(
|
||||
theme_id=config.defaults.theme_id,
|
||||
locked_theme=config.defaults.locked_theme,
|
||||
distribution_mode=config.defaults.distribution_mode,
|
||||
# Don't expose brand_prefix or youtube_description_template
|
||||
),
|
||||
allowed_email_domains=config.auth.allowed_email_domains,
|
||||
)
|
||||
|
||||
|
||||
class TenantConfigResponse(BaseModel):
|
||||
"""Response from GET /api/tenant/config endpoint."""
|
||||
|
||||
tenant: Optional[TenantPublicConfig] = Field(
|
||||
None, description="Tenant config if found, null for default Nomad Karaoke"
|
||||
)
|
||||
is_default: bool = Field(
|
||||
True, description="True if using default Nomad Karaoke config"
|
||||
)
|
||||
+52
-109
@@ -6,10 +6,11 @@ Supports:
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from typing import Optional, List, Dict
|
||||
from typing import Optional, List
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
@@ -19,6 +20,14 @@ class UserRole(str, Enum):
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
@@ -30,42 +39,17 @@ class CreditTransaction(BaseModel):
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class PushSubscription(BaseModel):
|
||||
"""
|
||||
Web Push subscription for a user's device.
|
||||
|
||||
Stores the push subscription endpoint and encryption keys needed
|
||||
to send push notifications to the user's browser/device.
|
||||
|
||||
Subscriptions are scoped by tenant_id to prevent cross-tenant
|
||||
notification leakage (e.g., a consumer job notification appearing
|
||||
on a tenant portal's service worker).
|
||||
"""
|
||||
endpoint: str # Push service endpoint URL
|
||||
keys: Dict[str, str] # p256dh and auth keys for encryption
|
||||
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
|
||||
tenant_id: Optional[str] = None # Tenant scope (None = consumer/default portal)
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
last_used_at: Optional[datetime] = None # Last time a notification was sent
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
|
||||
For multi-tenant white-label portals, users are scoped to a tenant.
|
||||
Users with tenant_id=None are default Nomad Karaoke users.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
@@ -85,35 +69,16 @@ class User(BaseModel):
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Payment tracking (cached from stripe_payments, in cents)
|
||||
total_spent: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# i18n locale preference (set from Accept-Language on login, used for emails/push)
|
||||
locale: Optional[str] = None
|
||||
|
||||
# Push notification subscriptions (Web Push API)
|
||||
# Users can subscribe from multiple devices/browsers
|
||||
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
|
||||
|
||||
# Feedback-for-credits program
|
||||
has_submitted_feedback: bool = False
|
||||
|
||||
# Anti-abuse: IP address and device fingerprint used during signup
|
||||
signup_ip: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
|
||||
# Flag to prevent duplicate welcome credit grants (idempotency)
|
||||
welcome_credits_granted: bool = False
|
||||
|
||||
# Referral system
|
||||
referral_code: Optional[str] = None # User's own referral code
|
||||
referred_by_code: Optional[str] = None # Code used when this user signed up
|
||||
referred_at: Optional[datetime] = None # When the referral was applied
|
||||
referral_discount_expires_at: Optional[datetime] = None # When referral discount ends
|
||||
stripe_connect_account_id: Optional[str] = None # For receiving referral payouts
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
@@ -122,9 +87,6 @@ class MagicLinkToken(BaseModel):
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
|
||||
For multi-tenant portals, the tenant_id determines which portal
|
||||
the user will be redirected to after verification.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
@@ -134,18 +96,6 @@ class MagicLinkToken(BaseModel):
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Referral attribution (travels with magic link for cross-device support)
|
||||
referral_code: Optional[str] = None
|
||||
|
||||
# Pre-computed credit evaluation (set asynchronously after magic link is sent)
|
||||
credit_eval_decision: Optional[str] = None # "grant", "deny", "pending_review"
|
||||
credit_eval_reasoning: Optional[str] = None
|
||||
credit_eval_error: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
@@ -154,8 +104,6 @@ class Session(BaseModel):
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
|
||||
For multi-tenant portals, sessions are scoped to a tenant.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
@@ -164,20 +112,14 @@ class Session(BaseModel):
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
device_fingerprint: Optional[str] = None
|
||||
referral_code: Optional[str] = None # From referral interstitial
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
@@ -197,9 +139,6 @@ class VerifyMagicLinkResponse(BaseModel):
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
tenant_subdomain: Optional[str] = None
|
||||
credits_granted: int = 0
|
||||
credit_status: str = "not_applicable" # "granted", "denied", "pending_review", "already_granted", "not_applicable"
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
@@ -210,16 +149,6 @@ class UserPublic(BaseModel):
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
tenant_id: Optional[str] = None
|
||||
feedback_eligible: bool = False
|
||||
total_spent: int = 0
|
||||
referral_code: Optional[str] = None
|
||||
has_active_referral_discount: bool = False
|
||||
referral_discount_percent: Optional[int] = None
|
||||
referral_discount_expires_at: Optional[str] = None # ISO string
|
||||
referred_by_code: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_login_at: Optional[str] = None
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
@@ -245,18 +174,18 @@ class UserListResponse(BaseModel):
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# User Feedback-for-Credits Models
|
||||
# Beta Tester Program Models
|
||||
# ============================================================================
|
||||
|
||||
class UserFeedback(BaseModel):
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from any user (not just beta testers).
|
||||
Feedback from a beta tester after using the service.
|
||||
|
||||
Stored in Firestore 'user_feedback' collection.
|
||||
Users earn 1 free credit for submitting detailed feedback.
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
"""
|
||||
id: str
|
||||
id: str # Unique feedback ID
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
@@ -275,11 +204,33 @@ class UserFeedback(BaseModel):
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web"
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
|
||||
|
||||
class UserFeedbackRequest(BaseModel):
|
||||
"""Request to submit user feedback for credits."""
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
@@ -296,16 +247,8 @@ class UserFeedbackRequest(BaseModel):
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class UserFeedbackResponse(BaseModel):
|
||||
"""Response after submitting user feedback."""
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int = 0
|
||||
|
||||
|
||||
class FeedbackEligibilityResponse(BaseModel):
|
||||
"""Response for feedback eligibility check."""
|
||||
eligible: bool
|
||||
has_submitted: bool
|
||||
jobs_completed: int
|
||||
credits_reward: int = 1
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
@@ -1,48 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Download ensemble preset models at Docker build time.
|
||||
|
||||
Bakes models into the image so they're available immediately at runtime,
|
||||
avoiding cold-start model download latency in Cloud Run Jobs.
|
||||
|
||||
Usage: python backend/scripts/download_models.py /models
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import logging
|
||||
import importlib.resources as resources
|
||||
from audio_separator.separator import Separator
|
||||
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Presets to bake into the GPU image
|
||||
PRESETS_TO_DOWNLOAD = ["instrumental_clean", "karaoke"]
|
||||
|
||||
|
||||
def download_preset_models(model_dir: str) -> None:
|
||||
"""Download all models for the configured presets individually.
|
||||
|
||||
Ensemble presets resolve to multiple model filenames. We must download
|
||||
each model individually via load_model(model_filename) — calling
|
||||
load_model() with no args on an ensemble preset doesn't trigger downloads.
|
||||
"""
|
||||
with resources.open_text("audio_separator", "ensemble_presets.json") as f:
|
||||
presets = json.load(f)["presets"]
|
||||
|
||||
models_to_download = set()
|
||||
for preset_name in PRESETS_TO_DOWNLOAD:
|
||||
models_to_download.update(presets[preset_name]["models"])
|
||||
|
||||
logger.info(f"Downloading {len(models_to_download)} models for presets: {PRESETS_TO_DOWNLOAD}")
|
||||
for model in sorted(models_to_download):
|
||||
logger.info(f" Downloading: {model}")
|
||||
sep = Separator(model_file_dir=model_dir)
|
||||
sep.load_model(model)
|
||||
logger.info(f" Done: {model}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
model_dir = sys.argv[1] if len(sys.argv) > 1 else "/models"
|
||||
download_preset_models(model_dir)
|
||||
logger.info(f"All preset models downloaded to {model_dir}")
|
||||
@@ -1,265 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Generate a Mermaid state diagram from the job state machine.
|
||||
|
||||
This script reads the STATE_TRANSITIONS dictionary from models/job.py
|
||||
and generates a Mermaid diagram that can be embedded in documentation.
|
||||
|
||||
Usage:
|
||||
python scripts/generate_state_diagram.py > docs/state-machine.mmd
|
||||
# Or to update ARCHITECTURE.md directly:
|
||||
python scripts/generate_state_diagram.py --update-docs
|
||||
|
||||
Created as part of state machine robustness improvements (2026-02-02).
|
||||
"""
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
# Add backend to path for imports
|
||||
backend_dir = Path(__file__).parent.parent
|
||||
sys.path.insert(0, str(backend_dir))
|
||||
|
||||
# Import the state machine data
|
||||
# Note: Must be run from an environment where karaoke_gen is installed
|
||||
try:
|
||||
from models.job import STATE_TRANSITIONS, JobStatus
|
||||
except ImportError:
|
||||
# Fallback: define the state machine inline for documentation purposes
|
||||
from enum import Enum
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
PENDING = "pending"
|
||||
SEARCHING_AUDIO = "searching_audio"
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection"
|
||||
DOWNLOADING_AUDIO = "downloading_audio"
|
||||
DOWNLOADING = "downloading"
|
||||
SEPARATING_STAGE1 = "separating_stage1"
|
||||
SEPARATING_STAGE2 = "separating_stage2"
|
||||
AUDIO_COMPLETE = "audio_complete"
|
||||
TRANSCRIBING = "transcribing"
|
||||
CORRECTING = "correcting"
|
||||
LYRICS_COMPLETE = "lyrics_complete"
|
||||
GENERATING_SCREENS = "generating_screens"
|
||||
APPLYING_PADDING = "applying_padding"
|
||||
AWAITING_REVIEW = "awaiting_review"
|
||||
IN_REVIEW = "in_review"
|
||||
REVIEW_COMPLETE = "review_complete"
|
||||
RENDERING_VIDEO = "rendering_video"
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection"
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected"
|
||||
GENERATING_VIDEO = "generating_video"
|
||||
ENCODING = "encoding"
|
||||
PACKAGING = "packaging"
|
||||
UPLOADING = "uploading"
|
||||
NOTIFYING = "notifying"
|
||||
COMPLETE = "complete"
|
||||
PREP_COMPLETE = "prep_complete"
|
||||
FAILED = "failed"
|
||||
CANCELLED = "cancelled"
|
||||
QUEUED = "queued"
|
||||
PROCESSING = "processing"
|
||||
ERROR = "error"
|
||||
READY_FOR_FINALIZATION = "ready_for_finalization"
|
||||
FINALIZING = "finalizing"
|
||||
|
||||
STATE_TRANSITIONS = {
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.FAILED],
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
}
|
||||
|
||||
|
||||
def generate_mermaid() -> str:
|
||||
"""
|
||||
Generate a Mermaid state diagram from STATE_TRANSITIONS.
|
||||
|
||||
Returns:
|
||||
Mermaid diagram as a string
|
||||
"""
|
||||
lines = ["stateDiagram-v2"]
|
||||
lines.append(" direction TB")
|
||||
lines.append("")
|
||||
|
||||
# Group states by category for better visualization
|
||||
categories = {
|
||||
"Initial": [JobStatus.PENDING],
|
||||
"Audio Search": [
|
||||
JobStatus.SEARCHING_AUDIO,
|
||||
JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
JobStatus.DOWNLOADING_AUDIO,
|
||||
],
|
||||
"Processing": [
|
||||
JobStatus.DOWNLOADING,
|
||||
JobStatus.SEPARATING_STAGE1,
|
||||
JobStatus.SEPARATING_STAGE2,
|
||||
JobStatus.AUDIO_COMPLETE,
|
||||
JobStatus.TRANSCRIBING,
|
||||
JobStatus.CORRECTING,
|
||||
JobStatus.LYRICS_COMPLETE,
|
||||
],
|
||||
"Screens": [
|
||||
JobStatus.GENERATING_SCREENS,
|
||||
JobStatus.APPLYING_PADDING,
|
||||
],
|
||||
"Review": [
|
||||
JobStatus.AWAITING_REVIEW,
|
||||
JobStatus.IN_REVIEW,
|
||||
JobStatus.REVIEW_COMPLETE,
|
||||
],
|
||||
"Video": [
|
||||
JobStatus.RENDERING_VIDEO,
|
||||
JobStatus.INSTRUMENTAL_SELECTED,
|
||||
JobStatus.GENERATING_VIDEO,
|
||||
JobStatus.ENCODING,
|
||||
JobStatus.PACKAGING,
|
||||
],
|
||||
"Distribution": [
|
||||
JobStatus.UPLOADING,
|
||||
JobStatus.NOTIFYING,
|
||||
],
|
||||
"Terminal": [
|
||||
JobStatus.COMPLETE,
|
||||
JobStatus.PREP_COMPLETE,
|
||||
JobStatus.FAILED,
|
||||
JobStatus.CANCELLED,
|
||||
],
|
||||
}
|
||||
|
||||
# Add state definitions with notes for key states
|
||||
lines.append(" %% State definitions")
|
||||
key_states = {
|
||||
JobStatus.PENDING: "Job created",
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: "⚠️ USER ACTION",
|
||||
JobStatus.AWAITING_REVIEW: "⚠️ USER ACTION",
|
||||
JobStatus.COMPLETE: "✅ Success",
|
||||
JobStatus.FAILED: "❌ Error",
|
||||
}
|
||||
|
||||
for status, note in key_states.items():
|
||||
lines.append(f" {status.value}: {status.value}")
|
||||
lines.append(f" note right of {status.value}: {note}")
|
||||
|
||||
lines.append("")
|
||||
lines.append(" %% Transitions")
|
||||
|
||||
# Generate transitions
|
||||
for from_status, to_statuses in STATE_TRANSITIONS.items():
|
||||
for to_status in to_statuses:
|
||||
# Skip deprecated statuses
|
||||
if to_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
|
||||
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
|
||||
continue
|
||||
if from_status in [JobStatus.QUEUED, JobStatus.PROCESSING, JobStatus.ERROR,
|
||||
JobStatus.READY_FOR_FINALIZATION, JobStatus.FINALIZING]:
|
||||
continue
|
||||
|
||||
lines.append(f" {from_status.value} --> {to_status.value}")
|
||||
|
||||
# Add start and end markers
|
||||
lines.append("")
|
||||
lines.append(" %% Start and end")
|
||||
lines.append(" [*] --> pending")
|
||||
lines.append(" complete --> [*]")
|
||||
lines.append(" prep_complete --> [*]")
|
||||
lines.append(" failed --> [*]")
|
||||
lines.append(" cancelled --> [*]")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def generate_markdown_section() -> str:
|
||||
"""
|
||||
Generate a markdown section with the state diagram.
|
||||
|
||||
Returns:
|
||||
Markdown string with the diagram
|
||||
"""
|
||||
diagram = generate_mermaid()
|
||||
|
||||
return f"""## Job State Machine
|
||||
|
||||
The following diagram shows all valid state transitions for a karaoke generation job.
|
||||
|
||||
Key states:
|
||||
- **PENDING**: Initial state when job is created
|
||||
- **AWAITING_AUDIO_SELECTION**: Human selects audio source (for artist+title search)
|
||||
- **AWAITING_REVIEW**: Human reviews lyrics and selects instrumental
|
||||
- **COMPLETE**: All processing finished successfully
|
||||
|
||||
```mermaid
|
||||
{diagram}
|
||||
```
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
Jobs use parallel processing for audio separation and lyrics transcription:
|
||||
|
||||
1. **DOWNLOADING** triggers both:
|
||||
- Audio worker → SEPARATING_STAGE1 → SEPARATING_STAGE2 → AUDIO_COMPLETE
|
||||
- Lyrics worker → TRANSCRIBING → CORRECTING → LYRICS_COMPLETE
|
||||
|
||||
2. When BOTH complete, the screens worker is triggered automatically.
|
||||
|
||||
### Human Review Points
|
||||
|
||||
Jobs wait for human input at these points:
|
||||
- **AWAITING_AUDIO_SELECTION**: User selects audio source from search results
|
||||
- **AWAITING_REVIEW**: User reviews/corrects lyrics AND selects instrumental
|
||||
|
||||
### Error Handling
|
||||
|
||||
- Any state can transition to **FAILED** on unrecoverable errors
|
||||
- **CANCELLED** is set when user cancels the job
|
||||
|
||||
"""
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Generate state machine diagram")
|
||||
parser.add_argument(
|
||||
"--update-docs",
|
||||
action="store_true",
|
||||
help="Update ARCHITECTURE.md with the diagram"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--markdown",
|
||||
action="store_true",
|
||||
help="Output markdown section instead of raw Mermaid"
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.update_docs:
|
||||
# TODO: Implement auto-update of ARCHITECTURE.md
|
||||
print("Auto-update of ARCHITECTURE.md not yet implemented")
|
||||
print("Please copy the output below to the appropriate section:")
|
||||
print()
|
||||
print(generate_markdown_section())
|
||||
elif args.markdown:
|
||||
print(generate_markdown_section())
|
||||
else:
|
||||
print(generate_mermaid())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -5,7 +5,6 @@ This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio analysis capabilities.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
@@ -188,78 +187,7 @@ class AudioAnalysisService:
|
||||
)
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def cache_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
cache_gcs_path: str,
|
||||
num_points: int = 1000,
|
||||
) -> tuple[list[float], float]:
|
||||
"""
|
||||
Generate waveform data and cache it as JSON in GCS.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
cache_gcs_path: GCS path to store the cached waveform JSON
|
||||
num_points: Number of data points to generate
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Generating and caching waveform data: {gcs_audio_path}")
|
||||
|
||||
amplitudes, duration = self.get_waveform_data(
|
||||
gcs_audio_path=gcs_audio_path,
|
||||
job_id=job_id,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
cache_data = {
|
||||
"amplitudes": list(amplitudes),
|
||||
"duration_seconds": duration,
|
||||
"num_points": num_points,
|
||||
"source_gcs_path": gcs_audio_path,
|
||||
}
|
||||
self.storage_service.upload_json(cache_gcs_path, cache_data)
|
||||
logger.info(f"[{job_id}] Waveform data cached at {cache_gcs_path}")
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def load_cached_waveform(
|
||||
self,
|
||||
cache_gcs_path: str,
|
||||
) -> Optional[tuple[list[float], float]]:
|
||||
"""
|
||||
Load cached waveform data from GCS.
|
||||
|
||||
Args:
|
||||
cache_gcs_path: GCS path to the cached waveform JSON
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds) or None if not found
|
||||
"""
|
||||
try:
|
||||
if not self.storage_service.file_exists(cache_gcs_path):
|
||||
return None
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_path = os.path.join(temp_dir, "waveform.json")
|
||||
self.storage_service.download_file(cache_gcs_path, local_path)
|
||||
with open(local_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
amplitudes = data.get("amplitudes", [])
|
||||
duration = data.get("duration_seconds", 0.0)
|
||||
if not amplitudes:
|
||||
return None
|
||||
|
||||
return amplitudes, duration
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load cached waveform from {cache_gcs_path}: {e}")
|
||||
return None
|
||||
|
||||
|
||||
def generate_waveform_with_mutes(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
|
||||
@@ -1,180 +0,0 @@
|
||||
"""
|
||||
Audio editing service — server-side FFmpeg operations for trim, cut, mute, join.
|
||||
|
||||
All operations are lossless (FLAC in, FLAC out). Playback URLs use OGG Opus
|
||||
via the existing AudioTranscodingService.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
import subprocess
|
||||
import tempfile
|
||||
from dataclasses import dataclass, asdict
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class AudioMetadata:
|
||||
duration_seconds: float
|
||||
sample_rate: int
|
||||
channels: int
|
||||
format: str
|
||||
file_size_bytes: int
|
||||
|
||||
|
||||
class AudioEditService:
|
||||
"""Server-side audio editing via FFmpeg."""
|
||||
|
||||
def __init__(self, storage_service: StorageService | None = None):
|
||||
self.storage = storage_service or StorageService()
|
||||
|
||||
def get_metadata(self, audio_path: str) -> AudioMetadata:
|
||||
"""Get audio metadata from a local file using ffprobe."""
|
||||
cmd = [
|
||||
"ffprobe", "-v", "quiet", "-print_format", "json",
|
||||
"-show_format", "-show_streams", audio_path,
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"ffprobe failed: {result.stderr}")
|
||||
|
||||
info = json.loads(result.stdout)
|
||||
fmt = info.get("format", {})
|
||||
streams = info.get("streams", [])
|
||||
audio_stream = next((s for s in streams if s.get("codec_type") == "audio"), {})
|
||||
|
||||
return AudioMetadata(
|
||||
duration_seconds=float(fmt.get("duration", 0)),
|
||||
sample_rate=int(audio_stream.get("sample_rate", 44100)),
|
||||
channels=int(audio_stream.get("channels", 2)),
|
||||
format=fmt.get("format_name", "unknown"),
|
||||
file_size_bytes=int(fmt.get("size", 0)),
|
||||
)
|
||||
|
||||
def get_metadata_from_gcs(self, gcs_path: str) -> AudioMetadata:
|
||||
"""Download from GCS and get metadata."""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_path = os.path.join(temp_dir, "audio")
|
||||
self.storage.download_file(gcs_path, local_path)
|
||||
return self.get_metadata(local_path)
|
||||
|
||||
def trim_start(self, input_path: str, end_seconds: float, output_path: str) -> AudioMetadata:
|
||||
"""Remove audio from 0 to end_seconds (skip the first N seconds)."""
|
||||
self._run_ffmpeg([
|
||||
"-ss", str(end_seconds),
|
||||
"-i", input_path,
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def trim_end(self, input_path: str, start_seconds: float, output_path: str) -> AudioMetadata:
|
||||
"""Keep audio from 0 to start_seconds (remove from start_seconds to end)."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-t", str(start_seconds),
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def cut_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
|
||||
"""Remove a region from start to end, joining the remaining parts."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-filter_complex",
|
||||
f"[0]atrim=0:{start},asetpts=PTS-STARTPTS[a];"
|
||||
f"[0]atrim={end},asetpts=PTS-STARTPTS[b];"
|
||||
f"[a][b]concat=n=2:v=0:a=1[out]",
|
||||
"-map", "[out]",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def mute_region(self, input_path: str, start: float, end: float, output_path: str) -> AudioMetadata:
|
||||
"""Silence a region (preserve duration)."""
|
||||
self._run_ffmpeg([
|
||||
"-i", input_path,
|
||||
"-af", f"volume=enable='between(t,{start},{end})':volume=0",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def join_audio(self, input_path: str, other_path: str, position: str, output_path: str) -> AudioMetadata:
|
||||
"""Join two audio files. position: 'start' (prepend other) or 'end' (append other)."""
|
||||
if position == "start":
|
||||
first, second = other_path, input_path
|
||||
else:
|
||||
first, second = input_path, other_path
|
||||
|
||||
self._run_ffmpeg([
|
||||
"-i", first,
|
||||
"-i", second,
|
||||
"-filter_complex", "[0:a][1:a]concat=n=2:v=0:a=1[out]",
|
||||
"-map", "[out]",
|
||||
"-c:a", "flac",
|
||||
output_path,
|
||||
])
|
||||
return self.get_metadata(output_path)
|
||||
|
||||
def apply_edit(
|
||||
self,
|
||||
input_gcs_path: str,
|
||||
operation: str,
|
||||
params: dict,
|
||||
output_gcs_path: str,
|
||||
job_id: str,
|
||||
) -> tuple[AudioMetadata, str]:
|
||||
"""
|
||||
Apply an edit operation to an audio file in GCS.
|
||||
|
||||
Returns (metadata, output_gcs_path).
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_input = os.path.join(temp_dir, "input.flac")
|
||||
local_output = os.path.join(temp_dir, "output.flac")
|
||||
|
||||
# Download current audio
|
||||
self.storage.download_file(input_gcs_path, local_input)
|
||||
|
||||
# Apply the operation
|
||||
if operation == "trim_start":
|
||||
metadata = self.trim_start(local_input, params["end_seconds"], local_output)
|
||||
elif operation == "trim_end":
|
||||
metadata = self.trim_end(local_input, params["start_seconds"], local_output)
|
||||
elif operation == "cut":
|
||||
metadata = self.cut_region(
|
||||
local_input, params["start_seconds"], params["end_seconds"], local_output
|
||||
)
|
||||
elif operation == "mute":
|
||||
metadata = self.mute_region(
|
||||
local_input, params["start_seconds"], params["end_seconds"], local_output
|
||||
)
|
||||
elif operation in ("join_start", "join_end"):
|
||||
# Download the upload file
|
||||
upload_gcs_path = params["upload_gcs_path"]
|
||||
local_other = os.path.join(temp_dir, "other.flac")
|
||||
self.storage.download_file(upload_gcs_path, local_other)
|
||||
position = "start" if operation == "join_start" else "end"
|
||||
metadata = self.join_audio(local_input, local_other, position, local_output)
|
||||
else:
|
||||
raise ValueError(f"Unknown operation: {operation}")
|
||||
|
||||
# Upload edited audio
|
||||
self.storage.upload_file(local_output, output_gcs_path)
|
||||
logger.info(f"[{job_id}] Applied {operation}, uploaded to {output_gcs_path}")
|
||||
|
||||
return metadata, output_gcs_path
|
||||
|
||||
def _run_ffmpeg(self, args: list[str]) -> None:
|
||||
"""Run an ffmpeg command with standard options."""
|
||||
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-y"] + args
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(f"ffmpeg failed (exit {result.returncode}): {result.stderr}")
|
||||
@@ -204,9 +204,7 @@ class AudioSearchService:
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
# source_id: use explicit source_id field first (YouTube video ID, Spotify track ID),
|
||||
# fall back to info_hash for torrent sources (RED/OPS)
|
||||
source_id=item.get("source_id") or item.get("info_hash"),
|
||||
source_id=item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
@@ -612,13 +610,13 @@ class AudioSearchService:
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Runs the sync search in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
@@ -630,10 +628,10 @@ class AudioSearchService:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Runs the sync download in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
@@ -651,11 +649,8 @@ class AudioSearchService:
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
|
||||
Runs the sync download_by_id in a thread executor to avoid blocking
|
||||
the event loop.
|
||||
"""
|
||||
loop = asyncio.get_running_loop()
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
|
||||
@@ -1,181 +0,0 @@
|
||||
"""
|
||||
Audio transcoding service for review UI optimization.
|
||||
|
||||
Transcodes FLAC audio files to OGG Opus for efficient browser playback.
|
||||
Original FLAC files are never modified — transcoded versions are cached
|
||||
in GCS at jobs/{job_id}/review-audio/{filename}.ogg.
|
||||
|
||||
Eager transcoding: called from screens_worker before AWAITING_REVIEW
|
||||
so all files are cached before the user opens the review UI.
|
||||
Falls back to original FLAC signed URLs if transcoded version is missing.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioTranscodingService:
|
||||
"""
|
||||
Transcodes audio files to OGG Opus and caches them in GCS.
|
||||
|
||||
Used by the review UI to serve ~3 MB OGG files instead of 35+ MB FLACs.
|
||||
"""
|
||||
|
||||
def __init__(self, storage_service: Optional[StorageService] = None):
|
||||
self.storage = storage_service or StorageService()
|
||||
|
||||
def _get_cache_path(self, source_gcs_path: str) -> str:
|
||||
"""
|
||||
Derive the cache path for a transcoded file.
|
||||
|
||||
Examples:
|
||||
jobs/abc123/input/song.flac -> jobs/abc123/review-audio/song.ogg
|
||||
jobs/abc123/stems/instrumental_clean.flac -> jobs/abc123/review-audio/instrumental_clean.ogg
|
||||
"""
|
||||
parts = Path(source_gcs_path).parts
|
||||
# Find "jobs/{job_id}" prefix
|
||||
try:
|
||||
jobs_idx = list(parts).index("jobs")
|
||||
job_id = parts[jobs_idx + 1]
|
||||
except (ValueError, IndexError):
|
||||
# Fallback: use a hash of the full path to avoid collisions
|
||||
job_id = "_" + hashlib.sha256(source_gcs_path.encode()).hexdigest()[:12]
|
||||
|
||||
filename = Path(source_gcs_path).stem + ".ogg"
|
||||
return f"jobs/{job_id}/review-audio/{filename}"
|
||||
|
||||
def _transcode_and_upload(self, source_gcs_path: str, cache_path: str) -> str:
|
||||
"""
|
||||
Download FLAC from GCS, transcode to OGG Opus, upload to cache path.
|
||||
|
||||
Returns the cache GCS path on success.
|
||||
Raises on failure.
|
||||
"""
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_input = os.path.join(temp_dir, "input.flac")
|
||||
local_output = os.path.join(temp_dir, "output.ogg")
|
||||
|
||||
# Download source
|
||||
self.storage.download_file(source_gcs_path, local_input)
|
||||
|
||||
# Transcode: FLAC -> OGG Opus 128kbps
|
||||
cmd = [
|
||||
"ffmpeg", "-hide_banner", "-loglevel", "error",
|
||||
"-y",
|
||||
"-i", local_input,
|
||||
"-c:a", "libopus",
|
||||
"-b:a", "128k",
|
||||
"-vn",
|
||||
local_output,
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
|
||||
if result.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"ffmpeg transcoding failed (exit {result.returncode}): {result.stderr}"
|
||||
)
|
||||
|
||||
# Upload transcoded file
|
||||
self.storage.upload_file(local_output, cache_path)
|
||||
|
||||
return cache_path
|
||||
|
||||
def transcode_if_needed(self, source_gcs_path: str) -> str:
|
||||
"""
|
||||
Return the GCS path of the transcoded OGG file, transcoding if not cached.
|
||||
|
||||
Returns the cache path (always an OGG file in review-audio/).
|
||||
Raises if transcoding fails.
|
||||
"""
|
||||
cache_path = self._get_cache_path(source_gcs_path)
|
||||
|
||||
if self.storage.file_exists(cache_path):
|
||||
logger.debug(f"Cache hit: {cache_path}")
|
||||
return cache_path
|
||||
|
||||
logger.info(f"Cache miss, transcoding: {source_gcs_path} -> {cache_path}")
|
||||
return self._transcode_and_upload(source_gcs_path, cache_path)
|
||||
|
||||
def get_review_audio_url(
|
||||
self, source_gcs_path: str, expiration_minutes: int = 120
|
||||
) -> str:
|
||||
"""
|
||||
Get a signed URL for the transcoded audio, falling back to original FLAC.
|
||||
"""
|
||||
try:
|
||||
cache_path = self.transcode_if_needed(source_gcs_path)
|
||||
return self.storage.generate_signed_url(cache_path, expiration_minutes)
|
||||
except Exception as e:
|
||||
# Source may be deleted but cache from earlier transcode persists
|
||||
try:
|
||||
cache_path = self._get_cache_path(source_gcs_path)
|
||||
if self.storage.file_exists(cache_path):
|
||||
logger.info(f"Source gone but cache exists, serving: {cache_path}")
|
||||
return self.storage.generate_signed_url(cache_path, expiration_minutes)
|
||||
except Exception:
|
||||
pass # Fall through to FLAC fallback below
|
||||
logger.warning(
|
||||
f"Transcoding failed for {source_gcs_path}, falling back to FLAC: {e}"
|
||||
)
|
||||
return self.storage.generate_signed_url(source_gcs_path, expiration_minutes)
|
||||
|
||||
async def get_review_audio_url_async(
|
||||
self, source_gcs_path: str, expiration_minutes: int = 120
|
||||
) -> str:
|
||||
"""Async wrapper around get_review_audio_url via asyncio.to_thread."""
|
||||
return await asyncio.to_thread(
|
||||
self.get_review_audio_url, source_gcs_path, expiration_minutes
|
||||
)
|
||||
|
||||
def prepare_review_audio_for_job(self, job) -> list[str]:
|
||||
"""
|
||||
Transcode all review audio files for a job (eager transcoding).
|
||||
|
||||
Called from screens_worker before AWAITING_REVIEW transition
|
||||
and from the admin backfill endpoint.
|
||||
|
||||
Returns list of cache paths that were transcoded.
|
||||
"""
|
||||
job_id = job.job_id
|
||||
transcoded = []
|
||||
|
||||
# Collect all audio paths to transcode
|
||||
paths_to_transcode = []
|
||||
|
||||
# Main input audio
|
||||
if job.input_media_gcs_path:
|
||||
paths_to_transcode.append(("input", job.input_media_gcs_path))
|
||||
|
||||
# Stems
|
||||
stems = job.file_urls.get("stems", {})
|
||||
for stem_key in ["instrumental_clean", "instrumental_with_backing", "backing_vocals"]:
|
||||
path = stems.get(stem_key)
|
||||
if path:
|
||||
paths_to_transcode.append((stem_key, path))
|
||||
|
||||
for label, source_path in paths_to_transcode:
|
||||
try:
|
||||
cache_path = self.transcode_if_needed(source_path)
|
||||
transcoded.append(cache_path)
|
||||
logger.info(f"[{job_id}] Transcoded {label}: {cache_path}")
|
||||
except Exception as e:
|
||||
# Non-fatal: review UI will fall back to FLAC
|
||||
logger.warning(
|
||||
f"[{job_id}] Failed to transcode {label} ({source_path}): {e}"
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Review audio preparation complete: "
|
||||
f"{len(transcoded)}/{len(paths_to_transcode)} files transcoded"
|
||||
)
|
||||
return transcoded
|
||||
@@ -3,7 +3,6 @@ Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- Google OIDC tokens (from Cloud Scheduler / Cloud Tasks service accounts)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
@@ -20,9 +19,6 @@ from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from google.auth.transport import requests as google_auth_requests
|
||||
from google.oauth2 import id_token as google_id_token
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
@@ -53,7 +49,6 @@ class AuthResult:
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
tenant_id: Optional[str] = None # Tenant scope from token or session
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
@@ -76,7 +71,6 @@ class AuthService:
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
self._scheduler_service_account = self.settings.scheduler_service_account
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
@@ -88,56 +82,6 @@ class AuthService:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
|
||||
|
||||
def _validate_oidc_token(self, token: str) -> Optional[AuthResult]:
|
||||
"""
|
||||
Validate a Google OIDC token (from Cloud Scheduler or Cloud Tasks).
|
||||
|
||||
Returns AuthResult if token is a valid Google OIDC JWT from an allowed
|
||||
service account, None if the token is not a JWT or validation fails.
|
||||
"""
|
||||
# Quick check: JWTs have 3 dot-separated parts
|
||||
if token.count('.') != 2:
|
||||
return None
|
||||
|
||||
try:
|
||||
# Verify the token signature and claims with Google
|
||||
# audience=None skips audience check — Cloud Scheduler sets audience
|
||||
# to the endpoint URL, which varies per endpoint
|
||||
claims = google_id_token.verify_oauth2_token(
|
||||
token,
|
||||
google_auth_requests.Request(),
|
||||
audience=None,
|
||||
)
|
||||
|
||||
token_email = claims.get("email", "")
|
||||
|
||||
# Only accept tokens from the configured service account
|
||||
if token_email != self._scheduler_service_account:
|
||||
logger.warning(
|
||||
f"OIDC token from unexpected service account: {token_email} "
|
||||
f"(expected {self._scheduler_service_account})"
|
||||
)
|
||||
return None
|
||||
|
||||
logger.info(f"OIDC token validated for service account: {token_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="OIDC service account access granted",
|
||||
user_email=token_email,
|
||||
is_admin=True,
|
||||
)
|
||||
|
||||
except ValueError as e:
|
||||
# verify_oauth2_token raises ValueError for invalid/expired tokens
|
||||
logger.debug(f"OIDC token validation failed: {e}")
|
||||
return None
|
||||
except Exception as e:
|
||||
# Network errors fetching Google certs, etc.
|
||||
logger.warning(f"OIDC token verification error: {e}")
|
||||
return None
|
||||
|
||||
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
|
||||
"""
|
||||
@@ -268,12 +212,6 @@ class AuthService:
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check for Google OIDC tokens (Cloud Scheduler / Cloud Tasks)
|
||||
# Done before Firestore lookup to avoid wasting a read on JWT strings
|
||||
oidc_result = self._validate_oidc_token(token)
|
||||
if oidc_result is not None:
|
||||
return oidc_result
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
@@ -346,7 +284,6 @@ class AuthService:
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
token_tenant_id = token_data.get("tenant_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
@@ -370,8 +307,7 @@ class AuthService:
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
@@ -384,8 +320,7 @@ class AuthService:
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
@@ -398,8 +333,7 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -409,8 +343,7 @@ class AuthService:
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
@@ -424,8 +357,7 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
@@ -439,8 +371,7 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -450,8 +381,7 @@ class AuthService:
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -461,8 +391,7 @@ class AuthService:
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
@@ -477,8 +406,7 @@ class AuthService:
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
@@ -487,8 +415,7 @@ class AuthService:
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -498,8 +425,7 @@ class AuthService:
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id,
|
||||
tenant_id=token_tenant_id
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
@@ -511,11 +437,10 @@ class AuthService:
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for an auth_token and track the job.
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
Note: Credit deduction for session-based users is handled in
|
||||
JobManager.create_job(), not here. This method only tracks usage
|
||||
for legacy auth_tokens stored in Firestore.
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
@@ -540,11 +465,32 @@ class AuthService:
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Only increment usage for auth_tokens in Firestore
|
||||
# Session tokens (magic link auth) have credits managed by JobManager.create_job()
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
return True
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
|
||||
@@ -1,225 +0,0 @@
|
||||
"""
|
||||
Brand code allocation service using Firestore atomic counters.
|
||||
|
||||
Replaces the previous Dropbox folder-scanning approach which was susceptible to
|
||||
TOCTOU race conditions when concurrent jobs read the same folder state.
|
||||
|
||||
Uses Firestore transactions to atomically allocate unique brand codes.
|
||||
Supports recycling numbers when E2E cleanup deletes test brand codes.
|
||||
"""
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
BRAND_CODE_COUNTERS_COLLECTION = "brand_code_counters"
|
||||
|
||||
# Singleton instance
|
||||
_brand_code_service: Optional["BrandCodeService"] = None
|
||||
|
||||
|
||||
class BrandCodeService:
|
||||
"""Atomic brand code allocation using Firestore counters."""
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
if db is None:
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
else:
|
||||
self.db = db
|
||||
|
||||
def allocate_brand_code(self, prefix: str, dropbox_path: str) -> str:
|
||||
"""
|
||||
Atomically allocate the next brand code for a given prefix.
|
||||
|
||||
Uses a Firestore transaction to ensure no two concurrent callers
|
||||
receive the same number. If recycled numbers are available (from
|
||||
E2E cleanup), the smallest recycled number is reused first.
|
||||
|
||||
On first use for a prefix, initializes the counter by scanning
|
||||
Dropbox to find the current max number.
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix (e.g., "NOMAD" or "NOMADNP")
|
||||
dropbox_path: Dropbox path for one-time initialization scan
|
||||
|
||||
Returns:
|
||||
Brand code string (e.g., "NOMAD-1264")
|
||||
"""
|
||||
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
|
||||
|
||||
@firestore.transactional
|
||||
def allocate_in_transaction(transaction):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
# First use — initialize from Dropbox scan
|
||||
next_number = self._get_initial_next_number(prefix, dropbox_path)
|
||||
allocated = next_number
|
||||
transaction.set(doc_ref, {
|
||||
"prefix": prefix,
|
||||
"next_number": next_number + 1,
|
||||
"recycled": [],
|
||||
"initialized_at": datetime.now(timezone.utc),
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
return allocated
|
||||
|
||||
data = doc.to_dict()
|
||||
recycled = data.get("recycled", [])
|
||||
|
||||
if recycled:
|
||||
# Reuse smallest recycled number
|
||||
recycled.sort()
|
||||
allocated = recycled.pop(0)
|
||||
transaction.update(doc_ref, {
|
||||
"recycled": recycled,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
else:
|
||||
# Allocate next sequential number
|
||||
allocated = data["next_number"]
|
||||
transaction.update(doc_ref, {
|
||||
"next_number": allocated + 1,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
|
||||
return allocated
|
||||
|
||||
transaction = self.db.transaction()
|
||||
number = allocate_in_transaction(transaction)
|
||||
brand_code = f"{prefix}-{number:04d}"
|
||||
logger.info(f"Allocated brand code: {brand_code}")
|
||||
return brand_code
|
||||
|
||||
def recycle_brand_code(self, prefix: str, number: int) -> None:
|
||||
"""
|
||||
Return a brand code number to the recycled pool.
|
||||
|
||||
Called when E2E cleanup deletes a brand code's distribution.
|
||||
The number will be reused on the next allocation for this prefix.
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix (e.g., "NOMAD")
|
||||
number: The numeric portion to recycle (e.g., 1234)
|
||||
"""
|
||||
doc_ref = self.db.collection(BRAND_CODE_COUNTERS_COLLECTION).document(prefix)
|
||||
|
||||
@firestore.transactional
|
||||
def recycle_in_transaction(transaction):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
|
||||
if not doc.exists:
|
||||
# Counter doesn't exist yet — nothing to recycle into.
|
||||
# The number will naturally be below next_number when
|
||||
# the counter is initialized, so it would create a gap.
|
||||
# Just log and skip.
|
||||
logger.warning(
|
||||
f"Cannot recycle {prefix}-{number:04d}: counter doc doesn't exist"
|
||||
)
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
next_number = data.get("next_number", 0)
|
||||
|
||||
if number <= 0 or number >= next_number:
|
||||
logger.warning(
|
||||
f"Cannot recycle {prefix}-{number:04d}: "
|
||||
f"invalid (next_number={next_number})"
|
||||
)
|
||||
return
|
||||
|
||||
recycled = data.get("recycled", [])
|
||||
|
||||
if number not in recycled:
|
||||
recycled.append(number)
|
||||
transaction.update(doc_ref, {
|
||||
"recycled": recycled,
|
||||
"last_updated": datetime.now(timezone.utc),
|
||||
})
|
||||
logger.info(f"Recycled brand code number: {prefix}-{number:04d}")
|
||||
else:
|
||||
logger.warning(f"Number {number} already in recycled pool for {prefix}")
|
||||
|
||||
transaction = self.db.transaction()
|
||||
recycle_in_transaction(transaction)
|
||||
|
||||
def _get_initial_next_number(self, prefix: str, dropbox_path: str) -> int:
|
||||
"""
|
||||
Scan Dropbox once to determine the starting next_number.
|
||||
|
||||
This is only called when the counter document doesn't exist yet
|
||||
(one-time migration from the old Dropbox-scanning approach).
|
||||
|
||||
Args:
|
||||
prefix: Brand prefix to scan for
|
||||
dropbox_path: Dropbox path containing organized folders
|
||||
|
||||
Returns:
|
||||
The next number to allocate (max existing + 1, minimum 1)
|
||||
"""
|
||||
try:
|
||||
from backend.services.dropbox_service import get_dropbox_service
|
||||
|
||||
dropbox = get_dropbox_service()
|
||||
if not dropbox.is_configured:
|
||||
logger.warning(
|
||||
f"Dropbox not configured for initialization of {prefix} counter, starting at 1"
|
||||
)
|
||||
return 1
|
||||
|
||||
folders = dropbox.list_folders(dropbox_path)
|
||||
pattern = re.compile(rf"^{re.escape(prefix)}-(\d{{4}})")
|
||||
|
||||
existing_nums: set[int] = set()
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
existing_nums.add(int(match.group(1)))
|
||||
|
||||
if existing_nums:
|
||||
next_num = max(existing_nums) + 1
|
||||
logger.info(
|
||||
f"Initialized {prefix} counter from Dropbox: "
|
||||
f"max existing={max(existing_nums)}, next_number={next_num}"
|
||||
)
|
||||
return next_num
|
||||
|
||||
logger.info(f"No existing {prefix} codes found in Dropbox, starting at 1")
|
||||
return 1
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to initialize {prefix} counter from Dropbox: {e}")
|
||||
raise
|
||||
|
||||
@staticmethod
|
||||
def parse_brand_code(brand_code: str) -> tuple[str, int]:
|
||||
"""
|
||||
Parse a brand code string into prefix and number.
|
||||
|
||||
Args:
|
||||
brand_code: e.g., "NOMAD-1234" or "NOMADNP-0012"
|
||||
|
||||
Returns:
|
||||
Tuple of (prefix, number), e.g., ("NOMAD", 1234)
|
||||
|
||||
Raises:
|
||||
ValueError: If brand_code doesn't match expected format
|
||||
"""
|
||||
match = re.match(r"^([A-Z]+)-(\d+)$", brand_code)
|
||||
if not match:
|
||||
raise ValueError(f"Invalid brand code format: {brand_code}")
|
||||
return match.group(1), int(match.group(2))
|
||||
|
||||
|
||||
def get_brand_code_service() -> BrandCodeService:
|
||||
"""Get or create the singleton BrandCodeService instance."""
|
||||
global _brand_code_service
|
||||
if _brand_code_service is None:
|
||||
_brand_code_service = BrandCodeService()
|
||||
return _brand_code_service
|
||||
@@ -1,105 +0,0 @@
|
||||
"""
|
||||
Proxy service for karaoke-decide catalog API.
|
||||
|
||||
Forwards artist and track search requests to the karaoke-decide backend,
|
||||
providing autocomplete data from MusicBrainz + Spotify catalogs.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# In-memory TTL cache: { cache_key: (expiry_timestamp, data) }
|
||||
_cache: dict[str, tuple[float, Any]] = {}
|
||||
CACHE_TTL_SECONDS = 300 # 5 minutes
|
||||
|
||||
KARAOKE_DECIDE_BASE_URL = "https://decide.nomadkaraoke.com"
|
||||
REQUEST_TIMEOUT = 10 # seconds
|
||||
|
||||
|
||||
def _get_base_url() -> str:
|
||||
"""Get the karaoke-decide API base URL from config or default."""
|
||||
return getattr(settings, "karaoke_decide_api_url", None) or KARAOKE_DECIDE_BASE_URL
|
||||
|
||||
|
||||
def _cache_get(key: str) -> Any | None:
|
||||
"""Get a value from cache if it exists and hasn't expired."""
|
||||
if key in _cache:
|
||||
expiry, data = _cache[key]
|
||||
if time.monotonic() < expiry:
|
||||
return data
|
||||
del _cache[key]
|
||||
return None
|
||||
|
||||
|
||||
def _cache_set(key: str, data: Any, ttl: float = CACHE_TTL_SECONDS) -> None:
|
||||
"""Store a value in cache with TTL."""
|
||||
_cache[key] = (time.monotonic() + ttl, data)
|
||||
|
||||
|
||||
async def search_artists(query: str, limit: int = 10) -> list[dict]:
|
||||
"""
|
||||
Search for artists via karaoke-decide catalog API.
|
||||
|
||||
Returns list of artist dicts with canonical MusicBrainz names.
|
||||
Falls back to empty list on any error.
|
||||
"""
|
||||
cache_key = f"artists:{query.lower()}:{limit}"
|
||||
cached = _cache_get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
base_url = _get_base_url()
|
||||
url = f"{base_url}/api/catalog/artists"
|
||||
params = {"q": query, "limit": limit}
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get(url, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# karaoke-decide wraps results: {"artists": [...], "total": N}
|
||||
results = data.get("artists", data) if isinstance(data, dict) else data
|
||||
_cache_set(cache_key, results)
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to search artists from karaoke-decide: {e}")
|
||||
return []
|
||||
|
||||
|
||||
async def search_tracks(query: str, artist: str | None = None, limit: int = 10) -> list[dict]:
|
||||
"""
|
||||
Search for tracks via karaoke-decide catalog API.
|
||||
|
||||
Returns list of track dicts with canonical Spotify/MusicBrainz names.
|
||||
Falls back to empty list on any error.
|
||||
"""
|
||||
cache_key = f"tracks:{query.lower()}:{(artist or '').lower()}:{limit}"
|
||||
cached = _cache_get(cache_key)
|
||||
if cached is not None:
|
||||
return cached
|
||||
|
||||
base_url = _get_base_url()
|
||||
url = f"{base_url}/api/catalog/tracks"
|
||||
params: dict[str, Any] = {"q": query, "limit": limit}
|
||||
if artist:
|
||||
params["artist"] = artist
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=REQUEST_TIMEOUT) as client:
|
||||
resp = await client.get(url, params=params)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
# karaoke-decide wraps results: {"tracks": [...], "total": N}
|
||||
results = data.get("tracks", data) if isinstance(data, dict) else data
|
||||
_cache_set(cache_key, results)
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to search tracks from karaoke-decide: {e}")
|
||||
return []
|
||||
@@ -1,441 +0,0 @@
|
||||
"""
|
||||
AI-powered credit evaluation service.
|
||||
|
||||
Evaluates whether to grant free credits (welcome or feedback) by collecting
|
||||
abuse signals and asking Gemini for a grant/deny decision. Fail-open: if
|
||||
the AI call fails for any reason, credits are granted.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import uuid
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CREDIT_EVALUATIONS_COLLECTION = "credit_evaluations"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CreditEvaluation:
|
||||
"""Result of an AI credit evaluation."""
|
||||
decision: str # "grant" or "deny"
|
||||
reasoning: str
|
||||
confidence: float
|
||||
error: Optional[str] = None # Set if evaluation failed (decision defaults to "grant")
|
||||
signals: Optional[dict] = None # Collected abuse signals (for admin review emails)
|
||||
|
||||
|
||||
SYSTEM_PROMPT = """You are an anti-abuse evaluator for Nomad Karaoke, a web service that creates professional karaoke videos. Each job costs the service real money in API credits and cloud compute.
|
||||
|
||||
New users get 1 free welcome credit to try the service. Users who complete 2 jobs and submit feedback get 1 more credit. Some users abuse this by creating multiple accounts to get unlimited free karaoke generation.
|
||||
|
||||
Your job: Given the signals below, decide whether to GRANT or DENY free credits to this user.
|
||||
|
||||
## Decision Guidelines
|
||||
|
||||
**DENY if any of these are true:**
|
||||
- The user's device fingerprint matches other accounts (strong signal of multi-accounting)
|
||||
- The user's signup IP matches other accounts AND those accounts also have suspicious patterns (e.g., no spend, same user agent)
|
||||
- There's a clear pattern of account creation from the same device/network to farm free credits
|
||||
|
||||
**GRANT if:**
|
||||
- This appears to be a genuine new user with no suspicious correlations
|
||||
- The IP match is explainable (e.g., different fingerprints suggest different actual users on a shared network like a school or office)
|
||||
- There's insufficient evidence to confidently flag abuse
|
||||
|
||||
**When in doubt, GRANT.** False positives (blocking real users) are worse than false negatives (letting an occasional abuser through). We'd rather lose a few dollars than alienate a real customer.
|
||||
|
||||
## Response Format
|
||||
|
||||
Respond with ONLY a JSON object, no other text:
|
||||
{"decision": "grant" or "deny", "reasoning": "Brief explanation", "confidence": 0.0 to 1.0}
|
||||
"""
|
||||
|
||||
|
||||
def _build_user_prompt(
|
||||
email: str,
|
||||
grant_type: str,
|
||||
user_data: dict,
|
||||
fingerprint_matches: list,
|
||||
ip_matches: list,
|
||||
ip_geo: Optional[dict],
|
||||
recent_signups_ip: int,
|
||||
recent_signups_fp: int,
|
||||
user_agent: Optional[str],
|
||||
feedback_content: Optional[dict] = None,
|
||||
) -> str:
|
||||
"""Build the user prompt with all abuse signals."""
|
||||
lines = [
|
||||
f"## Credit Grant Request",
|
||||
f"- **Email:** {email}",
|
||||
f"- **Grant type:** {grant_type} credits",
|
||||
f"- **User's device fingerprint:** {user_data.get('device_fingerprint', 'unknown')}",
|
||||
f"- **User's signup IP:** {user_data.get('signup_ip', 'unknown')}",
|
||||
f"- **User agent:** {user_agent or 'unknown'}",
|
||||
f"- **Account created:** {user_data.get('created_at', 'unknown')}",
|
||||
f"- **Jobs created:** {user_data.get('total_jobs_created', 0)}",
|
||||
f"- **Jobs completed:** {user_data.get('total_jobs_completed', 0)}",
|
||||
f"- **Total spent (real money):** ${user_data.get('total_spent', 0) / 100:.2f}",
|
||||
f"- **Current credits:** {user_data.get('credits', 0)}",
|
||||
"",
|
||||
]
|
||||
|
||||
# IP geolocation
|
||||
if ip_geo and ip_geo.get("status") == "success":
|
||||
lines.extend([
|
||||
f"## IP Geolocation",
|
||||
f"- **Country:** {ip_geo.get('country', 'unknown')} ({ip_geo.get('country_code', '')})",
|
||||
f"- **City/Region:** {ip_geo.get('city', '')}, {ip_geo.get('region', '')}",
|
||||
f"- **ISP:** {ip_geo.get('isp', 'unknown')}",
|
||||
f"- **Organization:** {ip_geo.get('org', 'unknown')}",
|
||||
f"- **ASN:** {ip_geo.get('as_number', '')} {ip_geo.get('as_name', '')}",
|
||||
"",
|
||||
])
|
||||
|
||||
# Recent signup patterns
|
||||
lines.extend([
|
||||
f"## Recent Signup Patterns (last 24 hours)",
|
||||
f"- **New accounts from same IP:** {recent_signups_ip}",
|
||||
f"- **New accounts from same fingerprint:** {recent_signups_fp}",
|
||||
"",
|
||||
])
|
||||
|
||||
# Fingerprint correlations
|
||||
if fingerprint_matches:
|
||||
lines.append("## Other Accounts with Same Device Fingerprint (RED FLAG)")
|
||||
for match in fingerprint_matches:
|
||||
lines.append(
|
||||
f"- {match.get('email', '?')} | "
|
||||
f"Jobs: {match.get('total_jobs_created', 0)} | "
|
||||
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
|
||||
f"Created: {match.get('created_at', '?')}"
|
||||
)
|
||||
lines.append("")
|
||||
else:
|
||||
lines.extend(["## Other Accounts with Same Device Fingerprint", "None found.", ""])
|
||||
|
||||
# IP correlations
|
||||
if ip_matches:
|
||||
lines.append("## Other Accounts from Same Signup IP")
|
||||
for match in ip_matches:
|
||||
lines.append(
|
||||
f"- {match.get('email', '?')} | "
|
||||
f"Fingerprint: {match.get('device_fingerprint', 'unknown')} | "
|
||||
f"Jobs: {match.get('total_jobs_created', 0)} | "
|
||||
f"Spent: ${match.get('total_spent', 0) / 100:.2f} | "
|
||||
f"Created: {match.get('created_at', '?')}"
|
||||
)
|
||||
lines.append("")
|
||||
else:
|
||||
lines.extend(["## Other Accounts from Same Signup IP", "None found.", ""])
|
||||
|
||||
# Feedback content (if applicable)
|
||||
if feedback_content:
|
||||
lines.extend([
|
||||
"## Feedback Submitted",
|
||||
f"- **Overall rating:** {feedback_content.get('overall_rating', '?')}/5",
|
||||
f"- **What went well:** {feedback_content.get('what_went_well', 'N/A')[:200]}",
|
||||
f"- **What could improve:** {feedback_content.get('what_could_improve', 'N/A')[:200]}",
|
||||
f"- **Additional comments:** {feedback_content.get('additional_comments', 'N/A')[:200]}",
|
||||
"",
|
||||
])
|
||||
|
||||
lines.append("Based on these signals, should we GRANT or DENY free credits to this user?")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _parse_gemini_response(response_text: str) -> CreditEvaluation:
|
||||
"""Parse Gemini's JSON response into a CreditEvaluation."""
|
||||
try:
|
||||
# Strip markdown code fences if present
|
||||
text = response_text.strip()
|
||||
if text.startswith("```"):
|
||||
text = text.split("\n", 1)[1] if "\n" in text else text[3:]
|
||||
if text.endswith("```"):
|
||||
text = text.rsplit("```", 1)[0]
|
||||
text = text.strip()
|
||||
if text.startswith("json"):
|
||||
text = text[4:].strip()
|
||||
|
||||
data = json.loads(text)
|
||||
decision = data.get("decision", "").lower()
|
||||
if decision not in ("grant", "deny"):
|
||||
decision = "pending_review" # fail-closed: unknown decision → manual review
|
||||
|
||||
return CreditEvaluation(
|
||||
decision=decision,
|
||||
reasoning=str(data.get("reasoning", "No reasoning provided")),
|
||||
confidence=float(data.get("confidence", 0.5)),
|
||||
)
|
||||
except (json.JSONDecodeError, ValueError, TypeError) as e:
|
||||
logger.warning(f"Failed to parse Gemini response: {e}. Response: {response_text[:200]}")
|
||||
return CreditEvaluation(
|
||||
decision="pending_review",
|
||||
reasoning="Failed to parse AI response — pending manual review",
|
||||
confidence=0.0,
|
||||
error=f"Parse error: {e}",
|
||||
)
|
||||
|
||||
|
||||
class CreditEvaluationService:
|
||||
"""Evaluates credit grant requests using Gemini AI."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
|
||||
def _collect_signals(self, email: str) -> dict:
|
||||
"""Collect all abuse signals for evaluation."""
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.services.ip_geolocation_service import get_ip_geolocation_service
|
||||
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(email)
|
||||
|
||||
if not user:
|
||||
return {
|
||||
"user_data": {},
|
||||
"fingerprint_matches": [],
|
||||
"ip_matches": [],
|
||||
"ip_geo": None,
|
||||
"recent_signups_ip": 0,
|
||||
"recent_signups_fp": 0,
|
||||
"user_agent": None,
|
||||
}
|
||||
|
||||
user_data = {
|
||||
"email": user.email,
|
||||
"device_fingerprint": user.device_fingerprint,
|
||||
"signup_ip": user.signup_ip,
|
||||
"credits": user.credits,
|
||||
"total_jobs_created": user.total_jobs_created,
|
||||
"total_jobs_completed": user.total_jobs_completed,
|
||||
"total_spent": user.total_spent,
|
||||
"created_at": str(user.created_at),
|
||||
}
|
||||
|
||||
# Find other accounts with same fingerprint
|
||||
fingerprint_matches = []
|
||||
if user.device_fingerprint:
|
||||
fp_users = user_service.find_users_by_fingerprint(user.device_fingerprint)
|
||||
fingerprint_matches = [
|
||||
{
|
||||
"email": u.email,
|
||||
"total_jobs_created": u.total_jobs_created,
|
||||
"total_spent": u.total_spent,
|
||||
"created_at": str(u.created_at),
|
||||
}
|
||||
for u in fp_users if u.email != email
|
||||
]
|
||||
|
||||
# Find other accounts from same IP
|
||||
ip_matches = []
|
||||
if user.signup_ip:
|
||||
ip_users = user_service.find_users_by_signup_ip(user.signup_ip)
|
||||
ip_matches = [
|
||||
{
|
||||
"email": u.email,
|
||||
"device_fingerprint": u.device_fingerprint,
|
||||
"total_jobs_created": u.total_jobs_created,
|
||||
"total_spent": u.total_spent,
|
||||
"created_at": str(u.created_at),
|
||||
}
|
||||
for u in ip_users if u.email != email
|
||||
]
|
||||
|
||||
# IP geolocation
|
||||
ip_geo = None
|
||||
if user.signup_ip:
|
||||
try:
|
||||
geo_service = get_ip_geolocation_service()
|
||||
ip_geo = geo_service.lookup_ip(user.signup_ip)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Recent signup counts
|
||||
recent_signups_ip = 0
|
||||
recent_signups_fp = 0
|
||||
if user.signup_ip:
|
||||
recent_signups_ip = user_service.count_recent_signups_from_ip(user.signup_ip)
|
||||
if user.device_fingerprint:
|
||||
recent_signups_fp = user_service.count_recent_signups_from_fingerprint(user.device_fingerprint)
|
||||
|
||||
# User agent from latest session
|
||||
user_agent = None
|
||||
try:
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
from google.cloud import firestore as fs
|
||||
sessions = (
|
||||
user_service.db.collection("sessions")
|
||||
.where(filter=FieldFilter("user_email", "==", email))
|
||||
.order_by("created_at", direction=fs.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.stream()
|
||||
)
|
||||
for session_doc in sessions:
|
||||
user_agent = session_doc.to_dict().get("user_agent")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"user_data": user_data,
|
||||
"fingerprint_matches": fingerprint_matches,
|
||||
"ip_matches": ip_matches,
|
||||
"ip_geo": ip_geo,
|
||||
"recent_signups_ip": recent_signups_ip,
|
||||
"recent_signups_fp": recent_signups_fp,
|
||||
"user_agent": user_agent,
|
||||
}
|
||||
|
||||
def _call_gemini(self, prompt: str) -> str:
|
||||
"""Call Gemini and return the response text."""
|
||||
from google import genai
|
||||
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=self.settings.google_cloud_project,
|
||||
location="global",
|
||||
)
|
||||
response = client.models.generate_content(
|
||||
model=self.settings.credit_eval_model,
|
||||
contents=prompt,
|
||||
config=genai.types.GenerateContentConfig(
|
||||
system_instruction=SYSTEM_PROMPT,
|
||||
),
|
||||
)
|
||||
return response.text
|
||||
|
||||
def _log_evaluation(
|
||||
self,
|
||||
email: str,
|
||||
grant_type: str,
|
||||
evaluation: CreditEvaluation,
|
||||
signals: dict,
|
||||
):
|
||||
"""Log the evaluation decision to Firestore for audit trail."""
|
||||
try:
|
||||
from google.cloud import firestore as fs
|
||||
db = fs.Client(project=self.settings.google_cloud_project)
|
||||
doc_id = str(uuid.uuid4())
|
||||
db.collection(CREDIT_EVALUATIONS_COLLECTION).document(doc_id).set({
|
||||
"id": doc_id,
|
||||
"email": email,
|
||||
"grant_type": grant_type,
|
||||
"decision": evaluation.decision,
|
||||
"reasoning": evaluation.reasoning,
|
||||
"confidence": evaluation.confidence,
|
||||
"error": evaluation.error,
|
||||
"model": self.settings.credit_eval_model,
|
||||
"signals_snapshot": {
|
||||
"fingerprint_match_count": len(signals.get("fingerprint_matches", [])),
|
||||
"ip_match_count": len(signals.get("ip_matches", [])),
|
||||
"recent_signups_ip": signals.get("recent_signups_ip", 0),
|
||||
"recent_signups_fp": signals.get("recent_signups_fp", 0),
|
||||
"user_ip": signals.get("user_data", {}).get("signup_ip"),
|
||||
"user_fingerprint": signals.get("user_data", {}).get("device_fingerprint"),
|
||||
},
|
||||
"evaluated_at": datetime.now(timezone.utc).isoformat(),
|
||||
})
|
||||
except Exception:
|
||||
logger.exception(f"Failed to log credit evaluation for {email}")
|
||||
|
||||
def evaluate(
|
||||
self,
|
||||
email: str,
|
||||
grant_type: str,
|
||||
feedback_content: Optional[dict] = None,
|
||||
) -> CreditEvaluation:
|
||||
"""
|
||||
Evaluate whether to grant free credits.
|
||||
|
||||
Args:
|
||||
email: User's email
|
||||
grant_type: "welcome" or "feedback"
|
||||
feedback_content: Feedback form data (for feedback grants only)
|
||||
|
||||
Returns:
|
||||
CreditEvaluation with decision, reasoning, confidence.
|
||||
Fail-closed: returns "pending_review" on any error (admin notified).
|
||||
"""
|
||||
if not self.settings.credit_eval_enabled:
|
||||
return CreditEvaluation(
|
||||
decision="grant",
|
||||
reasoning="Credit evaluation disabled — auto-granting",
|
||||
confidence=1.0,
|
||||
)
|
||||
|
||||
try:
|
||||
# Collect signals
|
||||
signals = self._collect_signals(email)
|
||||
|
||||
# Quick-grant: if no correlations found at all, skip AI call
|
||||
if (
|
||||
not signals["fingerprint_matches"]
|
||||
and not signals["ip_matches"]
|
||||
and signals["recent_signups_ip"] <= 1
|
||||
and signals["recent_signups_fp"] <= 1
|
||||
):
|
||||
evaluation = CreditEvaluation(
|
||||
decision="grant",
|
||||
reasoning="No suspicious correlations found — clean user",
|
||||
confidence=1.0,
|
||||
signals=signals,
|
||||
)
|
||||
self._log_evaluation(email, grant_type, evaluation, signals)
|
||||
return evaluation
|
||||
|
||||
# Build prompt and call Gemini
|
||||
prompt = _build_user_prompt(
|
||||
email=email,
|
||||
grant_type=grant_type,
|
||||
user_data=signals["user_data"],
|
||||
fingerprint_matches=signals["fingerprint_matches"],
|
||||
ip_matches=signals["ip_matches"],
|
||||
ip_geo=signals["ip_geo"],
|
||||
recent_signups_ip=signals["recent_signups_ip"],
|
||||
recent_signups_fp=signals["recent_signups_fp"],
|
||||
user_agent=signals["user_agent"],
|
||||
feedback_content=feedback_content,
|
||||
)
|
||||
|
||||
response_text = self._call_gemini(prompt)
|
||||
evaluation = _parse_gemini_response(response_text)
|
||||
evaluation.signals = signals
|
||||
|
||||
self._log_evaluation(email, grant_type, evaluation, signals)
|
||||
logger.info(
|
||||
f"Credit evaluation for {email} ({grant_type}): "
|
||||
f"{evaluation.decision} (confidence={evaluation.confidence:.2f})"
|
||||
)
|
||||
return evaluation
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Credit evaluation failed for {email} — pending manual review (fail-closed)")
|
||||
# Include whatever signals were collected before the failure
|
||||
collected_signals = locals().get("signals")
|
||||
evaluation = CreditEvaluation(
|
||||
decision="pending_review",
|
||||
reasoning="Evaluation failed — pending manual review",
|
||||
confidence=0.0,
|
||||
error=str(e),
|
||||
signals=collected_signals,
|
||||
)
|
||||
try:
|
||||
self._log_evaluation(email, grant_type, evaluation, collected_signals or {})
|
||||
except Exception:
|
||||
pass
|
||||
return evaluation
|
||||
|
||||
|
||||
_service_instance: Optional[CreditEvaluationService] = None
|
||||
|
||||
|
||||
def get_credit_evaluation_service() -> CreditEvaluationService:
|
||||
"""Get or create the singleton credit evaluation service."""
|
||||
global _service_instance
|
||||
if _service_instance is None:
|
||||
_service_instance = CreditEvaluationService()
|
||||
return _service_instance
|
||||
@@ -1,129 +0,0 @@
|
||||
"""
|
||||
Disposable Domain Sync Service.
|
||||
|
||||
Fetches and syncs disposable email domain blocklists from external sources.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Set
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.services.email_validation_service import (
|
||||
BLOCKLISTS_COLLECTION,
|
||||
BLOCKLIST_CONFIG_DOC,
|
||||
DEFAULT_DISPOSABLE_DOMAINS,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
EXTERNAL_BLOCKLIST_URL = (
|
||||
"https://raw.githubusercontent.com/disposable-email-domains/"
|
||||
"disposable-email-domains/refs/heads/main/disposable_email_blocklist.conf"
|
||||
)
|
||||
FETCH_TIMEOUT_SECONDS = 30
|
||||
MAX_RESPONSE_BYTES = 2 * 1024 * 1024 # 2 MB
|
||||
MAX_DOMAIN_COUNT = 50_000
|
||||
|
||||
|
||||
def parse_blocklist_text(text: str) -> Set[str]:
|
||||
"""Parse newline-delimited blocklist text into a set of lowercase domains."""
|
||||
domains = set()
|
||||
for line in text.splitlines():
|
||||
domain = line.strip().lower()
|
||||
if domain:
|
||||
domains.add(domain)
|
||||
return domains
|
||||
|
||||
|
||||
async def fetch_external_blocklist() -> Set[str]:
|
||||
"""Fetch the external disposable domain blocklist from GitHub.
|
||||
|
||||
Checks Content-Length header and response body size against safety limits.
|
||||
"""
|
||||
async with httpx.AsyncClient(timeout=FETCH_TIMEOUT_SECONDS) as client:
|
||||
response = await client.get(EXTERNAL_BLOCKLIST_URL)
|
||||
response.raise_for_status()
|
||||
|
||||
content_length = response.headers.get("content-length")
|
||||
if content_length and int(content_length) > MAX_RESPONSE_BYTES:
|
||||
raise ValueError(
|
||||
f"Response size {content_length} exceeds maximum {MAX_RESPONSE_BYTES}"
|
||||
)
|
||||
|
||||
text = response.text
|
||||
if len(text.encode()) > MAX_RESPONSE_BYTES:
|
||||
raise ValueError(
|
||||
f"Response body size exceeds maximum {MAX_RESPONSE_BYTES}"
|
||||
)
|
||||
|
||||
domains = parse_blocklist_text(text)
|
||||
|
||||
if len(domains) > MAX_DOMAIN_COUNT:
|
||||
raise ValueError(
|
||||
f"Domain count {len(domains)} exceeds maximum {MAX_DOMAIN_COUNT}"
|
||||
)
|
||||
|
||||
logger.info(f"Fetched {len(domains)} domains from external blocklist")
|
||||
return domains
|
||||
|
||||
|
||||
def sync_disposable_domains(db, external_domains: Set[str]) -> dict:
|
||||
"""
|
||||
Sync external domains into Firestore.
|
||||
|
||||
Handles first-run migration from old disposable_domains field,
|
||||
and cleans up redundant manual domains on subsequent runs.
|
||||
|
||||
Note: This is a synchronous function (Firestore SDK is sync).
|
||||
Call from async code with `await asyncio.to_thread(sync_disposable_domains, db, domains)`.
|
||||
|
||||
Returns a summary dict with counts.
|
||||
"""
|
||||
doc_ref = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
data = doc.to_dict() if doc.exists else {}
|
||||
result = {"external_count": len(external_domains), "added": 0, "removed": 0, "migrated_to_manual": []}
|
||||
|
||||
is_migration = "external_domains" not in data
|
||||
|
||||
if is_migration:
|
||||
old_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
|
||||
manual_only = old_domains - external_domains
|
||||
result["migrated_to_manual"] = sorted(manual_only)
|
||||
|
||||
new_data = {
|
||||
**data, # preserve all existing fields
|
||||
"external_domains": sorted(external_domains),
|
||||
"manual_domains": sorted(manual_only),
|
||||
"allowlisted_domains": data.get("allowlisted_domains", []),
|
||||
"last_sync_at": datetime.now(timezone.utc),
|
||||
"last_sync_count": len(external_domains),
|
||||
}
|
||||
new_data.pop("disposable_domains", None) # remove legacy field
|
||||
doc_ref.set(new_data)
|
||||
else:
|
||||
old_external = set(data.get("external_domains", []))
|
||||
manual_domains = set(data.get("manual_domains", []))
|
||||
|
||||
result["added"] = len(external_domains - old_external)
|
||||
result["removed"] = len(old_external - external_domains)
|
||||
|
||||
cleaned_manual = manual_domains - external_domains
|
||||
|
||||
new_data = {
|
||||
**data,
|
||||
"external_domains": sorted(external_domains),
|
||||
"manual_domains": sorted(cleaned_manual),
|
||||
"last_sync_at": datetime.now(timezone.utc),
|
||||
"last_sync_count": len(external_domains),
|
||||
}
|
||||
doc_ref.set(new_data)
|
||||
|
||||
logger.info(
|
||||
f"Sync complete: {result['external_count']} external domains, "
|
||||
f"{result['added']} added, {result['removed']} removed"
|
||||
)
|
||||
return result
|
||||
@@ -109,46 +109,30 @@ class DropboxService:
|
||||
|
||||
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
|
||||
"""
|
||||
Calculate next brand code from existing folders, filling gaps in the sequence.
|
||||
|
||||
NOTE: This method is only used for one-time initialization of the Firestore
|
||||
counter (see brand_code_service.py). All runtime brand code allocation now
|
||||
goes through BrandCodeService.allocate_brand_code() for atomic safety.
|
||||
Calculate next brand code from existing folders.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the first available code, filling any gaps left by deleted outputs.
|
||||
returns the next sequential code.
|
||||
|
||||
Args:
|
||||
path: Dropbox path containing organized folders
|
||||
brand_prefix: Brand prefix (e.g., "NOMAD")
|
||||
|
||||
Returns:
|
||||
Next brand code (e.g., "NOMAD-0003" if 0003 is the first gap)
|
||||
Next brand code (e.g., "NOMAD-1164")
|
||||
"""
|
||||
folders = self.list_folders(path)
|
||||
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
|
||||
|
||||
existing_nums: set[int] = set()
|
||||
max_num = 0
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
existing_nums.add(int(match.group(1)))
|
||||
num = int(match.group(1))
|
||||
max_num = max(max_num, num)
|
||||
|
||||
max_existing = max(existing_nums) if existing_nums else 0
|
||||
|
||||
# Only fill gaps for numbers >= 1001 (preserve legacy gaps below 1000)
|
||||
if max_existing >= 1000:
|
||||
next_num = 1001
|
||||
while next_num in existing_nums:
|
||||
next_num += 1
|
||||
else:
|
||||
next_num = max_existing + 1
|
||||
|
||||
next_code = f"{brand_prefix}-{next_num:04d}"
|
||||
if max_existing >= 1000 and next_num <= max_existing:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (filling gap, max existing: {max_existing})")
|
||||
else:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_existing})")
|
||||
next_code = f"{brand_prefix}-{max_num + 1:04d}"
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_num})")
|
||||
return next_code
|
||||
|
||||
def upload_file(self, local_path: str, remote_path: str) -> None:
|
||||
|
||||
+627
-1570
File diff suppressed because it is too large.
Load diff
@@ -1,856 +0,0 @@
|
||||
"""
|
||||
Email Validation Service.
|
||||
|
||||
Provides email validation, normalization, and blocklist checking
|
||||
to prevent abuse during enrollment and other flows.
|
||||
|
||||
Uses a tiered disposable domain detection strategy:
|
||||
1. Static blocklist (instant, from Firestore)
|
||||
2. Verified-clean domain cache (skip API calls for 7 days)
|
||||
3. DeBounce API (free, <50ms)
|
||||
4. verifymail.io API (paid, only if DeBounce says clean)
|
||||
|
||||
Auto-learns: flagged domains are persisted to manual_domains,
|
||||
clean domains are cached for 7 days in verified_clean_domains.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, Tuple, Set
|
||||
|
||||
import httpx
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Module-level HTTP client for connection pooling (external API checks)
|
||||
_http_client: Optional[httpx.Client] = None
|
||||
_HTTP_TIMEOUT = 2.0 # seconds per external API call
|
||||
|
||||
|
||||
def _get_http_client() -> httpx.Client:
|
||||
"""Get or create the module-level HTTP client."""
|
||||
global _http_client
|
||||
if _http_client is None or _http_client.is_closed:
|
||||
_http_client = httpx.Client(timeout=_HTTP_TIMEOUT)
|
||||
return _http_client
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask an email address for privacy-safe logging."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 2:
|
||||
masked_local = "*" * len(local)
|
||||
else:
|
||||
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
|
||||
return f"{masked_local}@{domain}"
|
||||
|
||||
|
||||
# Firestore collection for blocklist data
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
BLOCKLIST_CONFIG_DOC = "config"
|
||||
|
||||
# Legacy default set — kept as empty set for import compatibility.
|
||||
# The external sync (disposable_domain_sync_service) replaces this with ~4,800
|
||||
# domains from the community-curated disposable-email-domains repo.
|
||||
# The migration path in sync_disposable_domains() references this set to ensure
|
||||
# no domains are lost during the transition.
|
||||
DEFAULT_DISPOSABLE_DOMAINS: Set[str] = set()
|
||||
|
||||
# Gmail-like domains that support alias normalization
|
||||
GMAIL_LIKE_DOMAINS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
}
|
||||
|
||||
# Well-known email providers that are never disposable.
|
||||
# These skip all external API checks for zero-latency validation.
|
||||
WELL_KNOWN_PROVIDERS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
"yahoo.com",
|
||||
"yahoo.co.uk",
|
||||
"yahoo.co.jp",
|
||||
"yahoo.fr",
|
||||
"hotmail.com",
|
||||
"hotmail.co.uk",
|
||||
"outlook.com",
|
||||
"outlook.fr",
|
||||
"live.com",
|
||||
"live.co.uk",
|
||||
"msn.com",
|
||||
"icloud.com",
|
||||
"me.com",
|
||||
"mac.com",
|
||||
"aol.com",
|
||||
"protonmail.com",
|
||||
"proton.me",
|
||||
"zoho.com",
|
||||
"yandex.com",
|
||||
"yandex.ru",
|
||||
"mail.com",
|
||||
"gmx.com",
|
||||
"gmx.de",
|
||||
"fastmail.com",
|
||||
"hey.com",
|
||||
"tutanota.com",
|
||||
"tuta.io",
|
||||
}
|
||||
|
||||
# How long a verified-clean domain result is trusted before re-checking
|
||||
VERIFIED_CLEAN_TTL = timedelta(days=7)
|
||||
|
||||
# External API URLs
|
||||
DEBOUNCE_API_URL = "https://disposable.debounce.io/"
|
||||
VERIFYMAIL_API_URL = "https://verifymail.io/api"
|
||||
|
||||
|
||||
class EmailValidationService:
|
||||
"""
|
||||
Service for email validation and abuse prevention.
|
||||
|
||||
Features:
|
||||
- Email normalization (Gmail alias stripping)
|
||||
- Disposable domain detection
|
||||
- Blocklist checking (emails, IPs)
|
||||
- IP-based enrollment rate limiting
|
||||
"""
|
||||
|
||||
_instance: Optional["EmailValidationService"] = None
|
||||
_blocklist_cache: Optional[dict] = None
|
||||
_blocklist_cache_time: Optional[datetime] = None
|
||||
_cache_ttl_seconds: int = 300 # 5 minutes
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize the email validation service."""
|
||||
if db is None:
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
db = get_firestore_client()
|
||||
self.db = db
|
||||
|
||||
@classmethod
|
||||
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
|
||||
"""Get singleton instance of the service."""
|
||||
if cls._instance is None:
|
||||
cls._instance = cls(db)
|
||||
return cls._instance
|
||||
|
||||
def normalize_email(self, email: str) -> str:
|
||||
"""
|
||||
Normalize an email address.
|
||||
|
||||
For Gmail-like domains:
|
||||
- Removes dots from local part
|
||||
- Removes everything after + in local part
|
||||
- Converts to lowercase
|
||||
|
||||
For other domains:
|
||||
- Only converts to lowercase
|
||||
|
||||
Args:
|
||||
email: Email address to normalize
|
||||
|
||||
Returns:
|
||||
Normalized email address
|
||||
|
||||
Examples:
|
||||
j.o.h.n+spam@gmail.com -> john@gmail.com
|
||||
John.Doe@example.com -> john.doe@example.com
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return email.lower().strip() if email else ""
|
||||
|
||||
email = email.lower().strip()
|
||||
local, domain = email.rsplit("@", 1)
|
||||
|
||||
if domain in GMAIL_LIKE_DOMAINS:
|
||||
# Remove dots from local part
|
||||
local = local.replace(".", "")
|
||||
# Remove everything after +
|
||||
if "+" in local:
|
||||
local = local.split("+")[0]
|
||||
|
||||
return f"{local}@{domain}"
|
||||
|
||||
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
|
||||
"""
|
||||
Get blocklist configuration from Firestore.
|
||||
|
||||
Uses caching to reduce Firestore reads.
|
||||
|
||||
Returns:
|
||||
Dict with disposable_domains, blocked_emails, blocked_ips sets
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache validity
|
||||
if (
|
||||
not force_refresh
|
||||
and self._blocklist_cache is not None
|
||||
and self._blocklist_cache_time is not None
|
||||
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
|
||||
):
|
||||
return self._blocklist_cache
|
||||
|
||||
# Fetch from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
if "external_domains" in data:
|
||||
# New data model: effective = (external + manual) - allowlisted
|
||||
external = set(data.get("external_domains", []))
|
||||
manual = set(data.get("manual_domains", []))
|
||||
allowlisted = set(data.get("allowlisted_domains", []))
|
||||
effective_domains = (external | manual) - allowlisted
|
||||
else:
|
||||
# Old data model (pre-migration): use disposable_domains + defaults
|
||||
allowlisted = set()
|
||||
effective_domains = set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS
|
||||
config = {
|
||||
"disposable_domains": effective_domains,
|
||||
"allowlisted_domains": allowlisted,
|
||||
"blocked_emails": set(data.get("blocked_emails", [])),
|
||||
"blocked_ips": set(data.get("blocked_ips", [])),
|
||||
"verified_clean_domains": data.get("verified_clean_domains", {}),
|
||||
}
|
||||
else:
|
||||
config = {
|
||||
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
|
||||
"allowlisted_domains": set(),
|
||||
"blocked_emails": set(),
|
||||
"blocked_ips": set(),
|
||||
"verified_clean_domains": {},
|
||||
}
|
||||
|
||||
# Update cache
|
||||
EmailValidationService._blocklist_cache = config
|
||||
EmailValidationService._blocklist_cache_time = now
|
||||
|
||||
return config
|
||||
|
||||
def is_disposable_domain(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email uses a disposable domain.
|
||||
|
||||
Uses a tiered checking strategy:
|
||||
0. Well-known providers (gmail, yahoo, etc.) → instant False
|
||||
1. Static blocklist from Firestore → instant
|
||||
2. Verified-clean cache (7-day TTL) → skip API calls
|
||||
3. DeBounce API (free) → fast external check
|
||||
4. verifymail.io API (paid) → only if DeBounce says clean
|
||||
|
||||
Flagged domains are auto-learned to manual_domains.
|
||||
Clean domains are cached for 7 days.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the domain is known to be disposable
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False
|
||||
|
||||
domain = email.lower().split("@")[-1]
|
||||
|
||||
# Tier 0: Well-known providers — skip everything
|
||||
if domain in WELL_KNOWN_PROVIDERS:
|
||||
return False
|
||||
|
||||
# Tier 1: Static blocklist (instant, always available)
|
||||
config = self.get_blocklist_config()
|
||||
if domain in config["disposable_domains"]:
|
||||
return True
|
||||
|
||||
# Tier 1.5: Explicitly allowlisted domains skip external API checks.
|
||||
# Without this, external APIs can override admin allowlist decisions
|
||||
# (e.g. inbox.testmail.app is a legitimate testing domain we allow).
|
||||
if domain in config.get("allowlisted_domains", set()):
|
||||
return False
|
||||
|
||||
# Tier 2: Recently verified clean? Skip API calls
|
||||
if self._is_domain_verified_clean(domain, config):
|
||||
return False
|
||||
|
||||
# Tier 3: DeBounce API (free, fast)
|
||||
debounce_result = self._check_debounce(email)
|
||||
if debounce_result is True:
|
||||
self._auto_learn_domain(domain)
|
||||
return True
|
||||
|
||||
# Tier 4: verifymail.io (paid, only if DeBounce gave definitive clean)
|
||||
if debounce_result is False:
|
||||
verifymail_result = self._check_verifymail(email)
|
||||
if verifymail_result is True:
|
||||
self._auto_learn_domain(domain)
|
||||
return True
|
||||
|
||||
# Both APIs say clean (or errored) — cache clean result
|
||||
if debounce_result is False:
|
||||
self._mark_domain_clean(domain)
|
||||
|
||||
return False
|
||||
|
||||
def _check_debounce(self, email: str) -> Optional[bool]:
|
||||
"""
|
||||
Check email against DeBounce disposable email API.
|
||||
|
||||
Returns:
|
||||
True if disposable, False if clean, None if error/timeout.
|
||||
"""
|
||||
try:
|
||||
client = _get_http_client()
|
||||
response = client.get(DEBOUNCE_API_URL, params={"email": email})
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
result = str(data.get("disposable", "")).lower() == "true"
|
||||
logger.info(
|
||||
f"DeBounce check for {_mask_email(email)}: "
|
||||
f"disposable={result}"
|
||||
)
|
||||
return result
|
||||
except httpx.TimeoutException:
|
||||
logger.warning(f"DeBounce API timeout for {_mask_email(email)}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"DeBounce API error for {_mask_email(email)}: {e}")
|
||||
return None
|
||||
|
||||
def _check_verifymail(self, email: str) -> Optional[bool]:
|
||||
"""
|
||||
Check email against verifymail.io API.
|
||||
|
||||
Only called when DeBounce says clean (to minimize paid API usage).
|
||||
|
||||
Returns:
|
||||
True if disposable/blocked, False if clean, None if error/timeout
|
||||
or API key not configured.
|
||||
"""
|
||||
api_key = settings.get_secret("verifymail-api-key")
|
||||
if not api_key:
|
||||
logger.debug("verifymail.io API key not configured, skipping")
|
||||
return None
|
||||
|
||||
try:
|
||||
client = _get_http_client()
|
||||
response = client.get(
|
||||
f"{VERIFYMAIL_API_URL}/{email}",
|
||||
params={"key": api_key},
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
# verifymail.io returns both "disposable" and "block" flags
|
||||
is_disposable = data.get("disposable") is True
|
||||
is_blocked = data.get("block") is True
|
||||
result = is_disposable or is_blocked
|
||||
logger.info(
|
||||
f"verifymail.io check for {_mask_email(email)}: "
|
||||
f"disposable={is_disposable}, block={is_blocked}"
|
||||
)
|
||||
return result
|
||||
except httpx.TimeoutException:
|
||||
logger.warning(f"verifymail.io API timeout for {_mask_email(email)}")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.warning(f"verifymail.io API error for {_mask_email(email)}: {e}")
|
||||
return None
|
||||
|
||||
def _is_domain_verified_clean(self, domain: str, config: dict) -> bool:
|
||||
"""Check if a domain was verified clean within the TTL period."""
|
||||
verified = config.get("verified_clean_domains", {})
|
||||
entry = verified.get(domain)
|
||||
if not entry:
|
||||
return False
|
||||
|
||||
checked_at = entry.get("checked_at")
|
||||
if not checked_at:
|
||||
return False
|
||||
|
||||
# Handle both datetime objects and ISO strings
|
||||
if isinstance(checked_at, str):
|
||||
try:
|
||||
checked_at = datetime.fromisoformat(checked_at)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
# Make timezone-aware if naive
|
||||
if checked_at.tzinfo is None:
|
||||
checked_at = checked_at.replace(tzinfo=timezone.utc)
|
||||
|
||||
return (datetime.now(timezone.utc) - checked_at) < VERIFIED_CLEAN_TTL
|
||||
|
||||
def _auto_learn_domain(self, domain: str) -> None:
|
||||
"""
|
||||
Add a domain flagged by external API to manual_domains in Firestore.
|
||||
|
||||
This persists the result so future checks don't need the API call.
|
||||
Fire-and-forget: errors are logged but don't affect the caller.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
|
||||
BLOCKLIST_CONFIG_DOC
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
manual = set(data.get("manual_domains", []))
|
||||
else:
|
||||
manual = set()
|
||||
|
||||
if domain not in manual:
|
||||
manual.add(domain)
|
||||
doc_ref.set(
|
||||
{
|
||||
"manual_domains": list(manual),
|
||||
"updated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"updated_by": "auto-learn",
|
||||
},
|
||||
merge=True,
|
||||
)
|
||||
logger.info(f"Auto-learned disposable domain: {domain}")
|
||||
|
||||
# Invalidate cache so the domain is blocked immediately
|
||||
EmailValidationService._blocklist_cache = None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to auto-learn domain {domain}: {e}")
|
||||
|
||||
def _mark_domain_clean(self, domain: str) -> None:
|
||||
"""
|
||||
Record that a domain was verified clean by external APIs.
|
||||
|
||||
Stored in verified_clean_domains dict keyed by domain.
|
||||
Future checks within VERIFIED_CLEAN_TTL will skip API calls.
|
||||
|
||||
Note: Uses read-modify-write instead of dot notation because
|
||||
domain names contain dots which Firestore interprets as nested paths.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(
|
||||
BLOCKLIST_CONFIG_DOC
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
verified = doc.to_dict().get("verified_clean_domains", {}) if doc.exists else {}
|
||||
verified[domain] = {
|
||||
"checked_at": datetime.now(timezone.utc).isoformat(),
|
||||
}
|
||||
doc_ref.set({"verified_clean_domains": verified}, merge=True)
|
||||
logger.debug(f"Marked domain as verified clean: {domain}")
|
||||
|
||||
# Invalidate cache so new data is picked up
|
||||
EmailValidationService._blocklist_cache = None
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to mark domain clean {domain}: {e}")
|
||||
|
||||
def is_email_blocked(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is explicitly blocked.
|
||||
|
||||
Checks both the raw email and normalized version.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the email is blocked
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
email_lower = email.lower().strip()
|
||||
email_normalized = self.normalize_email(email)
|
||||
|
||||
return (
|
||||
email_lower in config["blocked_emails"]
|
||||
or email_normalized in config["blocked_emails"]
|
||||
)
|
||||
|
||||
def is_ip_blocked(self, ip_address: str) -> bool:
|
||||
"""
|
||||
Check if an IP address is blocked.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to check
|
||||
|
||||
Returns:
|
||||
True if the IP is blocked
|
||||
"""
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
return ip_address in config["blocked_ips"]
|
||||
|
||||
def hash_ip(self, ip_address: str) -> str:
|
||||
"""
|
||||
Hash an IP address for privacy-preserving storage.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to hash
|
||||
|
||||
Returns:
|
||||
SHA-256 hash of the IP
|
||||
"""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management (Admin)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("manual_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["manual_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
|
||||
manual = set(data.get("manual_domains", []))
|
||||
external = set(data.get("external_domains", []))
|
||||
|
||||
if domain in manual:
|
||||
result["found"] = True
|
||||
manual.discard(domain)
|
||||
data["manual_domains"] = list(manual)
|
||||
elif domain in external:
|
||||
result["found"] = True
|
||||
allowlisted = set(data.get("allowlisted_domains", []))
|
||||
allowlisted.add(domain)
|
||||
data["allowlisted_domains"] = list(allowlisted)
|
||||
else:
|
||||
return
|
||||
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Add an email to the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
if not email:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
else:
|
||||
data = {}
|
||||
emails = set()
|
||||
|
||||
emails.add(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
|
||||
if email not in emails:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
emails.discard(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
else:
|
||||
data = {}
|
||||
ips = set()
|
||||
|
||||
ips.add(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
ips.discard(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the allowlisted domains list."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("allowlisted_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["allowlisted_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added allowlisted domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_allowlisted_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the allowlisted domains list."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("allowlisted_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["allowlisted_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed allowlisted domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def get_blocklist_raw_data(self) -> dict:
|
||||
"""Get raw blocklist data for admin display (not the effective set)."""
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return {
|
||||
"external_domains": [], "manual_domains": [], "allowlisted_domains": [],
|
||||
"blocked_emails": [], "blocked_ips": [],
|
||||
"last_sync_at": None, "last_sync_count": None,
|
||||
"updated_at": None, "updated_by": None,
|
||||
}
|
||||
data = doc.to_dict()
|
||||
return {
|
||||
"external_domains": sorted(data.get("external_domains", [])),
|
||||
"manual_domains": sorted(data.get("manual_domains", [])),
|
||||
"allowlisted_domains": sorted(data.get("allowlisted_domains", [])),
|
||||
"blocked_emails": sorted(data.get("blocked_emails", [])),
|
||||
"blocked_ips": sorted(data.get("blocked_ips", [])),
|
||||
"last_sync_at": data.get("last_sync_at"),
|
||||
"last_sync_count": data.get("last_sync_count"),
|
||||
"updated_at": data.get("updated_at"),
|
||||
"updated_by": data.get("updated_by"),
|
||||
}
|
||||
|
||||
def get_blocklist_stats(self) -> dict:
|
||||
"""Get statistics about current blocklists."""
|
||||
config = self.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Also get raw data for detailed counts
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
external_count = len(data.get("external_domains", []))
|
||||
manual_count = len(data.get("manual_domains", []))
|
||||
allowlisted_count = len(data.get("allowlisted_domains", []))
|
||||
else:
|
||||
external_count = 0
|
||||
manual_count = 0
|
||||
allowlisted_count = 0
|
||||
|
||||
return {
|
||||
"disposable_domains_count": len(config["disposable_domains"]),
|
||||
"blocked_emails_count": len(config["blocked_emails"]),
|
||||
"blocked_ips_count": len(config["blocked_ips"]),
|
||||
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
|
||||
"external_domains_count": external_count,
|
||||
"manual_domains_count": manual_count,
|
||||
"allowlisted_domains_count": allowlisted_count,
|
||||
}
|
||||
|
||||
|
||||
def get_email_validation_service(
|
||||
db: Optional[firestore.Client] = None,
|
||||
) -> EmailValidationService:
|
||||
"""Get the singleton EmailValidationService instance."""
|
||||
return EmailValidationService.get_instance(db)
|
||||
@@ -40,9 +40,6 @@ class EncodingInput:
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Instrumental selection (clean, with_backing, or custom)
|
||||
instrumental_selection: str = "clean"
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
@@ -200,7 +197,6 @@ class LocalEncodingBackend(EncodingBackend):
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
countdown_padding_seconds=input_config.options.get("countdown_padding_seconds"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
@@ -332,15 +328,8 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
"instrumental_selection": input_config.instrumental_selection,
|
||||
}
|
||||
|
||||
# Add countdown padding if present (for audio sync with countdown-padded vocals)
|
||||
countdown_padding = input_config.options.get("countdown_padding_seconds")
|
||||
if countdown_padding:
|
||||
encoding_config["countdown_padding_seconds"] = countdown_padding
|
||||
self.logger.info(f"GCE encoding with countdown_padding_seconds={countdown_padding}")
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
@@ -362,8 +351,8 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# Convert output_files from list of paths to dict if needed
|
||||
# GCE worker returns list like: ["path/output_4k_lossless.mp4", "path/output_720p.mp4"]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
@@ -371,20 +360,15 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
if "4k_lossless" in filename:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "4k_lossy" in filename:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
elif "720p" in filename:
|
||||
output_files["mp4_720p"] = path
|
||||
elif "with vocals" in filename_lower and filename.endswith(".mp4"):
|
||||
output_files["with_vocals_mp4"] = path
|
||||
elif filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
@@ -395,7 +379,6 @@ class GCEEncodingBackend(EncodingBackend):
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
with_vocals_mp4_path=output_files.get("with_vocals_mp4"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
|
||||
@@ -17,7 +17,6 @@ Usage:
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
@@ -26,24 +25,6 @@ from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts during deployments).
|
||||
#
|
||||
# During a CI deployment, the encoding worker is restarted via systemctl. The restart
|
||||
# involves downloading a new wheel from GCS, installing it, and starting uvicorn —
|
||||
# which can take 30-90 seconds. The retry window must exceed this to avoid failing
|
||||
# jobs that happen to hit the encoding stage during a deploy.
|
||||
#
|
||||
# With 7 retries and exponential backoff (5s, 10s, 15s, 15s, 15s, 15s, 15s),
|
||||
# total retry window is ~90s, sufficient to survive a full worker restart.
|
||||
MAX_RETRIES = 7
|
||||
INITIAL_BACKOFF_SECONDS = 5.0
|
||||
MAX_BACKOFF_SECONDS = 15.0
|
||||
|
||||
# Poll failure tolerance: number of consecutive poll failures allowed before giving up.
|
||||
# This prevents a single transient network blip during status polling from killing a
|
||||
# long-running encoding job. Similar pattern to flacfetch status polling (PR #446).
|
||||
MAX_CONSECUTIVE_POLL_FAILURES = 5
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
@@ -53,34 +34,6 @@ class EncodingService:
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
self._worker_manager = None
|
||||
self._cached_url = None
|
||||
self._url_cached_at = 0
|
||||
self._URL_CACHE_TTL = 30 # seconds
|
||||
|
||||
def set_worker_manager(self, manager):
|
||||
"""Set the worker manager for dynamic URL resolution from Firestore."""
|
||||
self._worker_manager = manager
|
||||
|
||||
def _get_worker_url(self) -> str:
|
||||
"""Get the current primary worker URL from Firestore (with TTL cache).
|
||||
|
||||
Falls back to static URL from config if worker_manager is not set.
|
||||
"""
|
||||
now = time.time()
|
||||
if self._cached_url and (now - self._url_cached_at) < self._URL_CACHE_TTL:
|
||||
return self._cached_url
|
||||
|
||||
if self._worker_manager:
|
||||
config = self._worker_manager.get_config()
|
||||
self._cached_url = config.primary_url
|
||||
self._url_cached_at = now
|
||||
return self._cached_url
|
||||
|
||||
# Fallback to static URL
|
||||
if not self._initialized:
|
||||
self._load_credentials()
|
||||
return self._url
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
@@ -110,110 +63,6 @@ class EncodingService:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
def _warmup_encoding_worker_fallback(self, job_id: str) -> None:
|
||||
"""Safety net: try to start the encoding worker VM on first connection failure.
|
||||
|
||||
If the worker_manager is available, calls ensure_primary_running() so the VM
|
||||
starts booting during the retry window. If worker_manager is not set (e.g. dev
|
||||
mode with static URL), this is a no-op.
|
||||
"""
|
||||
if not self._worker_manager:
|
||||
return
|
||||
try:
|
||||
result = self._worker_manager.ensure_primary_running()
|
||||
if result["started"]:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] Encoding worker unreachable — started VM {result['vm_name']} as fallback"
|
||||
)
|
||||
else:
|
||||
logger.info(
|
||||
f"[job:{job_id}] Encoding worker unreachable — VM {result['vm_name']} already running/starting"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"[job:{job_id}] Encoding worker warmup fallback failed (non-fatal): {e}")
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
headers: Dict[str, str],
|
||||
json_payload: Optional[Dict[str, Any]] = None,
|
||||
timeout: float = 30.0,
|
||||
job_id: str = "unknown",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make an HTTP request with retry logic for transient failures.
|
||||
|
||||
This handles connection errors that occur when the GCE worker is
|
||||
restarting (e.g., during deployments) by retrying with exponential backoff.
|
||||
|
||||
Args:
|
||||
method: HTTP method (GET, POST)
|
||||
url: Request URL
|
||||
headers: Request headers
|
||||
json_payload: JSON body for POST requests
|
||||
timeout: Request timeout in seconds
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- status (int): HTTP status code
|
||||
- json (Any): Parsed JSON response body (if status 200, else None)
|
||||
- text (str): Raw response text (if status != 200, else None)
|
||||
|
||||
Raises:
|
||||
aiohttp.ClientConnectorError: If all retries fail due to connection errors
|
||||
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
|
||||
asyncio.TimeoutError: If all retries fail due to timeout
|
||||
"""
|
||||
last_exception = None
|
||||
backoff = INITIAL_BACKOFF_SECONDS
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
if method.upper() == "POST":
|
||||
async with session.post(
|
||||
url, json=json_payload, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
# Return a copy of the response data since we exit the context
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
else: # GET
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt == 0:
|
||||
self._warmup_encoding_worker_fallback(job_id)
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
f"Retrying in {backoff:.1f}s..."
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
|
||||
)
|
||||
|
||||
raise last_exception
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
@@ -241,7 +90,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._get_worker_url()}/encode"
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
@@ -252,40 +101,17 @@ class EncodingService:
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(url, json=payload, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp.status} - {text}")
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
# Job already exists on worker — check if it already completed
|
||||
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
job_status = status.get("status", "unknown")
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] Job already complete on GCE worker, returning cached result")
|
||||
return {"status": "cached", "job_id": job_id, "output_files": status.get("output_files")}
|
||||
elif job_status in ("pending", "running"):
|
||||
logger.info(f"[job:{job_id}] Job still in progress on GCE worker")
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
else:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists with status: {job_status}")
|
||||
except RuntimeError as e:
|
||||
if "not found" in str(e).lower():
|
||||
# Job was in worker memory but got cleared (restart) — safe to raise original error
|
||||
raise RuntimeError(f"Encoding job {job_id} conflict: 409 but job not found on status check")
|
||||
raise
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
return await resp.json()
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
@@ -302,25 +128,20 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._get_worker_url()}/status/{job_id}"
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=headers,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=30) as resp:
|
||||
if resp.status == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp.status == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp.status != 200:
|
||||
text = await resp.text()
|
||||
raise RuntimeError(f"Failed to get job status: {resp.status} - {text}")
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
return await resp.json()
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
@@ -330,12 +151,7 @@ class EncodingService:
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion with tolerance for transient failures.
|
||||
|
||||
During a deployment, the encoding worker may restart while we're polling.
|
||||
Rather than failing the entire job on a single poll error, we tolerate up to
|
||||
MAX_CONSECUTIVE_POLL_FAILURES consecutive failures before giving up. This is
|
||||
the same pattern used in flacfetch status polling (PR #446).
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
@@ -348,40 +164,19 @@ class EncodingService:
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails or too many consecutive poll failures
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
consecutive_failures = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
# Reset failure counter on successful poll
|
||||
consecutive_failures = 0
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError,
|
||||
asyncio.TimeoutError, RuntimeError) as e:
|
||||
consecutive_failures += 1
|
||||
if consecutive_failures >= MAX_CONSECUTIVE_POLL_FAILURES:
|
||||
logger.error(
|
||||
f"[job:{job_id}] Status polling failed {consecutive_failures} consecutive times, giving up: {e}"
|
||||
)
|
||||
raise RuntimeError(
|
||||
f"Encoding job {job_id} lost contact with worker after "
|
||||
f"{consecutive_failures} consecutive poll failures: {e}"
|
||||
)
|
||||
logger.warning(
|
||||
f"[job:{job_id}] Status poll failed ({consecutive_failures}/{MAX_CONSECUTIVE_POLL_FAILURES}): {e}. "
|
||||
f"Worker may be restarting, will retry..."
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
continue
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
@@ -440,254 +235,7 @@ class EncodingService:
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
submit_result = await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# If cached, return immediately — encoding already done
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Encoding already cached, returning immediately")
|
||||
return {"status": "complete", "output_files": submit_result.get("output_files")}
|
||||
|
||||
# If in_progress, another request is encoding it — just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Encoding already in progress, joining poll")
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._get_worker_url()}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def submit_render_video_job(
|
||||
self,
|
||||
job_id: str,
|
||||
render_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a render-video job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
render_config: Configuration dict containing original_corrections_gcs_path,
|
||||
audio_gcs_path, output_gcs_prefix, artist, title, and any other render params.
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
RuntimeError: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._get_worker_url()}/render-video"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {"job_id": job_id, **render_config}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting render-video job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
# Job already exists on worker — check if it already completed
|
||||
logger.warning(f"[job:{job_id}] GCE worker returned 409, checking job status")
|
||||
try:
|
||||
status = await self.get_job_status(job_id)
|
||||
job_status = status.get("status", "unknown")
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] Render-video job already complete on GCE worker, returning cached result")
|
||||
return {
|
||||
"status": "cached",
|
||||
"job_id": job_id,
|
||||
"output_files": status.get("output_files"),
|
||||
"metadata": status.get("metadata"),
|
||||
}
|
||||
elif job_status in ("pending", "running"):
|
||||
logger.info(f"[job:{job_id}] Render-video job still in progress on GCE worker")
|
||||
return {"status": "in_progress", "job_id": job_id}
|
||||
else:
|
||||
raise RuntimeError(f"Render-video job {job_id} already exists with status: {job_status}")
|
||||
except RuntimeError as e:
|
||||
if "not found" in str(e).lower():
|
||||
raise RuntimeError(f"Render-video job {job_id} conflict: 409 but job not found on status check")
|
||||
raise
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit render-video job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def render_video_on_gce(
|
||||
self,
|
||||
job_id: str,
|
||||
render_config: Dict[str, Any],
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit render-video job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
render_config: Configuration dict for the render-video worker endpoint
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files and metadata
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_render_video_job(job_id, render_config)
|
||||
|
||||
# If cached, return immediately — rendering already done
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Render-video already cached, returning immediately")
|
||||
return {
|
||||
"status": "complete",
|
||||
"output_files": submit_result.get("output_files"),
|
||||
"metadata": submit_result.get("metadata"),
|
||||
}
|
||||
|
||||
# If in_progress, another request is rendering it — just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Render-video already in progress, joining poll")
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
@@ -706,7 +254,7 @@ class EncodingService:
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._get_worker_url()}/health"
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
@@ -728,20 +276,4 @@ def get_encoding_service() -> EncodingService:
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
# Wire up worker manager for dynamic URL resolution
|
||||
try:
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
from google.cloud import compute_v1, firestore
|
||||
settings = get_settings()
|
||||
db = firestore.Client(project=settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=settings.google_cloud_project,
|
||||
)
|
||||
_encoding_service.set_worker_manager(manager)
|
||||
except Exception:
|
||||
# Fallback to static URL if worker manager setup fails
|
||||
pass
|
||||
return _encoding_service
|
||||
@@ -1,239 +0,0 @@
|
||||
"""
|
||||
Blue-Green Encoding Worker Manager.
|
||||
|
||||
Manages the lifecycle and role assignment of two GCE encoding worker VMs
|
||||
(blue/green) via a Firestore config document as the single source of truth.
|
||||
|
||||
The config document at `config/encoding-worker` tracks which VM is primary
|
||||
(actively serving encoding requests) and which is secondary (idle or being
|
||||
deployed to). This enables zero-downtime deployments by swapping roles
|
||||
after deploying new code to the secondary.
|
||||
|
||||
Usage:
|
||||
from google.cloud import firestore
|
||||
from google.cloud.compute_v1 import InstancesClient
|
||||
|
||||
db = firestore.Client()
|
||||
compute = InstancesClient()
|
||||
manager = EncodingWorkerManager(db, compute)
|
||||
|
||||
# Get current config
|
||||
config = manager.get_config()
|
||||
print(f"Primary: {config.primary_vm} at {config.primary_url}")
|
||||
|
||||
# Ensure primary VM is running before dispatching work
|
||||
result = manager.ensure_primary_running()
|
||||
"""
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, UTC
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CONFIG_COLLECTION = "config"
|
||||
CONFIG_DOCUMENT = "encoding-worker"
|
||||
ENCODING_WORKER_PORT = 8080
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingWorkerConfig:
|
||||
"""Represents the Firestore config/encoding-worker document."""
|
||||
|
||||
primary_vm: str
|
||||
primary_ip: str
|
||||
primary_version: str
|
||||
primary_deployed_at: Optional[str]
|
||||
secondary_vm: str
|
||||
secondary_ip: str
|
||||
secondary_version: Optional[str]
|
||||
secondary_deployed_at: Optional[str]
|
||||
last_swap_at: Optional[str]
|
||||
last_activity_at: Optional[str]
|
||||
deploy_in_progress: bool
|
||||
deploy_in_progress_since: Optional[str]
|
||||
|
||||
@property
|
||||
def primary_url(self) -> str:
|
||||
return f"http://{self.primary_ip}:{ENCODING_WORKER_PORT}"
|
||||
|
||||
@property
|
||||
def secondary_url(self) -> str:
|
||||
return f"http://{self.secondary_ip}:{ENCODING_WORKER_PORT}"
|
||||
|
||||
|
||||
class EncodingWorkerManager:
|
||||
"""Manages blue-green encoding worker VMs via Firestore config and Compute API."""
|
||||
|
||||
def __init__(self, db, compute_client=None, project_id="nomadkaraoke", zone="us-central1-c"):
|
||||
self._db = db
|
||||
self._compute = compute_client
|
||||
self._project_id = project_id
|
||||
self._zone = zone
|
||||
|
||||
def _doc_ref(self):
|
||||
"""Return a reference to the config/encoding-worker document."""
|
||||
return self._db.collection(CONFIG_COLLECTION).document(CONFIG_DOCUMENT)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Firestore config operations (Task 1)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_config(self) -> EncodingWorkerConfig:
|
||||
"""Read config/encoding-worker from Firestore and return as dataclass."""
|
||||
snapshot = self._doc_ref().get()
|
||||
if not snapshot.exists:
|
||||
raise ValueError(
|
||||
f"Encoding worker config document not found at "
|
||||
f"{CONFIG_COLLECTION}/{CONFIG_DOCUMENT}"
|
||||
)
|
||||
data = snapshot.to_dict()
|
||||
return EncodingWorkerConfig(
|
||||
primary_vm=data["primary_vm"],
|
||||
primary_ip=data["primary_ip"],
|
||||
primary_version=data["primary_version"],
|
||||
primary_deployed_at=data.get("primary_deployed_at"),
|
||||
secondary_vm=data["secondary_vm"],
|
||||
secondary_ip=data["secondary_ip"],
|
||||
secondary_version=data.get("secondary_version"),
|
||||
secondary_deployed_at=data.get("secondary_deployed_at"),
|
||||
last_swap_at=data.get("last_swap_at"),
|
||||
last_activity_at=data.get("last_activity_at"),
|
||||
deploy_in_progress=data.get("deploy_in_progress", False),
|
||||
deploy_in_progress_since=data.get("deploy_in_progress_since"),
|
||||
)
|
||||
|
||||
def update_activity(self) -> None:
|
||||
"""Update last_activity_at timestamp to record recent usage."""
|
||||
now = datetime.now(UTC).isoformat()
|
||||
self._doc_ref().update({"last_activity_at": now})
|
||||
logger.info("Updated encoding worker last_activity_at to %s", now)
|
||||
|
||||
def swap_roles(
|
||||
self,
|
||||
new_primary_vm: str,
|
||||
new_primary_ip: str,
|
||||
new_primary_version: str,
|
||||
old_primary_vm: str,
|
||||
old_primary_ip: str,
|
||||
old_primary_version: str,
|
||||
) -> None:
|
||||
"""Atomically swap primary and secondary roles using a Firestore transaction.
|
||||
|
||||
Args:
|
||||
new_primary_vm: VM name that will become the new primary.
|
||||
new_primary_ip: Internal IP of the new primary.
|
||||
new_primary_version: Version deployed on the new primary.
|
||||
old_primary_vm: Current primary VM name (becomes secondary).
|
||||
old_primary_ip: Internal IP of the old primary.
|
||||
old_primary_version: Version on the old primary.
|
||||
"""
|
||||
transaction = self._db.transaction()
|
||||
doc_ref = self._doc_ref()
|
||||
now = datetime.now(UTC).isoformat()
|
||||
|
||||
@firestore.transactional
|
||||
def _swap_in_transaction(txn, ref):
|
||||
txn.update(ref, {
|
||||
"primary_vm": new_primary_vm,
|
||||
"primary_ip": new_primary_ip,
|
||||
"primary_version": new_primary_version,
|
||||
"primary_deployed_at": now,
|
||||
"secondary_vm": old_primary_vm,
|
||||
"secondary_ip": old_primary_ip,
|
||||
"secondary_version": old_primary_version,
|
||||
"secondary_deployed_at": None,
|
||||
"last_swap_at": now,
|
||||
"deploy_in_progress": False,
|
||||
"deploy_in_progress_since": None,
|
||||
})
|
||||
|
||||
_swap_in_transaction(transaction, doc_ref)
|
||||
logger.info(
|
||||
"Swapped encoding worker roles: %s -> primary, %s -> secondary",
|
||||
new_primary_vm, old_primary_vm,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# VM lifecycle operations (Task 2)
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_vm_status(self, vm_name: str) -> str:
|
||||
"""Get the current status of a GCE VM (e.g. RUNNING, TERMINATED, STAGING)."""
|
||||
instance = self._compute.get(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
return instance.status
|
||||
|
||||
def start_vm(self, vm_name: str) -> None:
|
||||
"""Start a VM if it is not already running or starting.
|
||||
|
||||
Fire-and-forget: returns immediately. Caller should poll get_vm_status
|
||||
if they need to wait for the VM to be fully RUNNING.
|
||||
"""
|
||||
status = self.get_vm_status(vm_name)
|
||||
if status in ("RUNNING", "STAGING"):
|
||||
logger.info("VM %s is already %s, skipping start", vm_name, status)
|
||||
return
|
||||
logger.info("Starting VM %s (current status: %s)", vm_name, status)
|
||||
self._compute.start(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
|
||||
def stop_vm(self, vm_name: str) -> None:
|
||||
"""Stop a VM if it is not already stopped or stopping."""
|
||||
status = self.get_vm_status(vm_name)
|
||||
if status in ("TERMINATED", "STOPPING"):
|
||||
logger.info("VM %s is already %s, skipping stop", vm_name, status)
|
||||
return
|
||||
logger.info("Stopping VM %s (current status: %s)", vm_name, status)
|
||||
self._compute.stop(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
|
||||
def ensure_primary_running(self) -> dict:
|
||||
"""Ensure the primary encoding worker VM is running.
|
||||
|
||||
Reads config, starts primary VM if stopped, and updates activity timestamp.
|
||||
|
||||
Returns:
|
||||
dict with keys:
|
||||
- started (bool): True if VM was started, False if already running
|
||||
- vm_name (str): Name of the primary VM
|
||||
- primary_url (str): URL of the primary encoding worker
|
||||
"""
|
||||
config = self.get_config()
|
||||
vm_name = config.primary_vm
|
||||
|
||||
status = self.get_vm_status(vm_name)
|
||||
started = False
|
||||
|
||||
if status not in ("RUNNING", "STAGING"):
|
||||
logger.info("Primary VM %s is %s, starting it", vm_name, status)
|
||||
self._compute.start(
|
||||
project=self._project_id,
|
||||
zone=self._zone,
|
||||
instance=vm_name,
|
||||
)
|
||||
started = True
|
||||
else:
|
||||
logger.info("Primary VM %s is already %s", vm_name, status)
|
||||
|
||||
self.update_activity()
|
||||
|
||||
return {
|
||||
"started": started,
|
||||
"vm_name": vm_name,
|
||||
"primary_url": config.primary_url,
|
||||
}
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""Production error monitoring service for Nomad Karaoke."""
|
||||
@@ -1,177 +0,0 @@
|
||||
"""Configuration for the production error monitoring service.
|
||||
|
||||
All constants, service lists, and environment-driven configuration
|
||||
functions used across the error monitor modules.
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# GCP project settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
GCP_PROJECT: str = os.environ.get("GCP_PROJECT", "nomadkaraoke")
|
||||
GCP_REGION: str = os.environ.get("GCP_REGION", "us-central1")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Log querying settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: How far back to look for errors on each run (minutes).
|
||||
LOOKBACK_MINUTES: int = 15
|
||||
|
||||
#: Maximum number of log entries to fetch per service per run.
|
||||
MAX_LOG_ENTRIES: int = 500
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Monitored services
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
MONITORED_CLOUD_RUN_SERVICES: list[str] = [
|
||||
"karaoke-backend",
|
||||
"karaoke-decide",
|
||||
"audio-separator",
|
||||
# Gen2 Cloud Functions appear as Cloud Run services in logs
|
||||
"gdrive-validator",
|
||||
"github-runner-manager",
|
||||
"backup-to-aws",
|
||||
"divebar-mirror",
|
||||
"kn-data-sync",
|
||||
"divebar-lookup",
|
||||
"encoding-worker-idle-shutdown",
|
||||
]
|
||||
|
||||
MONITORED_CLOUD_RUN_JOBS: list[str] = [
|
||||
"video-encoding-job",
|
||||
"lyrics-transcription-job",
|
||||
"audio-separation-job",
|
||||
"audio-download-job",
|
||||
]
|
||||
|
||||
# Gen2 Cloud Functions log as cloud_run_revision, not cloud_function.
|
||||
# Their names are in MONITORED_CLOUD_RUN_SERVICES above.
|
||||
MONITORED_CLOUD_FUNCTIONS: list[str] = []
|
||||
|
||||
MONITORED_GCE_INSTANCES: list[str] = [
|
||||
"encoding-worker-a",
|
||||
"encoding-worker-b",
|
||||
"flacfetch-vm",
|
||||
"divebar-sync-vm",
|
||||
]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Service dependency map
|
||||
# Human-readable description of which services depend on which workers.
|
||||
# Used as LLM context when generating root-cause analysis.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SERVICE_DEPENDENCY_MAP: str = """\
|
||||
karaoke-backend (main API)
|
||||
→ video-encoding-job (Cloud Run Job, triggered per order)
|
||||
→ lyrics-transcription-job (Cloud Run Job, triggered after audio-sep)
|
||||
→ audio-separation-job (Cloud Run Job, runs on Cloud Run GPU)
|
||||
→ audio-download-job (Cloud Run Job, uses flacfetch-vm)
|
||||
→ gdrive-validator (Cloud Function, validates output files)
|
||||
→ encoding-worker-a/b (GCE VMs, long-running encoding workers)
|
||||
|
||||
karaoke-decide (catalog / discovery API)
|
||||
→ kn_data_sync (Cloud Function, syncs KaraokeNerds catalog)
|
||||
→ divebar_mirror (Cloud Function, mirrors Divebar catalog)
|
||||
→ divebar_lookup (Cloud Function, single-song lookup)
|
||||
|
||||
audio-separator (Cloud Run service, GPU stem separation)
|
||||
→ audio-separation-job (Cloud Run Job, launched by audio-separator)
|
||||
|
||||
flacfetch-vm (GCE VM)
|
||||
→ audio-download-job (initiated by karaoke-backend)
|
||||
|
||||
divebar-sync-vm (GCE VM)
|
||||
→ divebar_mirror (Cloud Function, reads from this VM)
|
||||
|
||||
Shared infrastructure:
|
||||
→ runner_manager (Cloud Function, manages encoding worker lifecycle)
|
||||
→ encoding_worker_idle (Cloud Function, shuts down idle encoding workers)
|
||||
→ backup_to_aws (Cloud Function, daily GCS → S3 backup)
|
||||
"""
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM analysis settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
LLM_ANALYSIS_MODEL: str = os.environ.get("LLM_ANALYSIS_MODEL", "gemini-2.0-flash-001")
|
||||
LLM_VERTEX_LOCATION: str = os.environ.get("LLM_VERTEX_LOCATION", "us-central1")
|
||||
|
||||
#: Minimum number of active error patterns required before triggering LLM analysis.
|
||||
MIN_PATTERNS_FOR_ANALYSIS: int = 3
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Spike detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: An error count is a spike if it exceeds SPIKE_MULTIPLIER × baseline average.
|
||||
SPIKE_MULTIPLIER: float = 5.0
|
||||
|
||||
#: Minimum absolute error count before spike detection fires (avoids noise).
|
||||
SPIKE_MIN_COUNT: int = 5
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auto-resolution settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Multiply LOOKBACK_MINUTES by this to get auto-resolve window in minutes.
|
||||
AUTO_RESOLVE_MULTIPLIER: int = 8
|
||||
|
||||
#: Minimum hours before a pattern can be auto-resolved.
|
||||
AUTO_RESOLVE_MIN_HOURS: int = 6
|
||||
|
||||
#: Maximum hours before a pattern is force-resolved even if still active.
|
||||
AUTO_RESOLVE_MAX_HOURS: int = 168 # 1 week
|
||||
|
||||
#: Fallback resolve window (hours) if baseline data is unavailable.
|
||||
AUTO_RESOLVE_FALLBACK_HOURS: int = 48
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Discord settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maximum Discord messages to send per monitor run (rate-limit guard).
|
||||
MAX_DISCORD_MESSAGES_PER_RUN: int = 10
|
||||
|
||||
#: Discord API hard limit on message length.
|
||||
DISCORD_MAX_MESSAGE_LENGTH: int = 2000
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Pattern tracking settings
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maximum number of active error patterns to track in Firestore.
|
||||
MAX_ACTIVE_PATTERNS: int = 500
|
||||
|
||||
#: Rolling window used when computing error baselines.
|
||||
ROLLING_WINDOW_DAYS: int = 7
|
||||
|
||||
#: Maximum length of a normalized error message stored in Firestore.
|
||||
MAX_NORMALIZED_MESSAGE_LENGTH: int = 200
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helper functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def get_llm_enabled() -> bool:
|
||||
"""Return True if LLM analysis is enabled via the LLM_ANALYSIS_ENABLED env var."""
|
||||
return os.environ.get("LLM_ANALYSIS_ENABLED", "false").strip().lower() == "true"
|
||||
|
||||
|
||||
def get_discord_webhook_secret_name() -> str:
|
||||
"""Return the Secret Manager secret name for the Discord webhook URL."""
|
||||
return os.environ.get("DISCORD_WEBHOOK_SECRET_NAME", "error-monitor-discord-webhook")
|
||||
|
||||
|
||||
def get_digest_mode() -> bool:
|
||||
"""Return True if the monitor should run in digest mode (batch alerts instead of immediate).
|
||||
|
||||
In digest mode all alerts are bundled into a single Discord message sent at
|
||||
the end of the run, rather than sending one message per error pattern.
|
||||
"""
|
||||
return os.environ.get("DIGEST_MODE", "false").strip().lower() == "true"
|
||||
@@ -1,323 +0,0 @@
|
||||
"""Discord alert client for the production error monitoring service.
|
||||
|
||||
Provides formatting functions for each alert type, and an ErrorMonitorDiscord
|
||||
webhook client that rate-limits outbound messages to MAX_DISCORD_MESSAGES_PER_RUN
|
||||
per monitor run.
|
||||
|
||||
Usage:
|
||||
from backend.services.error_monitor.discord import (
|
||||
ErrorMonitorDiscord,
|
||||
format_new_pattern_alert,
|
||||
format_incident_alert,
|
||||
format_auto_resolved_alert,
|
||||
format_spike_alert,
|
||||
format_daily_digest,
|
||||
)
|
||||
|
||||
discord = ErrorMonitorDiscord(webhook_url="https://discord.com/api/webhooks/...")
|
||||
msg = format_new_pattern_alert(...)
|
||||
discord.send_message(msg)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import requests
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
DISCORD_MAX_MESSAGE_LENGTH,
|
||||
MAX_DISCORD_MESSAGES_PER_RUN,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
SEVERITY_EMOJI: dict[str, str] = {
|
||||
"P0": "🚨",
|
||||
"P1": "🔴",
|
||||
"P2": "⚠️",
|
||||
"P3": "🟡",
|
||||
}
|
||||
|
||||
_EASTERN = ZoneInfo("America/New_York")
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Private helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _now_eastern() -> str:
|
||||
"""Return the current Eastern time as 'HH:MM AM/PM ET' (DST-aware)."""
|
||||
now = datetime.now(tz=_EASTERN)
|
||||
return now.strftime("%I:%M %p ET").lstrip("0")
|
||||
|
||||
|
||||
def _truncate(msg: str) -> str:
|
||||
"""Truncate *msg* to DISCORD_MAX_MESSAGE_LENGTH characters."""
|
||||
if len(msg) <= DISCORD_MAX_MESSAGE_LENGTH:
|
||||
return msg
|
||||
suffix = "…"
|
||||
return msg[: DISCORD_MAX_MESSAGE_LENGTH - len(suffix)] + suffix
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Formatting functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def format_new_pattern_alert(
|
||||
service: str,
|
||||
resource_type: str,
|
||||
normalized_message: str,
|
||||
sample_message: str,
|
||||
count: int,
|
||||
first_seen: str,
|
||||
) -> str:
|
||||
"""Format a Discord alert for a newly-detected error pattern.
|
||||
|
||||
Args:
|
||||
service: Name of the service emitting the error.
|
||||
resource_type: GCP resource type (e.g. 'cloud_run_service', 'gce_instance').
|
||||
normalized_message: Normalised error pattern string.
|
||||
sample_message: A representative raw log message.
|
||||
count: Number of occurrences seen so far.
|
||||
first_seen: ISO 8601 timestamp when the pattern was first seen.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"🔴 **New Error Pattern Detected**",
|
||||
f"",
|
||||
f"**Service:** {service}",
|
||||
f"**Resource type:** {resource_type}",
|
||||
f"**Pattern:** {normalized_message}",
|
||||
f"**Sample:** {sample_message}",
|
||||
f"**Count:** {count}",
|
||||
f"**First seen:** {first_seen}",
|
||||
f"**Alerted at:** {_now_eastern()}",
|
||||
f"",
|
||||
f"To investigate: `/prod-investigate {service} \"{normalized_message}\"`",
|
||||
]
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_incident_alert(
|
||||
title: str,
|
||||
severity: str,
|
||||
root_cause: str,
|
||||
suggested_fix: str,
|
||||
primary_service: str,
|
||||
patterns: list[dict],
|
||||
) -> str:
|
||||
"""Format a Discord alert for an analysed incident (LLM-generated).
|
||||
|
||||
Args:
|
||||
title: Short incident title.
|
||||
severity: Severity level — one of P0, P1, P2, P3.
|
||||
root_cause: Human-readable root-cause analysis.
|
||||
suggested_fix: Recommended remediation steps.
|
||||
primary_service: The main service involved in the incident.
|
||||
patterns: List of dicts, each with keys: service, normalized_message, count.
|
||||
Only the first 5 are shown; normalized_message is truncated to 80 chars.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
emoji = SEVERITY_EMOJI.get(severity, "⚠️")
|
||||
lines = [
|
||||
f"{emoji} **[{severity}] {title}**",
|
||||
f"",
|
||||
f"**Primary service:** {primary_service}",
|
||||
f"**Root cause:** {root_cause}",
|
||||
f"**Suggested fix:** {suggested_fix}",
|
||||
]
|
||||
|
||||
shown_patterns = patterns[:5]
|
||||
if shown_patterns:
|
||||
lines.append("")
|
||||
lines.append(f"**Patterns ({len(shown_patterns)} of {len(patterns)}):**")
|
||||
for p in shown_patterns:
|
||||
svc = p.get("service", "unknown")
|
||||
msg = p.get("normalized_message", "")[:80]
|
||||
cnt = p.get("count", 0)
|
||||
lines.append(f" • `{svc}` — {msg} ({cnt}x)")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Alerted at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_auto_resolved_alert(resolved_patterns: list[dict]) -> str:
|
||||
"""Format a consolidated auto-resolved alert.
|
||||
|
||||
Args:
|
||||
resolved_patterns: List of dicts, each with keys: service, message, hours_silent.
|
||||
Only the first 8 are shown; message is truncated to 60 chars.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"✅ **Patterns Auto-Resolved**",
|
||||
f"",
|
||||
]
|
||||
|
||||
shown = resolved_patterns[:8]
|
||||
if shown:
|
||||
for p in shown:
|
||||
svc = p.get("service", "unknown")
|
||||
msg = p.get("message", "")[:60]
|
||||
hours = p.get("hours_silent", 0)
|
||||
lines.append(f" • `{svc}` — {msg} (silent {hours}h)")
|
||||
else:
|
||||
lines.append(" *(no patterns)*")
|
||||
|
||||
if len(resolved_patterns) > 8:
|
||||
lines.append(f" … and {len(resolved_patterns) - 8} more")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Resolved at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_spike_alert(
|
||||
service: str,
|
||||
normalized_message: str,
|
||||
current_count: int,
|
||||
rolling_average: float,
|
||||
) -> str:
|
||||
"""Format a Discord alert for an error rate spike.
|
||||
|
||||
Args:
|
||||
service: Name of the service.
|
||||
normalized_message: Normalised error pattern string.
|
||||
current_count: Error count in the current window.
|
||||
rolling_average: Baseline rolling average count.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
if rolling_average > 0:
|
||||
multiplier = current_count / rolling_average
|
||||
multiplier_str = f"{multiplier:.1f}x"
|
||||
else:
|
||||
multiplier_str = "∞x"
|
||||
|
||||
lines = [
|
||||
f"⚠️ **Error Rate Spike Detected**",
|
||||
f"",
|
||||
f"**Service:** {service}",
|
||||
f"**Pattern:** {normalized_message}",
|
||||
f"**Current count:** {current_count}",
|
||||
f"**Normal (rolling avg):** {rolling_average}",
|
||||
f"**Multiplier:** {multiplier_str} above normal",
|
||||
f"",
|
||||
f"**Alerted at:** {_now_eastern()}",
|
||||
]
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
def format_daily_digest(
|
||||
total_errors: int,
|
||||
new_patterns: int,
|
||||
resolved_patterns: int,
|
||||
active_patterns: int,
|
||||
per_service_counts: dict,
|
||||
) -> str:
|
||||
"""Format the daily 24h error digest.
|
||||
|
||||
Args:
|
||||
total_errors: Total error count in the last 24 hours.
|
||||
new_patterns: Number of new error patterns detected.
|
||||
resolved_patterns: Number of patterns auto-resolved.
|
||||
active_patterns: Total number of currently active patterns.
|
||||
per_service_counts: Mapping of service name → error count.
|
||||
Displayed sorted by count descending.
|
||||
|
||||
Returns:
|
||||
Formatted Discord message string, truncated to 2000 chars.
|
||||
"""
|
||||
lines = [
|
||||
f"📊 **Daily Error Digest (24h)**",
|
||||
f"",
|
||||
f"**Total errors:** {total_errors}",
|
||||
f"**New patterns:** {new_patterns}",
|
||||
f"**Resolved patterns:** {resolved_patterns}",
|
||||
f"**Active patterns:** {active_patterns}",
|
||||
]
|
||||
|
||||
if per_service_counts:
|
||||
lines.append("")
|
||||
lines.append("**Per-service breakdown:**")
|
||||
sorted_services = sorted(per_service_counts.items(), key=lambda kv: kv[1], reverse=True)
|
||||
for svc, cnt in sorted_services:
|
||||
lines.append(f" • `{svc}`: {cnt}")
|
||||
|
||||
lines.append("")
|
||||
lines.append(f"**Generated at:** {_now_eastern()}")
|
||||
|
||||
return _truncate("\n".join(lines))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Discord webhook client
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorMonitorDiscord:
|
||||
"""Webhook client for sending error monitor alerts to Discord.
|
||||
|
||||
Enforces a per-run rate limit of MAX_DISCORD_MESSAGES_PER_RUN to avoid
|
||||
flooding the channel during high-volume incident windows.
|
||||
|
||||
Attributes:
|
||||
webhook_url: The Discord incoming webhook URL.
|
||||
messages_sent: Counter of messages attempted this run (success or failure).
|
||||
"""
|
||||
|
||||
def __init__(self, webhook_url: str) -> None:
|
||||
self.webhook_url = webhook_url
|
||||
self.messages_sent: int = 0
|
||||
|
||||
def send_message(self, content: str) -> bool:
|
||||
"""Send a message to the configured Discord webhook.
|
||||
|
||||
Rate-limits: if messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN, the
|
||||
message is silently skipped and False is returned without incrementing
|
||||
the counter.
|
||||
|
||||
Args:
|
||||
content: The message text to send (should be pre-formatted and
|
||||
pre-truncated to ≤2000 chars).
|
||||
|
||||
Returns:
|
||||
True if the message was sent successfully, False otherwise.
|
||||
"""
|
||||
if self.messages_sent >= MAX_DISCORD_MESSAGES_PER_RUN:
|
||||
logger.warning(
|
||||
"Discord rate limit reached (%d messages sent this run); skipping message.",
|
||||
self.messages_sent,
|
||||
)
|
||||
return False
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
self.webhook_url,
|
||||
json={"content": content},
|
||||
timeout=30,
|
||||
)
|
||||
response.raise_for_status()
|
||||
logger.debug("Discord message sent successfully.")
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.error("Failed to send Discord message: %s", exc)
|
||||
return False
|
||||
finally:
|
||||
self.messages_sent += 1
|
||||
@@ -1,502 +0,0 @@
|
||||
"""Firestore adapter for the production error monitoring service.
|
||||
|
||||
Handles all CRUD for three Firestore collections:
|
||||
- error_patterns: normalized error patterns (pattern_id as doc ID)
|
||||
- error_incidents: groups of related patterns (incident_id as doc ID)
|
||||
- discord_alerts: audit trail for Discord messages (alert_id as doc ID)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import secrets
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
AUTO_RESOLVE_FALLBACK_HOURS,
|
||||
AUTO_RESOLVE_MAX_HOURS,
|
||||
AUTO_RESOLVE_MIN_HOURS,
|
||||
AUTO_RESOLVE_MULTIPLIER,
|
||||
ROLLING_WINDOW_DAYS,
|
||||
)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class PatternData:
|
||||
"""Input data for creating or updating an error pattern."""
|
||||
|
||||
pattern_id: str
|
||||
service: str
|
||||
resource_type: str
|
||||
normalized_message: str
|
||||
sample_message: str
|
||||
count: int
|
||||
timestamp: datetime
|
||||
|
||||
|
||||
@dataclass
|
||||
class UpsertResult:
|
||||
"""Result of an upsert_pattern operation."""
|
||||
|
||||
pattern_id: str
|
||||
is_new: bool
|
||||
previous_status: str | None = None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Collection names
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_COL_PATTERNS = "error_patterns"
|
||||
_COL_INCIDENTS = "error_incidents"
|
||||
_COL_ALERTS = "discord_alerts"
|
||||
|
||||
# Statuses that can be queried as "active"
|
||||
_ACTIVE_STATUSES = ["new", "acknowledged", "known"]
|
||||
|
||||
# Statuses from which a pattern is reactivated when the error recurs
|
||||
_REACTIVATION_STATUSES = {"auto_resolved", "fixed"}
|
||||
|
||||
# Statuses eligible for auto-resolve checking
|
||||
_AUTO_RESOLVE_ELIGIBLE = {"new", "acknowledged"}
|
||||
|
||||
|
||||
def _utcnow() -> datetime:
|
||||
"""Return current UTC datetime."""
|
||||
return datetime.now(tz=timezone.utc)
|
||||
|
||||
|
||||
def _iso(dt: datetime) -> str:
|
||||
"""Serialise a datetime to ISO-8601 string."""
|
||||
return dt.isoformat()
|
||||
|
||||
|
||||
def _generate_id(prefix: str, fmt: str) -> str:
|
||||
"""Generate a time-stamped ID with 8 hex random suffix.
|
||||
|
||||
Args:
|
||||
prefix: e.g. "inc" or "alert"
|
||||
fmt: strftime format string applied to utcnow(), e.g. "%Y%m%d_%H%M"
|
||||
"""
|
||||
ts = _utcnow().strftime(fmt)
|
||||
suffix = secrets.token_hex(4) # 8 hex chars
|
||||
return f"{prefix}_{ts}_{suffix}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Adapter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorPatternsAdapter:
|
||||
"""Firestore CRUD adapter for error monitoring collections.
|
||||
|
||||
Args:
|
||||
db: Optional Firestore client. If omitted, a default client for the
|
||||
``nomadkaraoke`` GCP project is constructed on first use.
|
||||
"""
|
||||
|
||||
def __init__(self, db=None) -> None:
|
||||
if db is None:
|
||||
from google.cloud import firestore # type: ignore[import]
|
||||
|
||||
db = firestore.Client(project="nomadkaraoke")
|
||||
self._db = db
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — read
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def get_pattern(self, pattern_id: str) -> dict | None:
|
||||
"""Return the pattern document as a dict, or None if it doesn't exist.
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID in the ``error_patterns`` collection.
|
||||
"""
|
||||
snap = self._db.collection(_COL_PATTERNS).document(pattern_id).get()
|
||||
if not snap.exists:
|
||||
return None
|
||||
return snap.to_dict()
|
||||
|
||||
def get_active_patterns(self) -> list[dict]:
|
||||
"""Return all patterns with status in [new, acknowledged, known]."""
|
||||
query = self._db.collection(_COL_PATTERNS).where(
|
||||
"status", "in", _ACTIVE_STATUSES
|
||||
)
|
||||
return [snap.to_dict() for snap in query.stream()]
|
||||
|
||||
def get_patterns_for_auto_resolve(self) -> list[dict]:
|
||||
"""Return active patterns eligible for auto-resolution (new or acknowledged)."""
|
||||
query = self._db.collection(_COL_PATTERNS).where(
|
||||
"status", "in", list(_AUTO_RESOLVE_ELIGIBLE)
|
||||
)
|
||||
return [snap.to_dict() for snap in query.stream()]
|
||||
|
||||
def get_unalerted_new_patterns(self) -> list[dict]:
|
||||
"""Return patterns with status="new" and alerted_at=None.
|
||||
|
||||
These are typically patterns written by a path other than the current
|
||||
monitor cycle — most notably frontend crashes ingested via
|
||||
``POST /api/client-errors``. The monitor uses this to ensure out-of-band
|
||||
patterns still get a Discord alert on the next scheduled run.
|
||||
"""
|
||||
query = self._db.collection(_COL_PATTERNS).where(
|
||||
"status", "==", "new"
|
||||
).where("alerted_at", "==", None)
|
||||
return [snap.to_dict() for snap in query.stream()]
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — upsert
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def upsert_pattern(self, data: PatternData) -> UpsertResult:
|
||||
"""Create a new pattern or update an existing one.
|
||||
|
||||
New pattern:
|
||||
- status="new", severity="P3"
|
||||
- Initialize ``rolling_counts`` with one entry for this run.
|
||||
|
||||
Existing pattern:
|
||||
- Increment ``total_count`` and append to ``rolling_counts``.
|
||||
- If status was ``auto_resolved`` or ``fixed`` → reactivate (set status="new").
|
||||
- Prune ``rolling_counts`` entries older than ROLLING_WINDOW_DAYS.
|
||||
|
||||
Returns:
|
||||
UpsertResult with ``is_new=True`` for brand-new patterns *and* for
|
||||
reactivated patterns (auto_resolved → new, fixed → new).
|
||||
"""
|
||||
doc_ref = self._db.collection(_COL_PATTERNS).document(data.pattern_id)
|
||||
snap = doc_ref.get()
|
||||
ts_iso = _iso(data.timestamp)
|
||||
|
||||
new_rolling_entry = {"ts": ts_iso, "count": data.count}
|
||||
|
||||
if not snap.exists:
|
||||
# ── brand new pattern ──────────────────────────────────────────
|
||||
doc_ref.set(
|
||||
{
|
||||
"pattern_id": data.pattern_id,
|
||||
"service": data.service,
|
||||
"resource_type": data.resource_type,
|
||||
"normalized_message": data.normalized_message,
|
||||
"sample_message": data.sample_message,
|
||||
"status": "new",
|
||||
"severity": "P3",
|
||||
"total_count": data.count,
|
||||
"rolling_counts": [new_rolling_entry],
|
||||
"first_seen": ts_iso,
|
||||
"last_seen": ts_iso,
|
||||
"alerted_at": None,
|
||||
"created_at": ts_iso,
|
||||
"updated_at": ts_iso,
|
||||
}
|
||||
)
|
||||
return UpsertResult(pattern_id=data.pattern_id, is_new=True)
|
||||
|
||||
# ── existing pattern ───────────────────────────────────────────────
|
||||
existing = snap.to_dict()
|
||||
prev_status = existing.get("status", "new")
|
||||
|
||||
rolling_counts: list[dict] = list(existing.get("rolling_counts") or [])
|
||||
rolling_counts.append(new_rolling_entry)
|
||||
rolling_counts = self._prune_rolling_counts(rolling_counts)
|
||||
|
||||
update: dict[str, Any] = {
|
||||
"total_count": existing.get("total_count", 0) + data.count,
|
||||
"rolling_counts": rolling_counts,
|
||||
"last_seen": ts_iso,
|
||||
"sample_message": data.sample_message,
|
||||
"updated_at": ts_iso,
|
||||
}
|
||||
|
||||
# Reactivation: pattern was considered resolved/fixed but error recurred
|
||||
is_new = False
|
||||
if prev_status in _REACTIVATION_STATUSES:
|
||||
update["status"] = "new"
|
||||
update["alerted_at"] = None
|
||||
is_new = True
|
||||
|
||||
doc_ref.update(update)
|
||||
return UpsertResult(
|
||||
pattern_id=data.pattern_id,
|
||||
is_new=is_new,
|
||||
previous_status=prev_status,
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Patterns — status changes
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def auto_resolve_pattern(self, pattern_id: str, hours_silent: float) -> None:
|
||||
"""Mark a pattern as auto-resolved.
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID.
|
||||
hours_silent: How many hours the pattern has been silent.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"status": "auto_resolved",
|
||||
"auto_resolve_hours_silent": hours_silent,
|
||||
"auto_resolved_at": _iso(_utcnow()),
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
def resolve_pattern(
|
||||
self,
|
||||
pattern_id: str,
|
||||
pr_url: str | None = None,
|
||||
note: str = "",
|
||||
) -> None:
|
||||
"""Mark a pattern as fixed (human-resolved).
|
||||
|
||||
Args:
|
||||
pattern_id: Firestore document ID.
|
||||
pr_url: Optional URL of the PR that fixed this pattern.
|
||||
note: Optional human-readable note about the fix.
|
||||
"""
|
||||
now_iso = _iso(_utcnow())
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"status": "fixed",
|
||||
"fixed_by": {
|
||||
"pr_url": pr_url,
|
||||
"note": note,
|
||||
"fixed_at": now_iso,
|
||||
},
|
||||
"updated_at": now_iso,
|
||||
}
|
||||
)
|
||||
|
||||
def update_pattern_alerted(self, pattern_id: str) -> None:
|
||||
"""Set the ``alerted_at`` timestamp on a pattern.
|
||||
|
||||
Called after a Discord alert has been successfully sent for the pattern.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"alerted_at": _iso(_utcnow()),
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
def merge_pattern(self, source_id: str, target_id: str, reason: str) -> None:
|
||||
"""Merge *source* into *target* by muting the source pattern.
|
||||
|
||||
Args:
|
||||
source_id: Pattern to mute.
|
||||
target_id: Pattern that the source is being merged into.
|
||||
reason: Human-readable rationale for the merge.
|
||||
"""
|
||||
self._db.collection(_COL_PATTERNS).document(source_id).update(
|
||||
{
|
||||
"status": "muted",
|
||||
"merged_into": target_id,
|
||||
"merge_reason": reason,
|
||||
"updated_at": _iso(_utcnow()),
|
||||
}
|
||||
)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Auto-resolve logic
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def check_auto_resolve(self, pattern: dict) -> float | None:
|
||||
"""Determine whether *pattern* should be auto-resolved.
|
||||
|
||||
Computes the frequency-aware silence threshold and compares it against
|
||||
how long the pattern has been silent.
|
||||
|
||||
Args:
|
||||
pattern: A pattern document dict (as returned by get_pattern).
|
||||
|
||||
Returns:
|
||||
The number of hours the pattern has been silent if the threshold is
|
||||
exceeded, or ``None`` if the pattern should not yet be resolved.
|
||||
"""
|
||||
status = pattern.get("status", "")
|
||||
if status not in _AUTO_RESOLVE_ELIGIBLE:
|
||||
return None
|
||||
|
||||
last_seen_str = pattern.get("last_seen")
|
||||
if not last_seen_str:
|
||||
return None
|
||||
|
||||
last_seen = datetime.fromisoformat(last_seen_str)
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
|
||||
# Ensure both datetimes are tz-aware for subtraction
|
||||
if last_seen.tzinfo is None:
|
||||
last_seen = last_seen.replace(tzinfo=timezone.utc)
|
||||
|
||||
hours_silent = (now - last_seen).total_seconds() / 3600.0
|
||||
|
||||
rolling_counts = pattern.get("rolling_counts") or []
|
||||
threshold = self._compute_resolve_threshold(rolling_counts)
|
||||
|
||||
if hours_silent >= threshold:
|
||||
return hours_silent
|
||||
return None
|
||||
|
||||
def _compute_resolve_threshold(self, rolling_counts: list[dict]) -> float:
|
||||
"""Compute the frequency-aware auto-resolve threshold in hours.
|
||||
|
||||
Algorithm:
|
||||
1. If fewer than 3 data points → return AUTO_RESOLVE_FALLBACK_HOURS (48).
|
||||
2. Compute mean interval between consecutive timestamps.
|
||||
3. threshold = mean_interval_hours * AUTO_RESOLVE_MULTIPLIER.
|
||||
4. Clamp result to [AUTO_RESOLVE_MIN_HOURS, AUTO_RESOLVE_MAX_HOURS].
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
|
||||
|
||||
Returns:
|
||||
Threshold in hours (float).
|
||||
"""
|
||||
if len(rolling_counts) < 3:
|
||||
return float(AUTO_RESOLVE_FALLBACK_HOURS)
|
||||
|
||||
# Parse timestamps and sort ascending
|
||||
timestamps: list[datetime] = []
|
||||
for entry in rolling_counts:
|
||||
ts = datetime.fromisoformat(entry["ts"])
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
timestamps.append(ts)
|
||||
timestamps.sort()
|
||||
|
||||
# Compute pairwise intervals
|
||||
intervals_hours: list[float] = []
|
||||
for i in range(1, len(timestamps)):
|
||||
delta = (timestamps[i] - timestamps[i - 1]).total_seconds() / 3600.0
|
||||
intervals_hours.append(delta)
|
||||
|
||||
if not intervals_hours:
|
||||
return float(AUTO_RESOLVE_FALLBACK_HOURS)
|
||||
|
||||
mean_interval = sum(intervals_hours) / len(intervals_hours)
|
||||
threshold = mean_interval * AUTO_RESOLVE_MULTIPLIER
|
||||
|
||||
# Clamp to configured bounds
|
||||
threshold = max(float(AUTO_RESOLVE_MIN_HOURS), threshold)
|
||||
threshold = min(float(AUTO_RESOLVE_MAX_HOURS), threshold)
|
||||
return threshold
|
||||
|
||||
def _prune_rolling_counts(self, rolling_counts: list[dict]) -> list[dict]:
|
||||
"""Remove rolling_count entries older than ROLLING_WINDOW_DAYS.
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"ts": ISO-string, "count": int}`` dicts.
|
||||
|
||||
Returns:
|
||||
Filtered list with only recent entries.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return []
|
||||
|
||||
cutoff = _utcnow() - timedelta(days=ROLLING_WINDOW_DAYS)
|
||||
result = []
|
||||
for entry in rolling_counts:
|
||||
ts = datetime.fromisoformat(entry["ts"])
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
if ts >= cutoff:
|
||||
result.append(entry)
|
||||
return result
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Incidents
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def create_incident(
|
||||
self,
|
||||
title: str,
|
||||
root_cause: str,
|
||||
severity: str,
|
||||
suggested_fix: str,
|
||||
primary_service: str,
|
||||
pattern_ids: list[str],
|
||||
used_llm: bool,
|
||||
) -> str:
|
||||
"""Create a new incident document and link associated patterns.
|
||||
|
||||
Args:
|
||||
title: Short human-readable title.
|
||||
root_cause: LLM-generated or manual root-cause description.
|
||||
severity: Severity string (e.g. "P1", "P2", "P3").
|
||||
suggested_fix: Suggested remediation steps.
|
||||
primary_service: The primary affected service.
|
||||
pattern_ids: List of pattern_ids that belong to this incident.
|
||||
used_llm: Whether LLM analysis was used to generate this incident.
|
||||
|
||||
Returns:
|
||||
The generated incident_id string.
|
||||
"""
|
||||
incident_id = _generate_id("inc", "%Y%m%d_%H%M")
|
||||
now_iso = _iso(_utcnow())
|
||||
|
||||
self._db.collection(_COL_INCIDENTS).document(incident_id).set(
|
||||
{
|
||||
"incident_id": incident_id,
|
||||
"title": title,
|
||||
"root_cause": root_cause,
|
||||
"severity": severity,
|
||||
"suggested_fix": suggested_fix,
|
||||
"primary_service": primary_service,
|
||||
"pattern_ids": pattern_ids,
|
||||
"used_llm": used_llm,
|
||||
"created_at": now_iso,
|
||||
"updated_at": now_iso,
|
||||
"status": "open",
|
||||
}
|
||||
)
|
||||
|
||||
# Link each pattern back to this incident
|
||||
for pattern_id in pattern_ids:
|
||||
self._db.collection(_COL_PATTERNS).document(pattern_id).update(
|
||||
{
|
||||
"incident_id": incident_id,
|
||||
"updated_at": now_iso,
|
||||
}
|
||||
)
|
||||
|
||||
return incident_id
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Discord audit trail
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def log_discord_alert(
|
||||
self,
|
||||
alert_type: str,
|
||||
content: str,
|
||||
success: bool,
|
||||
metadata: dict | None = None,
|
||||
) -> None:
|
||||
"""Write an audit record to the ``discord_alerts`` collection.
|
||||
|
||||
Args:
|
||||
alert_type: Category of alert (e.g. "new_pattern", "digest").
|
||||
content: The message body that was (or would have been) sent.
|
||||
success: Whether the Discord webhook call succeeded.
|
||||
metadata: Optional additional data (pattern_ids, incident_id, etc.).
|
||||
"""
|
||||
alert_id = _generate_id("alert", "%Y%m%d_%H%M%S")
|
||||
now_iso = _iso(_utcnow())
|
||||
|
||||
self._db.collection(_COL_ALERTS).document(alert_id).set(
|
||||
{
|
||||
"alert_id": alert_id,
|
||||
"alert_type": alert_type,
|
||||
"content": content,
|
||||
"success": success,
|
||||
"metadata": metadata or {},
|
||||
"created_at": now_iso,
|
||||
}
|
||||
)
|
||||
@@ -1,127 +0,0 @@
|
||||
"""Frontend error ingestion helpers.
|
||||
|
||||
Converts an inbound browser crash report into a ``PatternData`` suitable for
|
||||
the shared ``ErrorPatternsAdapter``. The adapter + existing error-monitor
|
||||
Cloud Run Job handle all alerting / Discord plumbing.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
|
||||
from backend.services.error_monitor.firestore_adapter import PatternData
|
||||
from backend.services.error_monitor.normalizer import (
|
||||
compute_pattern_hash,
|
||||
normalize_message,
|
||||
)
|
||||
|
||||
MAX_SAMPLE_MESSAGE_CHARS = 4000
|
||||
MAX_URL_CHARS = 512
|
||||
|
||||
|
||||
@dataclass
|
||||
class FrontendErrorReport:
|
||||
"""In-memory representation of an inbound crash report."""
|
||||
|
||||
message: str
|
||||
stack: str | None
|
||||
url: str
|
||||
user_agent: str
|
||||
release: str
|
||||
user_email: str | None
|
||||
viewport: dict | None
|
||||
locale: str
|
||||
extra: dict | None
|
||||
|
||||
|
||||
def sanitize_url(url: str) -> str:
|
||||
"""Strip query and fragment from a URL; cap length; tolerate junk."""
|
||||
if not url:
|
||||
return ""
|
||||
try:
|
||||
parts = urlsplit(url)
|
||||
if not parts.scheme:
|
||||
return url[:MAX_URL_CHARS]
|
||||
cleaned = urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))
|
||||
return cleaned[:MAX_URL_CHARS]
|
||||
except ValueError:
|
||||
return url[:MAX_URL_CHARS]
|
||||
|
||||
|
||||
def _stack_for_hashing(report: FrontendErrorReport) -> str:
|
||||
"""Pick the most stable signal we have for pattern dedup.
|
||||
|
||||
Prefer the stack (stable across invocations) over the message (sometimes
|
||||
has interpolated values). Falls back to message if no stack.
|
||||
"""
|
||||
if report.stack:
|
||||
return report.stack
|
||||
return report.message
|
||||
|
||||
|
||||
def build_pattern_data(
|
||||
report: FrontendErrorReport, now: datetime | None = None
|
||||
) -> PatternData:
|
||||
"""Convert an inbound report to a PatternData ready for upsert."""
|
||||
if now is None:
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
|
||||
raw = _stack_for_hashing(report)
|
||||
normalized = normalize_message(raw)
|
||||
pattern_id = compute_pattern_hash("frontend", normalized)
|
||||
|
||||
# sample_message is human-readable context. Keep the error message plus a
|
||||
# trimmed stack + sanitized URL so the Discord alert is self-contained.
|
||||
sample_parts: list[str] = []
|
||||
if report.message:
|
||||
sample_parts.append(report.message.strip())
|
||||
if report.stack and report.stack.strip() != (report.message or "").strip():
|
||||
sample_parts.append(report.stack.strip())
|
||||
clean_url = sanitize_url(report.url)
|
||||
if clean_url:
|
||||
sample_parts.append(f"URL: {clean_url}")
|
||||
if report.user_agent:
|
||||
sample_parts.append(f"UA: {report.user_agent[:200]}")
|
||||
if report.release:
|
||||
sample_parts.append(f"Build: {report.release}")
|
||||
sample_message = "\n".join(sample_parts)[:MAX_SAMPLE_MESSAGE_CHARS]
|
||||
|
||||
return PatternData(
|
||||
pattern_id=pattern_id,
|
||||
service="frontend",
|
||||
resource_type="browser",
|
||||
normalized_message=normalized,
|
||||
sample_message=sample_message,
|
||||
count=1,
|
||||
timestamp=now,
|
||||
)
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
"""In-memory sliding-window limiter. One instance per process is enough —
|
||||
this runs inside Cloud Run which scales to multiple instances, so the
|
||||
effective limit is (per_ip_per_minute * num_instances). That's fine for our
|
||||
threat model (non-malicious browsers reporting their own crashes).
|
||||
"""
|
||||
|
||||
def __init__(self, max_per_minute: int = 60) -> None:
|
||||
self._max = max_per_minute
|
||||
self._hits: dict[str, list[float]] = {}
|
||||
|
||||
def allow(self, ip: str, now_ts: float) -> bool:
|
||||
cutoff = now_ts - 60.0
|
||||
hits = [t for t in self._hits.get(ip, []) if t >= cutoff]
|
||||
if len(hits) >= self._max:
|
||||
self._hits[ip] = hits
|
||||
return False
|
||||
hits.append(now_ts)
|
||||
self._hits[ip] = hits
|
||||
# simple cleanup: if the map gets huge, drop stale keys
|
||||
if len(self._hits) > 10_000:
|
||||
self._hits = {
|
||||
k: [t for t in v if t >= cutoff]
|
||||
for k, v in self._hits.items()
|
||||
if any(t >= cutoff for t in v)
|
||||
}
|
||||
return True
|
||||
@@ -1,106 +0,0 @@
|
||||
"""Known infrastructure noise patterns for the production error monitoring service.
|
||||
|
||||
Filters out transient, expected log messages so the error monitor is not flooded
|
||||
with Cloud Run cold-start probes, deploy events, spot VM preemptions, and similar
|
||||
operational noise.
|
||||
|
||||
Usage:
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
|
||||
reason = should_ignore(service, message)
|
||||
if reason:
|
||||
# skip this log entry — it's known infrastructure noise
|
||||
...
|
||||
"""
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class IgnoreReason:
|
||||
"""Describes why a log entry should be suppressed.
|
||||
|
||||
Attributes:
|
||||
pattern_name: Machine-readable name of the matching ignore pattern.
|
||||
reason: Human-readable explanation of why this is infrastructure noise.
|
||||
"""
|
||||
|
||||
pattern_name: str
|
||||
reason: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compiled ignore patterns (applied in order)
|
||||
# All patterns use re.IGNORECASE for case-insensitive matching.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_IGNORE_PATTERNS: list[tuple[str, re.Pattern, str]] = [
|
||||
(
|
||||
"startup_probe",
|
||||
re.compile(r"startup probe failed", re.IGNORECASE),
|
||||
"Cloud Run cold start — startup probe failure is transient and expected",
|
||||
),
|
||||
(
|
||||
"ready_condition",
|
||||
re.compile(r"ready condition status changed", re.IGNORECASE),
|
||||
"Cloud Run deploy event — readiness condition change is expected on deploy",
|
||||
),
|
||||
(
|
||||
"spot_preemption",
|
||||
re.compile(r"instance was preempted", re.IGNORECASE),
|
||||
"Spot/preemptible VM preemption — expected behaviour for spot instances",
|
||||
),
|
||||
(
|
||||
"idle_shutdown",
|
||||
re.compile(r"stopping idle encoding worker", re.IGNORECASE),
|
||||
"Encoding worker idle shutdown — expected behaviour when no jobs are queued",
|
||||
),
|
||||
(
|
||||
"health_check_404",
|
||||
re.compile(
|
||||
r"(?:GET|HEAD)\b[^\n]*?/health[^\n]*?\b(?:404|not found)\b",
|
||||
re.IGNORECASE,
|
||||
),
|
||||
"Load balancer health check returning 404 — transient during cold start",
|
||||
),
|
||||
(
|
||||
"scheduler_retry",
|
||||
re.compile(r"cloud scheduler.*retry", re.IGNORECASE),
|
||||
"Cloud Scheduler transient retry — expected on intermittent job failures",
|
||||
),
|
||||
(
|
||||
"runner_startup",
|
||||
re.compile(r"github\.runner\..*(?:starting|stopping|idle)", re.IGNORECASE),
|
||||
"GitHub Actions runner lifecycle event — expected during runner start/stop",
|
||||
),
|
||||
(
|
||||
"container_shutdown",
|
||||
re.compile(r"container called exit\(0\)", re.IGNORECASE),
|
||||
"Container clean exit — exit code 0 indicates a graceful, expected shutdown",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def should_ignore(service: str, message: str) -> IgnoreReason | None:
|
||||
"""Check whether a log entry matches a known infrastructure noise pattern.
|
||||
|
||||
The check is purely message-based; ``service`` is accepted for API symmetry
|
||||
and future service-specific overrides but does not gate any pattern today.
|
||||
|
||||
Args:
|
||||
service: The originating service name (e.g. ``"karaoke-backend"``).
|
||||
message: The raw log message string to evaluate.
|
||||
|
||||
Returns:
|
||||
An :class:`IgnoreReason` if the message matches a known noise pattern,
|
||||
or ``None`` if the message should be processed normally.
|
||||
"""
|
||||
if not message:
|
||||
return None
|
||||
|
||||
for pattern_name, pattern, reason in _IGNORE_PATTERNS:
|
||||
if pattern.search(message):
|
||||
return IgnoreReason(pattern_name=pattern_name, reason=reason)
|
||||
|
||||
return None
|
||||
@@ -1,429 +0,0 @@
|
||||
"""LLM-powered incident analysis and duplicate detection for the error monitor.
|
||||
|
||||
Uses Gemini Flash via the google-generativeai library to:
|
||||
1. Group related error patterns into incidents with root-cause analysis.
|
||||
2. Identify near-duplicate patterns that the regex normalizer missed.
|
||||
|
||||
Both functions fall back gracefully when the LLM is unavailable or returns
|
||||
unparseable output.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from google import genai
|
||||
from google.genai import types as genai_types
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
GCP_PROJECT,
|
||||
LLM_ANALYSIS_MODEL,
|
||||
LLM_VERTEX_LOCATION,
|
||||
MIN_PATTERNS_FOR_ANALYSIS,
|
||||
SERVICE_DEPENDENCY_MAP,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Dataclasses
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclass
|
||||
class Incident:
|
||||
"""A grouped incident produced by LLM analysis or the fallback grouper."""
|
||||
|
||||
title: str
|
||||
root_cause: str | None
|
||||
severity: str # P0 / P1 / P2 / P3
|
||||
suggested_fix: str | None
|
||||
primary_service: str
|
||||
pattern_indices: list[int]
|
||||
|
||||
|
||||
@dataclass
|
||||
class IncidentAnalysis:
|
||||
"""The result of an incident grouping pass."""
|
||||
|
||||
incidents: list[Incident]
|
||||
used_llm: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class DuplicateGroup:
|
||||
"""A canonical pattern plus the new patterns that duplicate it."""
|
||||
|
||||
canonical_index: int
|
||||
duplicate_indices: list[int]
|
||||
reason: str
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM call helper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_RETRY_ATTEMPTS = 3
|
||||
_RETRY_BASE_DELAY = 2.0 # seconds
|
||||
|
||||
|
||||
def _call_llm(
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
temperature: float = 0.2,
|
||||
) -> str:
|
||||
"""Call Gemini and return the response text.
|
||||
|
||||
Retries up to _RETRY_ATTEMPTS times with exponential back-off on failure.
|
||||
|
||||
Args:
|
||||
system_prompt: System instruction for the model.
|
||||
user_prompt: User-facing prompt content.
|
||||
temperature: Sampling temperature (lower = more deterministic).
|
||||
|
||||
Returns:
|
||||
The model's response text.
|
||||
|
||||
Raises:
|
||||
Exception: Re-raises the last exception after all retries are exhausted.
|
||||
"""
|
||||
last_exc: Exception | None = None
|
||||
|
||||
for attempt in range(_RETRY_ATTEMPTS):
|
||||
try:
|
||||
client = genai.Client(
|
||||
vertexai=True,
|
||||
project=GCP_PROJECT,
|
||||
location=LLM_VERTEX_LOCATION,
|
||||
)
|
||||
response = client.models.generate_content(
|
||||
model=LLM_ANALYSIS_MODEL,
|
||||
contents=user_prompt,
|
||||
config=genai_types.GenerateContentConfig(
|
||||
system_instruction=system_prompt,
|
||||
temperature=temperature,
|
||||
),
|
||||
)
|
||||
return response.text
|
||||
except Exception as exc: # noqa: BLE001
|
||||
last_exc = exc
|
||||
delay = _RETRY_BASE_DELAY * (2**attempt)
|
||||
logger.warning(
|
||||
"LLM call failed (attempt %d/%d): %s — retrying in %.1fs",
|
||||
attempt + 1,
|
||||
_RETRY_ATTEMPTS,
|
||||
exc,
|
||||
delay,
|
||||
)
|
||||
time.sleep(delay)
|
||||
|
||||
raise last_exc # type: ignore[misc]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Response parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MD_CODE_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)```", re.IGNORECASE)
|
||||
|
||||
|
||||
def _strip_markdown_code_block(text: str) -> str:
|
||||
"""Remove surrounding ```json ... ``` or ``` ... ``` fences if present."""
|
||||
match = _MD_CODE_BLOCK_RE.search(text)
|
||||
if match:
|
||||
return match.group(1).strip()
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _parse_llm_response(text: str) -> IncidentAnalysis | None:
|
||||
"""Parse the LLM response text into an IncidentAnalysis.
|
||||
|
||||
Args:
|
||||
text: Raw LLM output, possibly wrapped in a markdown code block.
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis on success, None on any parse failure.
|
||||
"""
|
||||
if not text:
|
||||
return None
|
||||
|
||||
cleaned = _strip_markdown_code_block(text)
|
||||
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("LLM returned non-JSON response (first 200 chars): %.200s", text)
|
||||
return None
|
||||
|
||||
if "incidents" not in data:
|
||||
logger.warning("LLM response missing 'incidents' key: %s", list(data.keys()))
|
||||
return None
|
||||
|
||||
incidents: list[Incident] = []
|
||||
for raw in data["incidents"]:
|
||||
try:
|
||||
incidents.append(
|
||||
Incident(
|
||||
title=raw.get("title", "Unknown incident"),
|
||||
root_cause=raw.get("root_cause"),
|
||||
severity=raw.get("severity", "P2"),
|
||||
suggested_fix=raw.get("suggested_fix"),
|
||||
primary_service=raw.get("primary_service", "unknown"),
|
||||
pattern_indices=raw.get("pattern_indices", []),
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Skipping malformed incident entry: %s — %s", raw, exc)
|
||||
|
||||
return IncidentAnalysis(incidents=incidents, used_llm=True)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fallback grouper
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _fallback_group_by_service(patterns: list[dict]) -> IncidentAnalysis:
|
||||
"""Group patterns by service name when the LLM is unavailable.
|
||||
|
||||
Each unique service becomes one P2 incident containing all of that
|
||||
service's pattern indices.
|
||||
|
||||
Args:
|
||||
patterns: List of error pattern dicts (must include 'service' key).
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis with used_llm=False.
|
||||
"""
|
||||
service_to_indices: dict[str, list[int]] = defaultdict(list)
|
||||
for idx, pattern in enumerate(patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
service_to_indices[service].append(idx)
|
||||
|
||||
incidents = [
|
||||
Incident(
|
||||
title=f"Errors in {service}",
|
||||
root_cause=None,
|
||||
severity="P2",
|
||||
suggested_fix=None,
|
||||
primary_service=service,
|
||||
pattern_indices=indices,
|
||||
)
|
||||
for service, indices in service_to_indices.items()
|
||||
]
|
||||
|
||||
return IncidentAnalysis(incidents=incidents, used_llm=False)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main analysis functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_ANALYZE_SYSTEM_PROMPT = f"""\
|
||||
You are an expert site-reliability engineer for Nomad Karaoke, a karaoke video
|
||||
generation platform running on Google Cloud Platform.
|
||||
|
||||
Your task is to analyse a list of active error patterns detected across
|
||||
production services, group them into meaningful incidents, and identify root
|
||||
causes.
|
||||
|
||||
## Service dependency map
|
||||
|
||||
{SERVICE_DEPENDENCY_MAP}
|
||||
|
||||
## Severity definitions
|
||||
|
||||
- P0: Service is completely down / total outage
|
||||
- P1: Major pipeline broken — orders cannot complete
|
||||
- P2: Degraded performance — some orders affected
|
||||
- P3: Minor / cosmetic issue with no direct user impact
|
||||
|
||||
## Output format
|
||||
|
||||
Respond with a single JSON object (no markdown fencing) with this schema:
|
||||
|
||||
{{
|
||||
"incidents": [
|
||||
{{
|
||||
"title": "<short human-readable title>",
|
||||
"root_cause": "<root cause or null>",
|
||||
"severity": "P0|P1|P2|P3",
|
||||
"suggested_fix": "<actionable suggestion or null>",
|
||||
"primary_service": "<service name>",
|
||||
"pattern_indices": [<list of 0-based indices from the input list>]
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
Group related patterns together into a single incident where they share a
|
||||
common root cause (e.g., a downstream dependency failure). Unrelated patterns
|
||||
should be in separate incidents. Every input pattern must appear in exactly
|
||||
one incident.
|
||||
"""
|
||||
|
||||
_DEDUP_SYSTEM_PROMPT = """\
|
||||
You are an expert at identifying near-duplicate error patterns in production
|
||||
monitoring systems.
|
||||
|
||||
Your task is to compare a list of NEW error patterns against a list of
|
||||
EXISTING (canonical) patterns and identify which new patterns are essentially
|
||||
duplicates of existing ones — i.e., they describe the same underlying error
|
||||
but with slightly different wording due to variation in log messages.
|
||||
|
||||
## Output format
|
||||
|
||||
Respond with a single JSON object (no markdown fencing) with this schema:
|
||||
|
||||
{
|
||||
"duplicates": [
|
||||
{
|
||||
"canonical_index": <0-based index into the EXISTING patterns list>,
|
||||
"duplicate_indices": [<0-based indices into the NEW patterns list>],
|
||||
"reason": "<brief explanation>"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
If no duplicates are found, return {"duplicates": []}.
|
||||
"""
|
||||
|
||||
|
||||
def analyze_patterns(patterns: list[dict]) -> IncidentAnalysis | None:
|
||||
"""Group error patterns into incidents using Gemini LLM analysis.
|
||||
|
||||
Returns None if there are fewer than MIN_PATTERNS_FOR_ANALYSIS patterns.
|
||||
Falls back to service-based grouping when the LLM fails or returns an
|
||||
unparseable response.
|
||||
|
||||
Args:
|
||||
patterns: List of error pattern dicts to analyse.
|
||||
|
||||
Returns:
|
||||
IncidentAnalysis, or None if insufficient patterns.
|
||||
"""
|
||||
if len(patterns) < MIN_PATTERNS_FOR_ANALYSIS:
|
||||
return None
|
||||
|
||||
user_prompt = _build_analyze_user_prompt(patterns)
|
||||
|
||||
try:
|
||||
raw_response = _call_llm(_ANALYZE_SYSTEM_PROMPT, user_prompt, temperature=0.2)
|
||||
result = _parse_llm_response(raw_response)
|
||||
if result is not None:
|
||||
return result
|
||||
|
||||
logger.warning("LLM returned unparseable response; falling back to service grouping")
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("LLM analysis failed: %s — falling back to service grouping", exc)
|
||||
|
||||
return _fallback_group_by_service(patterns)
|
||||
|
||||
|
||||
def find_duplicate_patterns(
|
||||
new_patterns: list[dict],
|
||||
existing_patterns: list[dict],
|
||||
) -> list[DuplicateGroup]:
|
||||
"""Identify near-duplicate patterns that the regex normalizer missed.
|
||||
|
||||
Returns an empty list if either input list is empty, or if the LLM call
|
||||
fails for any reason.
|
||||
|
||||
Args:
|
||||
new_patterns: Newly detected error patterns to check.
|
||||
existing_patterns: Currently tracked (canonical) error patterns.
|
||||
|
||||
Returns:
|
||||
List of DuplicateGroup objects (may be empty).
|
||||
"""
|
||||
if not new_patterns or not existing_patterns:
|
||||
return []
|
||||
|
||||
user_prompt = _build_dedup_user_prompt(new_patterns, existing_patterns)
|
||||
|
||||
try:
|
||||
raw_response = _call_llm(_DEDUP_SYSTEM_PROMPT, user_prompt, temperature=0.1)
|
||||
return _parse_duplicate_response(raw_response)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Duplicate detection LLM call failed: %s", exc)
|
||||
return []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prompt builders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_analyze_user_prompt(patterns: list[dict]) -> str:
|
||||
"""Build the user prompt listing patterns for incident grouping."""
|
||||
lines = ["Error patterns to analyse:\n"]
|
||||
for idx, pattern in enumerate(patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
count = pattern.get("count", 0)
|
||||
lines.append(f"[{idx}] service={service!r} count={count} message={message!r}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _build_dedup_user_prompt(
|
||||
new_patterns: list[dict],
|
||||
existing_patterns: list[dict],
|
||||
) -> str:
|
||||
"""Build the user prompt listing existing and new patterns for dedup."""
|
||||
lines = ["EXISTING (canonical) patterns:\n"]
|
||||
for idx, pattern in enumerate(existing_patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
lines.append(f"[{idx}] service={service!r} message={message!r}")
|
||||
|
||||
lines.append("\nNEW patterns to check:\n")
|
||||
for idx, pattern in enumerate(new_patterns):
|
||||
service = pattern.get("service", "unknown")
|
||||
message = pattern.get("normalized_message", "")
|
||||
lines.append(f"[{idx}] service={service!r} message={message!r}")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _parse_duplicate_response(text: str) -> list[DuplicateGroup]:
|
||||
"""Parse the LLM response for duplicate detection.
|
||||
|
||||
Args:
|
||||
text: Raw LLM output.
|
||||
|
||||
Returns:
|
||||
List of DuplicateGroup objects, or [] on any failure.
|
||||
"""
|
||||
if not text:
|
||||
return []
|
||||
|
||||
cleaned = _strip_markdown_code_block(text)
|
||||
|
||||
try:
|
||||
data = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
logger.warning("Dedup LLM returned non-JSON: %.200s", text)
|
||||
return []
|
||||
|
||||
if "duplicates" not in data:
|
||||
logger.warning("Dedup LLM response missing 'duplicates' key")
|
||||
return []
|
||||
|
||||
groups: list[DuplicateGroup] = []
|
||||
for raw in data["duplicates"]:
|
||||
try:
|
||||
groups.append(
|
||||
DuplicateGroup(
|
||||
canonical_index=int(raw["canonical_index"]),
|
||||
duplicate_indices=list(raw.get("duplicate_indices", [])),
|
||||
reason=raw.get("reason", ""),
|
||||
)
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Skipping malformed duplicate entry: %s — %s", raw, exc)
|
||||
|
||||
return groups
|
||||
@@ -1,673 +0,0 @@
|
||||
"""Monitor orchestrator for the production error monitoring service.
|
||||
|
||||
This is the main entry point. Run as::
|
||||
|
||||
python -m backend.services.error_monitor.monitor
|
||||
|
||||
The ``ErrorMonitor.run()`` method executes the full pipeline:
|
||||
|
||||
* Monitor cycle (default) — query Cloud Logging, detect new patterns, send
|
||||
Discord alerts, check auto-resolution.
|
||||
* Daily digest mode (``DIGEST_MODE=true``) — bundle 24h stats into one message.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from google.cloud import logging as cloud_logging
|
||||
from google.cloud import secretmanager
|
||||
|
||||
from backend.services.error_monitor.config import (
|
||||
GCP_PROJECT,
|
||||
LOOKBACK_MINUTES,
|
||||
MAX_LOG_ENTRIES,
|
||||
MONITORED_CLOUD_FUNCTIONS,
|
||||
MONITORED_CLOUD_RUN_JOBS,
|
||||
MONITORED_CLOUD_RUN_SERVICES,
|
||||
MONITORED_GCE_INSTANCES,
|
||||
SPIKE_MIN_COUNT,
|
||||
SPIKE_MULTIPLIER,
|
||||
get_digest_mode,
|
||||
get_discord_webhook_secret_name,
|
||||
get_llm_enabled,
|
||||
)
|
||||
from backend.services.error_monitor.discord import (
|
||||
ErrorMonitorDiscord,
|
||||
format_auto_resolved_alert,
|
||||
format_daily_digest,
|
||||
format_incident_alert,
|
||||
format_new_pattern_alert,
|
||||
format_spike_alert,
|
||||
)
|
||||
from backend.services.error_monitor.firestore_adapter import (
|
||||
ErrorPatternsAdapter,
|
||||
PatternData,
|
||||
UpsertResult,
|
||||
)
|
||||
from backend.services.error_monitor.known_issues import should_ignore
|
||||
from backend.services.error_monitor.llm_analysis import analyze_patterns, find_duplicate_patterns
|
||||
from backend.services.error_monitor.normalizer import compute_pattern_hash, normalize_message
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module-level constants
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: Maps log resource type → (label_key, service_name_list)
|
||||
RESOURCE_TYPE_FILTERS: dict[str, tuple[str, list[str]]] = {
|
||||
"cloud_run_revision": ("resource.labels.service_name", MONITORED_CLOUD_RUN_SERVICES),
|
||||
"cloud_run_job": ("resource.labels.job_name", MONITORED_CLOUD_RUN_JOBS),
|
||||
"cloud_function": ("resource.labels.function_name", MONITORED_CLOUD_FUNCTIONS),
|
||||
"gce_instance": ('labels."compute.googleapis.com/resource_name"', MONITORED_GCE_INSTANCES),
|
||||
}
|
||||
|
||||
#: Maps log resource type → canonical resource_type string stored in Firestore.
|
||||
RESOURCE_TYPE_MAP: dict[str, str] = {
|
||||
"cloud_run_revision": "cloud_run_service",
|
||||
"cloud_run_job": "cloud_run_job",
|
||||
"cloud_function": "cloud_function",
|
||||
"gce_instance": "gce_instance",
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Free functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _build_log_filter(
|
||||
resource_type: str,
|
||||
resource_names: list[str],
|
||||
lookback_minutes: int,
|
||||
) -> str:
|
||||
"""Build a Cloud Logging filter string.
|
||||
|
||||
Args:
|
||||
resource_type: GCP resource type (e.g. ``"cloud_run_revision"``).
|
||||
resource_names: List of service/job/function/instance names to include.
|
||||
lookback_minutes: How many minutes back to query.
|
||||
|
||||
Returns:
|
||||
A filter string suitable for ``logging_client.list_entries(filter_=...)``.
|
||||
"""
|
||||
label_key = RESOURCE_TYPE_FILTERS[resource_type][0]
|
||||
cutoff = datetime.now(tz=timezone.utc) - timedelta(minutes=lookback_minutes)
|
||||
cutoff_iso = cutoff.strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
|
||||
name_clause = " OR ".join(f'{label_key}="{name}"' for name in resource_names)
|
||||
if len(resource_names) > 1:
|
||||
name_clause = f"({name_clause})"
|
||||
|
||||
return (
|
||||
f'resource.type="{resource_type}" '
|
||||
f"severity>=ERROR "
|
||||
f'timestamp>="{cutoff_iso}" '
|
||||
f"{name_clause}"
|
||||
)
|
||||
|
||||
|
||||
def _extract_message(entry: Any) -> str | None:
|
||||
"""Extract the log message text from a Cloud Logging entry.
|
||||
|
||||
Handles several payload shapes:
|
||||
|
||||
* ``str`` payload — returned directly.
|
||||
* ``dict`` payload — tries ``"message"``, ``"textPayload"``, ``"error"``,
|
||||
``"msg"`` keys in that order.
|
||||
* ``None`` payload — returns ``None``.
|
||||
|
||||
Args:
|
||||
entry: A Cloud Logging ``LogEntry`` object.
|
||||
|
||||
Returns:
|
||||
The message string, or ``None`` if not extractable.
|
||||
"""
|
||||
payload = entry.payload
|
||||
if payload is None:
|
||||
return None
|
||||
if isinstance(payload, str):
|
||||
return payload if payload else None
|
||||
if isinstance(payload, dict):
|
||||
for key in ("message", "textPayload", "error", "msg"):
|
||||
value = payload.get(key)
|
||||
if value:
|
||||
return str(value)
|
||||
return None
|
||||
# Fallback: convert to string representation
|
||||
text = str(payload)
|
||||
return text if text else None
|
||||
|
||||
|
||||
def _extract_service_name(entry: Any, resource_type: str) -> str:
|
||||
"""Extract the service/job/function/instance name from a log entry.
|
||||
|
||||
Args:
|
||||
entry: A Cloud Logging ``LogEntry`` object.
|
||||
resource_type: GCP log resource type string.
|
||||
|
||||
Returns:
|
||||
The service name string (falls back to ``"unknown"`` if not found).
|
||||
"""
|
||||
labels = getattr(entry.resource, "labels", {}) or {}
|
||||
entry_labels = getattr(entry, "labels", {}) or {}
|
||||
|
||||
if resource_type == "cloud_run_revision":
|
||||
return labels.get("service_name") or "unknown"
|
||||
if resource_type == "cloud_run_job":
|
||||
return labels.get("job_name") or "unknown"
|
||||
if resource_type == "cloud_function":
|
||||
return labels.get("function_name") or "unknown"
|
||||
if resource_type == "gce_instance":
|
||||
# Prefer the human-readable resource name label
|
||||
name = entry_labels.get("compute.googleapis.com/resource_name")
|
||||
if name:
|
||||
return name
|
||||
return labels.get("instance_id") or "unknown"
|
||||
return "unknown"
|
||||
|
||||
|
||||
def _classify_resource_type(log_resource_type: str) -> str:
|
||||
"""Map a Cloud Logging resource type to the canonical type stored in Firestore.
|
||||
|
||||
Args:
|
||||
log_resource_type: Raw GCP resource type (e.g. ``"cloud_run_revision"``).
|
||||
|
||||
Returns:
|
||||
Canonical resource type string (falls back to the input if not in map).
|
||||
"""
|
||||
return RESOURCE_TYPE_MAP.get(log_resource_type, log_resource_type)
|
||||
|
||||
|
||||
def _group_by_pattern(entries: list[dict]) -> dict[str, dict]:
|
||||
"""Group normalised log entries by their pattern hash.
|
||||
|
||||
Args:
|
||||
entries: List of dicts with ``service``, ``normalized``, and ``message`` keys.
|
||||
|
||||
Returns:
|
||||
Dict mapping ``pattern_id`` → group dict containing ``service``,
|
||||
``resource_type``, ``normalized``, ``sample_message``, ``count``.
|
||||
"""
|
||||
groups: dict[str, dict] = {}
|
||||
for entry in entries:
|
||||
service = entry["service"]
|
||||
normalized = entry["normalized"]
|
||||
pattern_id = compute_pattern_hash(service, normalized)
|
||||
|
||||
if pattern_id not in groups:
|
||||
groups[pattern_id] = {
|
||||
"pattern_id": pattern_id,
|
||||
"service": service,
|
||||
"resource_type": entry.get("resource_type", "unknown"),
|
||||
"normalized": normalized,
|
||||
"sample_message": entry["message"],
|
||||
"count": 0,
|
||||
}
|
||||
groups[pattern_id]["count"] += 1
|
||||
return groups
|
||||
|
||||
|
||||
def _rolling_average(rolling_counts: list[dict]) -> float:
|
||||
"""Compute the mean count from a list of rolling_count dicts.
|
||||
|
||||
Args:
|
||||
rolling_counts: List of ``{"count": int, ...}`` dicts.
|
||||
|
||||
Returns:
|
||||
The arithmetic mean as a float, or ``0.0`` for an empty list.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return 0.0
|
||||
total = sum(entry.get("count", 0) for entry in rolling_counts)
|
||||
return total / len(rolling_counts)
|
||||
|
||||
|
||||
def _is_spike(current_count: int, rolling_counts: list[dict]) -> bool:
|
||||
"""Determine whether ``current_count`` represents an error rate spike.
|
||||
|
||||
A spike is detected when:
|
||||
- ``current_count`` >= ``SPIKE_MIN_COUNT`` (avoids noise from rare errors), AND
|
||||
- ``current_count`` > ``rolling_average * SPIKE_MULTIPLIER``
|
||||
|
||||
Args:
|
||||
current_count: Error count in the current monitoring window.
|
||||
rolling_counts: Historical rolling counts from Firestore.
|
||||
|
||||
Returns:
|
||||
``True`` if a spike is detected, ``False`` otherwise.
|
||||
"""
|
||||
if not rolling_counts:
|
||||
return False
|
||||
if current_count < SPIKE_MIN_COUNT:
|
||||
return False
|
||||
avg = _rolling_average(rolling_counts)
|
||||
if avg == 0.0:
|
||||
return False
|
||||
return current_count > avg * SPIKE_MULTIPLIER
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ErrorMonitor class
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class ErrorMonitor:
|
||||
"""Orchestrates the full production error monitoring pipeline.
|
||||
|
||||
On each run (typically every 15 minutes via Cloud Scheduler), it:
|
||||
|
||||
1. Queries Cloud Logging for recent errors across all monitored services.
|
||||
2. Groups errors into patterns using normalisation + hashing.
|
||||
3. Upserts patterns to Firestore, tracking new vs. recurrent.
|
||||
4. Optionally uses an LLM to deduplicate patterns and create incidents.
|
||||
5. Sends Discord alerts for new patterns, incidents, and spikes.
|
||||
6. Auto-resolves patterns that have been silent long enough.
|
||||
"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.logging_client = cloud_logging.Client(project=GCP_PROJECT)
|
||||
self.firestore_adapter = ErrorPatternsAdapter()
|
||||
self.discord = ErrorMonitorDiscord(webhook_url=self._get_discord_webhook())
|
||||
|
||||
def _get_discord_webhook(self) -> str:
|
||||
"""Retrieve the Discord webhook URL.
|
||||
|
||||
Checks ``DISCORD_WEBHOOK_URL`` environment variable first; falls back
|
||||
to Secret Manager.
|
||||
|
||||
Returns:
|
||||
Discord webhook URL string.
|
||||
"""
|
||||
url = os.environ.get("DISCORD_WEBHOOK_URL")
|
||||
if url:
|
||||
return url
|
||||
|
||||
secret_name = get_discord_webhook_secret_name()
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
secret_path = f"projects/{GCP_PROJECT}/secrets/{secret_name}/versions/latest"
|
||||
response = client.access_secret_version(name=secret_path)
|
||||
return response.payload.data.decode("utf-8").strip()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Main entry point
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def run(self) -> None:
|
||||
"""Execute the monitor pipeline (digest or cycle mode)."""
|
||||
if get_digest_mode():
|
||||
self._run_daily_digest()
|
||||
else:
|
||||
self._run_monitor_cycle()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Monitor cycle
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_monitor_cycle(self) -> None:
|
||||
"""Execute one monitor cycle: collect → group → upsert → alert → auto-resolve."""
|
||||
now = datetime.now(tz=timezone.utc)
|
||||
now_iso = now.isoformat()
|
||||
|
||||
# ── Step 1: Collect log entries ────────────────────────────────────
|
||||
raw_entries: list[dict] = []
|
||||
for resource_type, (_, resource_names) in RESOURCE_TYPE_FILTERS.items():
|
||||
log_filter = _build_log_filter(resource_type, resource_names, LOOKBACK_MINUTES)
|
||||
logger.info("Querying %s with filter: %s", resource_type, log_filter)
|
||||
try:
|
||||
entries = self.logging_client.list_entries(
|
||||
filter_=log_filter,
|
||||
max_results=MAX_LOG_ENTRIES,
|
||||
)
|
||||
for entry in entries:
|
||||
message = _extract_message(entry)
|
||||
if not message:
|
||||
continue
|
||||
service = _extract_service_name(entry, resource_type)
|
||||
ignore_reason = should_ignore(service, message)
|
||||
if ignore_reason:
|
||||
logger.debug(
|
||||
"Ignoring entry from %s (%s): %s",
|
||||
service,
|
||||
ignore_reason.pattern_name,
|
||||
ignore_reason.reason,
|
||||
)
|
||||
continue
|
||||
normalized = normalize_message(message)
|
||||
if not normalized:
|
||||
continue
|
||||
raw_entries.append(
|
||||
{
|
||||
"service": service,
|
||||
"resource_type": _classify_resource_type(resource_type),
|
||||
"normalized": normalized,
|
||||
"message": message,
|
||||
}
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.error("Failed to query %s logs: %s", resource_type, exc)
|
||||
|
||||
logger.info("Collected %d log entries after filtering", len(raw_entries))
|
||||
|
||||
# ── Step 2: Group by pattern ───────────────────────────────────────
|
||||
groups = _group_by_pattern(raw_entries)
|
||||
logger.info("Grouped into %d distinct patterns", len(groups))
|
||||
|
||||
# ── Step 3: Upsert to Firestore ────────────────────────────────────
|
||||
new_pattern_ids: list[str] = []
|
||||
spike_patterns: list[dict] = []
|
||||
|
||||
for pattern_id, group in groups.items():
|
||||
data = PatternData(
|
||||
pattern_id=pattern_id,
|
||||
service=group["service"],
|
||||
resource_type=group["resource_type"],
|
||||
normalized_message=group["normalized"],
|
||||
sample_message=group["sample_message"],
|
||||
count=group["count"],
|
||||
timestamp=now,
|
||||
)
|
||||
result: UpsertResult = self.firestore_adapter.upsert_pattern(data)
|
||||
if result.is_new:
|
||||
new_pattern_ids.append(pattern_id)
|
||||
logger.info("New pattern detected: %s (%s)", pattern_id, group["service"])
|
||||
|
||||
# Check for spike on existing patterns
|
||||
existing = self.firestore_adapter.get_pattern(pattern_id)
|
||||
if existing and not result.is_new:
|
||||
rolling = existing.get("rolling_counts") or []
|
||||
if _is_spike(group["count"], rolling):
|
||||
spike_patterns.append(
|
||||
{
|
||||
"service": group["service"],
|
||||
"normalized_message": group["normalized"],
|
||||
"current_count": group["count"],
|
||||
"rolling_average": _rolling_average(rolling),
|
||||
}
|
||||
)
|
||||
logger.info("Spike detected for pattern %s", pattern_id)
|
||||
|
||||
# ── Step 3b: Pick up out-of-band patterns (e.g. frontend crashes) ──
|
||||
# Patterns written via POST /api/client-errors land directly in
|
||||
# Firestore without going through the log-scraping pipeline, so they
|
||||
# won't be in new_pattern_ids / groups from Steps 1-3. Ensure any
|
||||
# pattern with status="new" and alerted_at=None still gets a Discord
|
||||
# alert on this cycle.
|
||||
try:
|
||||
unalerted = self.firestore_adapter.get_unalerted_new_patterns()
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.error("Failed to query unalerted patterns: %s", exc)
|
||||
unalerted = []
|
||||
|
||||
for pattern in unalerted:
|
||||
pid = pattern.get("pattern_id")
|
||||
if not pid or pid in groups:
|
||||
continue
|
||||
groups[pid] = {
|
||||
"service": pattern.get("service", "unknown"),
|
||||
"resource_type": pattern.get("resource_type", "unknown"),
|
||||
"normalized": pattern.get("normalized_message", ""),
|
||||
"sample_message": pattern.get("sample_message", ""),
|
||||
"count": pattern.get("total_count", 1),
|
||||
}
|
||||
if pid not in new_pattern_ids:
|
||||
new_pattern_ids.append(pid)
|
||||
logger.info(
|
||||
"Picked up out-of-band pattern %s (%s)",
|
||||
pid,
|
||||
pattern.get("service", "unknown"),
|
||||
)
|
||||
|
||||
# If nothing to alert on this cycle, go straight to auto-resolve.
|
||||
if not new_pattern_ids and not spike_patterns:
|
||||
logger.info("No new or unalerted patterns this cycle; checking auto-resolve.")
|
||||
self._check_auto_resolve()
|
||||
return
|
||||
|
||||
# ── Step 4: LLM dedup ─────────────────────────────────────────────
|
||||
if new_pattern_ids and get_llm_enabled():
|
||||
new_patterns_data = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in new_pattern_ids
|
||||
]
|
||||
existing_patterns = self.firestore_adapter.get_active_patterns()
|
||||
# Exclude newly added patterns from the canonical list
|
||||
canonical = [p for p in existing_patterns if p.get("pattern_id") not in new_pattern_ids]
|
||||
|
||||
if canonical:
|
||||
duplicate_groups = find_duplicate_patterns(new_patterns_data, canonical)
|
||||
for dup_group in duplicate_groups:
|
||||
# Merge each duplicate new pattern into the canonical
|
||||
canonical_pattern = canonical[dup_group.canonical_index]
|
||||
canonical_id = canonical_pattern.get("pattern_id", "")
|
||||
for dup_idx in dup_group.duplicate_indices:
|
||||
if dup_idx < len(new_pattern_ids):
|
||||
dup_id = new_pattern_ids[dup_idx]
|
||||
logger.info(
|
||||
"Merging duplicate pattern %s into %s: %s",
|
||||
dup_id,
|
||||
canonical_id,
|
||||
dup_group.reason,
|
||||
)
|
||||
self.firestore_adapter.merge_pattern(dup_id, canonical_id, dup_group.reason)
|
||||
# Remove from new_pattern_ids since it was merged
|
||||
new_pattern_ids = [pid for pid in new_pattern_ids if pid != dup_id]
|
||||
|
||||
# ── Step 5: LLM incident analysis ─────────────────────────────────
|
||||
incident_pattern_ids: set[str] = set()
|
||||
|
||||
if len(new_pattern_ids) >= 2 and get_llm_enabled():
|
||||
patterns_for_analysis = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in new_pattern_ids
|
||||
if pid in groups
|
||||
]
|
||||
analysis = analyze_patterns(patterns_for_analysis)
|
||||
|
||||
if analysis and analysis.incidents:
|
||||
for incident in analysis.incidents:
|
||||
incident_pids = [
|
||||
new_pattern_ids[idx]
|
||||
for idx in incident.pattern_indices
|
||||
if idx < len(new_pattern_ids)
|
||||
]
|
||||
if not incident_pids:
|
||||
continue
|
||||
|
||||
incident_id = self.firestore_adapter.create_incident(
|
||||
title=incident.title,
|
||||
root_cause=incident.root_cause or "",
|
||||
severity=incident.severity,
|
||||
suggested_fix=incident.suggested_fix or "",
|
||||
primary_service=incident.primary_service,
|
||||
pattern_ids=incident_pids,
|
||||
used_llm=analysis.used_llm,
|
||||
)
|
||||
logger.info(
|
||||
"Created incident %s (%s) with %d patterns",
|
||||
incident_id,
|
||||
incident.severity,
|
||||
len(incident_pids),
|
||||
)
|
||||
|
||||
# Build pattern list for the Discord alert
|
||||
pattern_dicts = [
|
||||
{
|
||||
"service": groups[pid]["service"],
|
||||
"normalized_message": groups[pid]["normalized"],
|
||||
"count": groups[pid]["count"],
|
||||
}
|
||||
for pid in incident_pids
|
||||
if pid in groups
|
||||
]
|
||||
msg = format_incident_alert(
|
||||
title=incident.title,
|
||||
severity=incident.severity,
|
||||
root_cause=incident.root_cause or "Unknown",
|
||||
suggested_fix=incident.suggested_fix or "Investigate logs",
|
||||
primary_service=incident.primary_service,
|
||||
patterns=pattern_dicts,
|
||||
)
|
||||
if self.discord.send_message(msg):
|
||||
for pid in incident_pids:
|
||||
try:
|
||||
self.firestore_adapter.update_pattern_alerted(pid)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Could not update alerted_at for %s: %s", pid, exc)
|
||||
|
||||
incident_pattern_ids.update(incident_pids)
|
||||
|
||||
# ── Step 6: Individual alerts for new patterns not in incidents ────
|
||||
for pattern_id in new_pattern_ids:
|
||||
if pattern_id in incident_pattern_ids:
|
||||
continue
|
||||
if pattern_id not in groups:
|
||||
continue
|
||||
group = groups[pattern_id]
|
||||
|
||||
# Fetch fresh copy to get first_seen timestamp
|
||||
existing = self.firestore_adapter.get_pattern(pattern_id)
|
||||
first_seen = existing.get("first_seen", now_iso) if existing else now_iso
|
||||
|
||||
msg = format_new_pattern_alert(
|
||||
service=group["service"],
|
||||
resource_type=group["resource_type"],
|
||||
normalized_message=group["normalized"],
|
||||
sample_message=group["sample_message"],
|
||||
count=group["count"],
|
||||
first_seen=first_seen,
|
||||
)
|
||||
if self.discord.send_message(msg):
|
||||
try:
|
||||
self.firestore_adapter.update_pattern_alerted(pattern_id)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Could not update alerted_at for %s: %s", pattern_id, exc)
|
||||
|
||||
# ── Step 7: Spike alerts ───────────────────────────────────────────
|
||||
for spike in spike_patterns:
|
||||
msg = format_spike_alert(
|
||||
service=spike["service"],
|
||||
normalized_message=spike["normalized_message"],
|
||||
current_count=spike["current_count"],
|
||||
rolling_average=spike["rolling_average"],
|
||||
)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
# ── Step 8: Auto-resolve check ─────────────────────────────────────
|
||||
self._check_auto_resolve()
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Auto-resolve helper
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _check_auto_resolve(self) -> None:
|
||||
"""Check all active patterns for auto-resolution and send a consolidated alert."""
|
||||
patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
|
||||
resolved: list[dict] = []
|
||||
|
||||
for pattern in patterns:
|
||||
pattern_id = pattern.get("pattern_id", "")
|
||||
hours_silent = self.firestore_adapter.check_auto_resolve(pattern)
|
||||
if hours_silent is not None:
|
||||
logger.info(
|
||||
"Auto-resolving pattern %s (silent %.1f hours)", pattern_id, hours_silent
|
||||
)
|
||||
self.firestore_adapter.auto_resolve_pattern(pattern_id, round(hours_silent, 1))
|
||||
resolved.append(
|
||||
{
|
||||
"service": pattern.get("service", "unknown"),
|
||||
"message": pattern.get("normalized_message", ""),
|
||||
"hours_silent": round(hours_silent, 1),
|
||||
}
|
||||
)
|
||||
|
||||
if resolved:
|
||||
msg = format_auto_resolved_alert(resolved)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
# Daily digest
|
||||
# ------------------------------------------------------------------
|
||||
|
||||
def _run_daily_digest(self) -> None:
|
||||
"""Gather 24h stats and send a daily digest to Discord."""
|
||||
# Optional dedup sweep (best-effort; LLM errors are non-fatal)
|
||||
if get_llm_enabled():
|
||||
try:
|
||||
active = self.firestore_adapter.get_active_patterns()
|
||||
if len(active) >= 2:
|
||||
mid = len(active) // 2
|
||||
duplicates = find_duplicate_patterns(active[:mid], active[mid:])
|
||||
for dup in duplicates:
|
||||
for idx in dup.duplicate_indices:
|
||||
target_half = active[mid:]
|
||||
source_half = active[:mid]
|
||||
if idx < len(target_half) and dup.canonical_index < len(source_half):
|
||||
self.firestore_adapter.merge_pattern(
|
||||
target_half[idx]["pattern_id"],
|
||||
source_half[dup.canonical_index]["pattern_id"],
|
||||
dup.reason,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001
|
||||
logger.warning("Digest dedup sweep failed (non-fatal): %s", exc)
|
||||
|
||||
# Gather stats from active patterns
|
||||
active_patterns = self.firestore_adapter.get_active_patterns()
|
||||
per_service_counts: dict[str, int] = {}
|
||||
new_count = 0
|
||||
total_errors = 0
|
||||
|
||||
for pattern in active_patterns:
|
||||
service = pattern.get("service", "unknown")
|
||||
count = pattern.get("total_count", 0)
|
||||
per_service_counts[service] = per_service_counts.get(service, 0) + count
|
||||
total_errors += count
|
||||
if pattern.get("status") == "new":
|
||||
new_count += 1
|
||||
|
||||
# Count recently auto-resolved patterns
|
||||
resolved_patterns = self.firestore_adapter.get_patterns_for_auto_resolve()
|
||||
resolved_count = sum(
|
||||
1
|
||||
for p in resolved_patterns
|
||||
if self.firestore_adapter.check_auto_resolve(p) is not None
|
||||
)
|
||||
|
||||
msg = format_daily_digest(
|
||||
total_errors=total_errors,
|
||||
new_patterns=new_count,
|
||||
resolved_patterns=resolved_count,
|
||||
active_patterns=len(active_patterns),
|
||||
per_service_counts=per_service_counts,
|
||||
)
|
||||
self.discord.send_message(msg)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Configure logging and run the error monitor."""
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
||||
)
|
||||
monitor = ErrorMonitor()
|
||||
monitor.run()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,137 +0,0 @@
|
||||
"""Message normalizer for the production error monitoring service.
|
||||
|
||||
Replaces variable parts of error messages (IDs, timestamps, paths, etc.)
|
||||
with stable placeholders so the same logical error always produces the same
|
||||
pattern hash, regardless of which specific resource or timestamp was involved.
|
||||
|
||||
Normalizer rules are applied in order — more specific patterns first.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import re
|
||||
|
||||
from backend.services.error_monitor.config import MAX_NORMALIZED_MESSAGE_LENGTH
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Compiled regex patterns (applied in order — more specific first)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_NORMALIZERS: list[tuple[re.Pattern, str]] = [
|
||||
# 1. GCS paths: gs://bucket/path/to/object → gs://<BUCKET>/<PATH>
|
||||
# Must come before Firestore and URL normalization.
|
||||
(
|
||||
re.compile(r"gs://([^/\s]+)/([^\s]+)"),
|
||||
r"gs://<BUCKET>/<PATH>",
|
||||
),
|
||||
# 1b. GCS bucket root (no object path): gs://bucket/ or gs://bucket
|
||||
# Uses a lookahead so the slash/end-of-string is not consumed, and uses
|
||||
# a character class that excludes '<' to avoid re-matching placeholders.
|
||||
(
|
||||
re.compile(r"gs://([^/<\s]+)(?=[/\s]|$)"),
|
||||
r"gs://<BUCKET>",
|
||||
),
|
||||
# 2. Firestore document paths (collection/doc/…/doc — at least 2 segments)
|
||||
# Negative lookbehind for '.' prevents matching inside domain names (URLs).
|
||||
# Negative lookbehind for '/' prevents matching inside already-replaced paths.
|
||||
(
|
||||
re.compile(
|
||||
r"(?<![./])\b([a-zA-Z_][a-zA-Z0-9_-]*/[a-zA-Z0-9_-]+(?:/[a-zA-Z0-9_-]+){1,})\b"
|
||||
),
|
||||
r"<DOC_PATH>",
|
||||
),
|
||||
# 3. UUIDs (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)
|
||||
(
|
||||
re.compile(
|
||||
r"\b[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\b"
|
||||
),
|
||||
"<ID>",
|
||||
),
|
||||
# 4. ISO 8601 timestamps (2024-01-15T10:30:45Z / …+05:30 / ….123Z)
|
||||
(
|
||||
re.compile(
|
||||
r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?(?:Z|[+-]\d{2}:\d{2})"
|
||||
),
|
||||
"<TS>",
|
||||
),
|
||||
# 5. Epoch timestamps: 10+ digit number optionally followed by fractional seconds
|
||||
(
|
||||
re.compile(r"\b\d{10,}(?:\.\d+)?\b"),
|
||||
"<EPOCH>",
|
||||
),
|
||||
# 6. Email addresses
|
||||
(
|
||||
re.compile(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b"),
|
||||
"<EMAIL>",
|
||||
),
|
||||
# 7. IP addresses (IPv4) — must come before large-number replacement
|
||||
(
|
||||
re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b"),
|
||||
"<IP>",
|
||||
),
|
||||
# 8. URLs — strip to domain only (https://host/path → https://host)
|
||||
# Handles http and https; preserves the scheme+domain for context.
|
||||
(
|
||||
re.compile(r"https?://([A-Za-z0-9.\-]+(:\d+)?)/[^\s]*"),
|
||||
r"https://\1",
|
||||
),
|
||||
# 9. Job IDs in paths (/jobs/<alphanumeric>)
|
||||
(
|
||||
re.compile(r"/jobs/[A-Za-z0-9]+"),
|
||||
"/jobs/<ID>",
|
||||
),
|
||||
# 10. Firebase UIDs (20–28 mixed-case alphanumeric characters)
|
||||
# Firebase Auth UIDs are typically 28 chars; allow 20-28 for flexibility.
|
||||
# Must contain both letters and digits (to avoid replacing plain words).
|
||||
(
|
||||
re.compile(r"\b(?=[A-Za-z0-9]{20,28}\b)(?=[A-Za-z0-9]*[A-Z])(?=[A-Za-z0-9]*[a-z])(?=[A-Za-z0-9]*\d)[A-Za-z0-9]{20,28}\b"),
|
||||
"<UID>",
|
||||
),
|
||||
# 11. Hex/alphanumeric IDs: 8+ chars containing both letters and digits
|
||||
# (e.g. git SHAs, Cloud Run revision IDs, job IDs)
|
||||
# Uses a lookahead to require at least one digit AND one letter.
|
||||
(
|
||||
re.compile(r"\b(?=[a-fA-F0-9]*[a-fA-F][a-fA-F0-9]*\d|[a-fA-F0-9]*\d[a-fA-F0-9]*[a-fA-F])[a-fA-F0-9]{8,}\b"),
|
||||
"<ID>",
|
||||
),
|
||||
# 12. Large numbers (4+ digits) — must come after epoch/IP/UUID replacements
|
||||
(
|
||||
re.compile(r"\b\d{4,}\b"),
|
||||
"<NUM>",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def normalize_message(message: str) -> str:
|
||||
"""Normalize a raw log message by replacing variable parts with placeholders.
|
||||
|
||||
Applies each normalizer rule in order (more specific patterns first) and
|
||||
truncates the result to MAX_NORMALIZED_MESSAGE_LENGTH characters.
|
||||
|
||||
Args:
|
||||
message: Raw log message string.
|
||||
|
||||
Returns:
|
||||
Normalized message string, at most MAX_NORMALIZED_MESSAGE_LENGTH chars.
|
||||
"""
|
||||
if not message:
|
||||
return ""
|
||||
|
||||
result = message
|
||||
for pattern, replacement in _NORMALIZERS:
|
||||
result = pattern.sub(replacement, result)
|
||||
|
||||
return result[:MAX_NORMALIZED_MESSAGE_LENGTH]
|
||||
|
||||
|
||||
def compute_pattern_hash(service: str, normalized_message: str) -> str:
|
||||
"""Compute a stable SHA-256 fingerprint for an error pattern.
|
||||
|
||||
Args:
|
||||
service: The originating service name (e.g. "karaoke-backend").
|
||||
normalized_message: The output of normalize_message().
|
||||
|
||||
Returns:
|
||||
64-character lowercase hex SHA-256 digest.
|
||||
"""
|
||||
payload = f"{service}:{normalized_message}"
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
@@ -10,8 +10,6 @@ from google.cloud.firestore_v1 import FieldFilter
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
from backend.models.review_session import ReviewSession
|
||||
from backend.models.audio_edit_session import AudioEditSession
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -77,20 +75,18 @@ class FirestoreService:
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
timeline_metadata: Optional[Dict[str, Any]] = None,
|
||||
**additional_fields
|
||||
) -> None:
|
||||
"""Update job status and add timeline event."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
|
||||
# Create timeline event
|
||||
timeline_event = TimelineEvent(
|
||||
status=status.value,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
progress=progress,
|
||||
message=message,
|
||||
metadata=timeline_metadata,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Prepare updates
|
||||
@@ -120,7 +116,6 @@ class FirestoreService:
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
@@ -133,7 +128,6 @@ class FirestoreService:
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
@@ -156,11 +150,6 @@ class FirestoreService:
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Filter by tenant_id (white-label portal scoping)
|
||||
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
|
||||
if tenant_id is not None:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
@@ -178,86 +167,6 @@ class FirestoreService:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
raise
|
||||
|
||||
# Fields needed by the dashboard summary view
|
||||
SUMMARY_FIELD_PATHS = [
|
||||
'job_id', 'status', 'progress', 'created_at', 'artist', 'title',
|
||||
'error_message', 'non_interactive', 'outputs_deleted_at', 'user_email', 'is_private',
|
||||
'url', 'filename', 'audio_search_artist', 'audio_search_title',
|
||||
'audio_source_type', 'source_name', 'source_id', 'target_file', 'download_url',
|
||||
'state_data.brand_code', 'state_data.youtube_url', 'state_data.dropbox_link',
|
||||
'state_data.audio_progress', 'state_data.lyrics_progress',
|
||||
'state_data.audio_complete', 'state_data.lyrics_complete',
|
||||
'state_data.backing_vocals_analysis', 'state_data.visibility_change_in_progress',
|
||||
'file_urls.finals', 'file_urls.videos', 'file_urls.packages',
|
||||
'processing_metadata',
|
||||
]
|
||||
|
||||
def list_jobs_summary(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
exclude_statuses: Optional[List[str]] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
List jobs with Firestore field projection for reduced payload.
|
||||
|
||||
Uses ``query.select()`` so Firestore only reads the requested fields,
|
||||
drastically cutting bandwidth and deserialization cost.
|
||||
|
||||
Returns raw dicts (not Job models) because projected documents would
|
||||
fail Pydantic validation for missing required fields.
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
# NOTE: exclude_statuses is filtered in Python after the query
|
||||
# because Firestore's not-in filter combined with order_by(created_at)
|
||||
# requires a composite index. Since documents are tiny after select(),
|
||||
# Python filtering is fast and avoids the index requirement.
|
||||
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Always filter: "" for consumer jobs, "vocalstar"/etc for tenant jobs
|
||||
if tenant_id is not None:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
query = query.select(self.SUMMARY_FIELD_PATHS)
|
||||
|
||||
docs = query.stream()
|
||||
results = [doc.to_dict() for doc in docs]
|
||||
|
||||
# Apply exclude_statuses filter in Python (see note above)
|
||||
if exclude_statuses:
|
||||
results = [r for r in results if r.get('status') not in exclude_statuses]
|
||||
|
||||
return results
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs summary: {e}")
|
||||
raise
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
@@ -601,476 +510,3 @@ class FirestoreService:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
# ============================================
|
||||
# Search Session Methods
|
||||
# ============================================
|
||||
|
||||
def create_search_session(self, session_data: dict) -> str:
|
||||
"""Store a search session for the guided job creation flow.
|
||||
|
||||
Sessions are short-lived (TTL enforced via Firestore TTL policy) and
|
||||
represent a completed audio search that hasn't yet been converted into
|
||||
a job. The document is consumed (deleted) when create-from-search runs.
|
||||
|
||||
Args:
|
||||
session_data: Dict with session_id, user_email, tenant_id, artist,
|
||||
title, results, remote_search_id, ttl_expiry.
|
||||
|
||||
Returns:
|
||||
session_id from session_data.
|
||||
"""
|
||||
try:
|
||||
session_id = session_data['session_id']
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc_ref.set(session_data)
|
||||
logger.info(f"Created search session {session_id}")
|
||||
return session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating search session: {e}")
|
||||
raise
|
||||
|
||||
def get_search_session(self, session_id: str) -> Optional[dict]:
|
||||
"""Retrieve a search session by ID.
|
||||
|
||||
Returns None only if the document does not exist.
|
||||
Raises on Firestore errors so callers can distinguish a missing session
|
||||
from a transient datastore failure.
|
||||
|
||||
TTL-expired documents may still appear briefly; callers should check
|
||||
ttl_expiry and treat expired sessions as missing.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
|
||||
Returns:
|
||||
Session dict or None.
|
||||
"""
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return doc.to_dict()
|
||||
|
||||
def consume_search_session(self, session_id: str) -> Optional[dict]:
|
||||
"""Atomically read and delete a search session.
|
||||
|
||||
Uses a Firestore transaction to prevent two concurrent job-creation
|
||||
requests from both reading the same session and creating duplicate jobs.
|
||||
|
||||
Returns the session dict if it existed, None if it did not exist.
|
||||
Raises on Firestore errors.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
|
||||
Returns:
|
||||
Session dict or None.
|
||||
"""
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
|
||||
@firestore.transactional
|
||||
def _read_and_delete(transaction, ref):
|
||||
doc = ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return None
|
||||
session_data = doc.to_dict()
|
||||
transaction.delete(ref)
|
||||
return session_data
|
||||
|
||||
transaction = self.db.transaction()
|
||||
return _read_and_delete(transaction, doc_ref)
|
||||
|
||||
# ============================================
|
||||
# Review Session Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store review session snapshots in a subcollection
|
||||
# (jobs/{job_id}/review_sessions) for lyrics review backup/restore.
|
||||
|
||||
def save_review_session(self, job_id: str, session: ReviewSession) -> str:
|
||||
"""
|
||||
Save a review session to the subcollection.
|
||||
|
||||
Stores at: jobs/{job_id}/review_sessions/{session_id}
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session: ReviewSession instance
|
||||
|
||||
Returns:
|
||||
session_id
|
||||
"""
|
||||
try:
|
||||
session.job_id = job_id
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
doc_ref = sessions_ref.document(session.session_id)
|
||||
doc_ref.set(session.to_dict())
|
||||
logger.info(f"Saved review session {session.session_id} for job {job_id}")
|
||||
return session.session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving review session for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_review_sessions(self, job_id: str) -> List[ReviewSession]:
|
||||
"""
|
||||
List all review sessions for a job, ordered by updated_at descending.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
List of ReviewSession instances (metadata only, no correction_data)
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
docs = query.stream()
|
||||
return [ReviewSession.from_dict(doc.to_dict()) for doc in docs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing review sessions for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_review_session(self, job_id: str, session_id: str) -> Optional[ReviewSession]:
|
||||
"""
|
||||
Get a single review session by ID.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session_id: Session ID
|
||||
|
||||
Returns:
|
||||
ReviewSession or None
|
||||
"""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("review_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return ReviewSession.from_dict(doc.to_dict())
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting review session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_review_session(self, job_id: str, session_id: str) -> None:
|
||||
"""
|
||||
Delete a review session.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
session_id: Session ID
|
||||
"""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("review_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted review session {session_id} for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting review session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_review_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all review sessions for a job (used when deleting a job).
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of sessions deleted
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = sessions_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} review sessions for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting review sessions subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def get_latest_review_session_hash(self, job_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get the data_hash of the most recent review session for deduplication.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
data_hash string or None if no sessions exist
|
||||
"""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = (
|
||||
sessions_ref
|
||||
.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.select(["data_hash"])
|
||||
)
|
||||
docs = list(query.stream())
|
||||
if docs:
|
||||
return docs[0].to_dict().get("data_hash", "")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting latest review session hash for job {job_id}: {e}")
|
||||
return None
|
||||
|
||||
def search_review_sessions(
|
||||
self,
|
||||
query_text: str = "",
|
||||
job_id: Optional[str] = None,
|
||||
limit: int = 20,
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Search review sessions across all jobs using collection group query.
|
||||
|
||||
Args:
|
||||
query_text: Search text (prefix-matched against artist_lower or title_lower)
|
||||
job_id: If provided, search only within this job
|
||||
limit: Maximum results
|
||||
|
||||
Returns:
|
||||
List of session dicts with job context
|
||||
"""
|
||||
try:
|
||||
if job_id:
|
||||
# Search within a specific job
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("review_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
|
||||
docs = query.stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
# Collection group query across all jobs
|
||||
sessions_ref = self.db.collection_group("review_sessions")
|
||||
|
||||
if query_text:
|
||||
# Over-fetch since we filter in Python (Firestore doesn't support LIKE/contains)
|
||||
fetch_limit = max(limit * 25, 500)
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(fetch_limit)
|
||||
docs = query.stream()
|
||||
results = [doc.to_dict() for doc in docs]
|
||||
|
||||
q_lower = query_text.lower()
|
||||
results = [
|
||||
r for r in results
|
||||
if q_lower in (r.get("artist") or "").lower()
|
||||
or q_lower in (r.get("title") or "").lower()
|
||||
or q_lower in (r.get("job_id") or "").lower()
|
||||
]
|
||||
return results[:limit]
|
||||
else:
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING).limit(limit)
|
||||
docs = query.stream()
|
||||
return [doc.to_dict() for doc in docs]
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error searching review sessions: {e}")
|
||||
return []
|
||||
|
||||
# ---- Audio Edit Sessions ----
|
||||
|
||||
def save_audio_edit_session(self, job_id: str, session: AudioEditSession) -> str:
|
||||
"""Save an audio edit session to the subcollection."""
|
||||
try:
|
||||
session.job_id = job_id
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
doc_ref = sessions_ref.document(session.session_id)
|
||||
doc_ref.set(session.to_dict())
|
||||
logger.info(f"Saved audio edit session {session.session_id} for job {job_id}")
|
||||
return session.session_id
|
||||
except Exception as e:
|
||||
logger.error(f"Error saving audio edit session for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_audio_edit_sessions(self, job_id: str) -> List[AudioEditSession]:
|
||||
"""List all audio edit sessions for a job, ordered by updated_at descending."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
query = sessions_ref.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
docs = query.stream()
|
||||
return [AudioEditSession.from_dict(doc.to_dict()) for doc in docs]
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing audio edit sessions for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_audio_edit_session(self, job_id: str, session_id: str) -> Optional[AudioEditSession]:
|
||||
"""Get a single audio edit session by ID."""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("audio_edit_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc = doc_ref.get()
|
||||
if not doc.exists:
|
||||
return None
|
||||
return AudioEditSession.from_dict(doc.to_dict())
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting audio edit session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_audio_edit_session(self, job_id: str, session_id: str) -> None:
|
||||
"""Delete a single audio edit session."""
|
||||
try:
|
||||
doc_ref = (
|
||||
self.db.collection(self.collection)
|
||||
.document(job_id)
|
||||
.collection("audio_edit_sessions")
|
||||
.document(session_id)
|
||||
)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted audio edit session {session_id} for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting audio edit session {session_id} for job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def delete_audio_edit_sessions_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""Delete all audio edit sessions for a job (used when deleting a job)."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
docs = sessions_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} audio edit sessions for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting audio edit sessions subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def get_latest_audio_edit_session_hash(self, job_id: str) -> Optional[str]:
|
||||
"""Get the data_hash of the most recent audio edit session for deduplication."""
|
||||
try:
|
||||
sessions_ref = self.db.collection(self.collection).document(job_id).collection("audio_edit_sessions")
|
||||
query = (
|
||||
sessions_ref
|
||||
.order_by("updated_at", direction=firestore.Query.DESCENDING)
|
||||
.limit(1)
|
||||
.select(["data_hash"])
|
||||
)
|
||||
docs = list(query.stream())
|
||||
if docs:
|
||||
return docs[0].to_dict().get("data_hash", "")
|
||||
return None
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting latest audio edit session hash for job {job_id}: {e}")
|
||||
return None
|
||||
|
||||
def delete_search_session(self, session_id: str) -> None:
|
||||
"""Delete a search session after it has been consumed by job creation.
|
||||
|
||||
Errors are logged but not re-raised — a missing or already-deleted
|
||||
session is a no-op.
|
||||
|
||||
Args:
|
||||
session_id: UUID string identifying the session.
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection('search_sessions').document(session_id)
|
||||
doc_ref.delete()
|
||||
logger.debug(f"Deleted search session {session_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting search session {session_id}: {e}")
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_firestore_client: Optional[firestore.Client] = None
|
||||
|
||||
|
||||
def get_firestore_client() -> firestore.Client:
|
||||
"""
|
||||
Get a shared Firestore client instance.
|
||||
|
||||
This returns a raw Firestore client (not the FirestoreService) for use
|
||||
in services that need direct Firestore access without the job-specific
|
||||
abstractions provided by FirestoreService.
|
||||
|
||||
Returns:
|
||||
Firestore client instance
|
||||
"""
|
||||
global _firestore_client
|
||||
if _firestore_client is None:
|
||||
_firestore_client = firestore.Client(project=settings.google_cloud_project)
|
||||
return _firestore_client
|
||||
|
||||
|
||||
_log_service: Optional["FirestoreService"] = None
|
||||
|
||||
|
||||
def _get_log_service() -> "FirestoreService":
|
||||
"""Return a cached FirestoreService instance for log_to_job."""
|
||||
global _log_service
|
||||
if _log_service is None:
|
||||
_log_service = FirestoreService()
|
||||
return _log_service
|
||||
|
||||
|
||||
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: Optional[Dict[str, Any]] = None) -> None:
|
||||
"""
|
||||
Write a log entry to a job's Firestore log subcollection.
|
||||
|
||||
This is a convenience function for API endpoints that need to log
|
||||
to a job's log history (workers already have JobLogAdapter for this).
|
||||
|
||||
Args:
|
||||
job_id: Job ID to log to
|
||||
worker: Worker/source name (e.g., "api", "edit", "admin")
|
||||
level: Log level (INFO, WARNING, ERROR)
|
||||
message: Log message
|
||||
metadata: Optional structured metadata dict
|
||||
"""
|
||||
try:
|
||||
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
|
||||
_get_log_service().append_log_to_subcollection(job_id, entry)
|
||||
except Exception as e:
|
||||
logger.debug(f"Failed to write job log for {job_id}: {e}")
|
||||
|
||||
@@ -34,8 +34,6 @@ from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@@ -44,104 +42,6 @@ class FlacfetchServiceError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _format_httpx_error(e: Exception) -> str:
|
||||
"""
|
||||
Format an httpx exception into a descriptive error string.
|
||||
|
||||
httpx.RequestError subclasses (ConnectError, ReadError, etc.) often have
|
||||
empty str() representations, which produces useless messages like
|
||||
"Status request failed: ". This helper always returns something meaningful.
|
||||
"""
|
||||
error_str = str(e)
|
||||
error_type = type(e).__name__
|
||||
|
||||
if error_str:
|
||||
return f"{error_type}: {error_str}"
|
||||
|
||||
# str(e) is empty - try to include request URL for context
|
||||
if isinstance(e, httpx.RequestError):
|
||||
try:
|
||||
request = e.request
|
||||
if request:
|
||||
return f"{error_type} for {request.method} {request.url}"
|
||||
except RuntimeError:
|
||||
pass # .request not set
|
||||
return error_type
|
||||
|
||||
|
||||
def should_retry_flacfetch_error(exception: Exception) -> bool:
|
||||
"""
|
||||
Determine if a flacfetch exception should be retried.
|
||||
|
||||
Retry on:
|
||||
- httpx.RequestError (network failures: timeout, connection refused, etc.)
|
||||
- httpx.HTTPStatusError with 5xx status (server errors)
|
||||
|
||||
Don't retry on:
|
||||
- httpx.HTTPStatusError with 4xx status (client errors: bad request, auth, not found)
|
||||
"""
|
||||
if isinstance(exception, httpx.RequestError):
|
||||
# Network-level failures - always retry
|
||||
return True
|
||||
|
||||
if isinstance(exception, httpx.HTTPStatusError):
|
||||
# HTTP errors - only retry on 5xx (server errors)
|
||||
return exception.response.status_code >= 500
|
||||
|
||||
# Don't retry on other exceptions (e.g., FlacfetchServiceError after wrapping)
|
||||
return False
|
||||
|
||||
|
||||
async def with_retry(func, *args, **kwargs):
|
||||
"""
|
||||
Execute an async function with retry logic using exponential backoff.
|
||||
|
||||
This implementation uses manual retry logic instead of tenacity decorators
|
||||
for better testability and runtime configuration.
|
||||
"""
|
||||
settings = get_settings()
|
||||
max_attempts = settings.flacfetch_retry_max_attempts
|
||||
min_wait = settings.flacfetch_retry_min_wait
|
||||
max_wait = settings.flacfetch_retry_max_wait
|
||||
|
||||
last_exception = None
|
||||
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
try:
|
||||
return await func(*args, **kwargs)
|
||||
except Exception as e:
|
||||
last_exception = e
|
||||
|
||||
# Check if we should retry this exception
|
||||
if not should_retry_flacfetch_error(e):
|
||||
# Don't retry - raise immediately
|
||||
raise
|
||||
|
||||
# Check if we have more attempts left
|
||||
if attempt >= max_attempts:
|
||||
# No more attempts - raise the last exception
|
||||
logger.warning(
|
||||
f"Flacfetch request failed after {max_attempts} attempts: "
|
||||
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}"
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate wait time with exponential backoff
|
||||
wait_time = min(min_wait * (2 ** (attempt - 1)), max_wait)
|
||||
|
||||
logger.warning(
|
||||
f"Flacfetch request failed (attempt {attempt}/{max_attempts}): "
|
||||
f"{_format_httpx_error(e) if isinstance(e, httpx.RequestError) else e}. "
|
||||
f"Retrying in {wait_time:.1f}s..."
|
||||
)
|
||||
|
||||
# Wait before next attempt
|
||||
await asyncio.sleep(wait_time)
|
||||
|
||||
# Should never reach here, but just in case
|
||||
raise last_exception
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
@@ -200,7 +100,7 @@ class FlacfetchClient:
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
@@ -245,10 +145,10 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
@@ -258,59 +158,47 @@ class FlacfetchClient:
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
Retries automatically on network errors and server errors (5xx) using
|
||||
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
|
||||
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure (after all retries exhausted)
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
return await with_retry(self._download_impl, search_id, result_index, output_filename, gcs_path)
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def _download_impl(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Implementation of download without error wrapping - lets httpx exceptions bubble up."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
@@ -326,9 +214,6 @@ class FlacfetchClient:
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Retries automatically on network errors and server errors (5xx) using
|
||||
exponential backoff. Configured via FLACFETCH_RETRY_* environment variables.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
@@ -341,68 +226,52 @@ class FlacfetchClient:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure (after all retries exhausted)
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
return await with_retry(
|
||||
self._download_by_id_impl,
|
||||
source_name, source_id, output_filename, target_file, download_url, gcs_path
|
||||
)
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def _download_by_id_impl(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Implementation of download_by_id without error wrapping - lets httpx exceptions bubble up."""
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
|
||||
Raises:
|
||||
httpx.RequestError: On network-level failures (connection refused, timeout, etc.)
|
||||
These are NOT wrapped so wait_for_download can handle retries.
|
||||
FlacfetchServiceError: On application-level failures (404, other HTTP errors)
|
||||
FlacfetchServiceError: On status check failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
@@ -413,10 +282,9 @@ class FlacfetchClient:
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError:
|
||||
# Let network errors bubble up so wait_for_download can handle retries
|
||||
raise
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
@@ -428,59 +296,35 @@ class FlacfetchClient:
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
max_consecutive_errors: int = 3,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
Tolerates transient network errors during status polling. A single
|
||||
connection hiccup won't fail the entire download -- only N consecutive
|
||||
poll failures will.
|
||||
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
max_consecutive_errors: Number of consecutive poll failures before giving up
|
||||
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure, timeout, or too many consecutive poll errors
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
"""
|
||||
elapsed = 0.0
|
||||
consecutive_errors = 0
|
||||
|
||||
|
||||
while elapsed < timeout:
|
||||
try:
|
||||
status = await self.get_download_status(download_id)
|
||||
consecutive_errors = 0 # Reset on success
|
||||
except httpx.RequestError as e:
|
||||
consecutive_errors += 1
|
||||
error_desc = _format_httpx_error(e)
|
||||
logger.warning(
|
||||
f"Status poll failed for download {download_id} "
|
||||
f"({consecutive_errors}/{max_consecutive_errors}): {error_desc}"
|
||||
)
|
||||
if consecutive_errors >= max_consecutive_errors:
|
||||
raise FlacfetchServiceError(
|
||||
f"Status polling failed after {consecutive_errors} consecutive "
|
||||
f"network errors (last: {error_desc})"
|
||||
)
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
continue
|
||||
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
@@ -489,7 +333,7 @@ class FlacfetchClient:
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
@@ -498,10 +342,10 @@ class FlacfetchClient:
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
@@ -522,7 +366,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
@@ -553,7 +397,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
@@ -587,7 +431,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
@@ -628,7 +472,7 @@ class FlacfetchClient:
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
@@ -658,55 +502,12 @@ class FlacfetchClient:
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# YouTube Availability Check
|
||||
# =========================================================================
|
||||
|
||||
async def check_youtube(self, url: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if a YouTube video is available for download from this server's location.
|
||||
|
||||
Calls the flacfetch /check-youtube endpoint which uses yt-dlp to detect
|
||||
geo-restrictions, private/removed videos, and other unavailability reasons.
|
||||
|
||||
Args:
|
||||
url: YouTube video URL or video ID
|
||||
|
||||
Returns:
|
||||
Dict with keys: available, video_id, title, error,
|
||||
is_geo_restricted, is_age_restricted, is_private, is_removed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/check-youtube",
|
||||
headers=self._headers(),
|
||||
json={"url": url},
|
||||
timeout=15,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"YouTube availability check failed: {_format_httpx_error(e)}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"YouTube availability check failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
@@ -728,7 +529,7 @@ class FlacfetchClient:
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {_format_httpx_error(e)}")
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
|
||||
@@ -1,141 +0,0 @@
|
||||
"""Service for generating personalized referral flyer PDFs."""
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "..", "templates", "printables")
|
||||
|
||||
CHROMIUM_PATHS = [
|
||||
"/usr/bin/chromium",
|
||||
"/usr/bin/chromium-browser",
|
||||
"/usr/bin/google-chrome",
|
||||
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
|
||||
]
|
||||
|
||||
VALID_THEMES = {"light", "dark"}
|
||||
MAX_QR_DATA_URL_BYTES = 500_000
|
||||
|
||||
|
||||
class FlyerError(Exception):
|
||||
"""Error during flyer generation."""
|
||||
|
||||
|
||||
class FlyerService:
|
||||
"""Generates personalized referral flyer PDFs."""
|
||||
|
||||
def __init__(self):
|
||||
self._chromium_path: str | None = None
|
||||
|
||||
def _get_chromium(self) -> str:
|
||||
"""Lazily find Chromium binary. Only needed when generating PDFs."""
|
||||
if self._chromium_path:
|
||||
return self._chromium_path
|
||||
for path in CHROMIUM_PATHS:
|
||||
if os.path.exists(path):
|
||||
self._chromium_path = path
|
||||
return path
|
||||
for name in ["chromium", "chromium-browser", "google-chrome"]:
|
||||
found = shutil.which(name)
|
||||
if found:
|
||||
self._chromium_path = found
|
||||
return found
|
||||
raise FlyerError("Chromium not found. Install chromium to generate flyer PDFs.")
|
||||
|
||||
def _validate_qr_data_url(self, qr_data_url: str) -> None:
|
||||
if not qr_data_url.startswith("data:image/"):
|
||||
raise FlyerError("Invalid QR data URL: must be a data:image/ URL")
|
||||
if len(qr_data_url) > MAX_QR_DATA_URL_BYTES:
|
||||
raise FlyerError(f"QR data URL too large (>{MAX_QR_DATA_URL_BYTES} bytes)")
|
||||
|
||||
def _render_template(
|
||||
self,
|
||||
theme: str,
|
||||
referral_code: str,
|
||||
discount_percent: int,
|
||||
qr_data_url: str,
|
||||
) -> str:
|
||||
if theme not in VALID_THEMES:
|
||||
raise FlyerError(f"Invalid theme: {theme}. Must be one of: {VALID_THEMES}")
|
||||
|
||||
suffix = "-dark" if theme == "dark" else ""
|
||||
template_path = os.path.join(TEMPLATE_DIR, f"website-referral-flyer{suffix}.html")
|
||||
|
||||
with open(template_path) as f:
|
||||
html = f.read()
|
||||
|
||||
html = html.replace("{{REFERRAL_CODE}}", referral_code.upper())
|
||||
html = html.replace("{{REFERRAL_CODE_LOWER}}", referral_code.lower())
|
||||
html = html.replace("{{DISCOUNT_PERCENT}}", str(discount_percent))
|
||||
html = html.replace("{{QR_DATA_URL}}", qr_data_url)
|
||||
|
||||
return html
|
||||
|
||||
def generate_pdf(
|
||||
self,
|
||||
theme: str,
|
||||
referral_code: str,
|
||||
discount_percent: int,
|
||||
qr_data_url: str,
|
||||
) -> bytes:
|
||||
"""Generate a personalized flyer PDF. Returns raw PDF bytes."""
|
||||
self._validate_qr_data_url(qr_data_url)
|
||||
html = self._render_template(theme, referral_code, discount_percent, qr_data_url)
|
||||
|
||||
with tempfile.NamedTemporaryFile(
|
||||
suffix=".html", delete=False, mode="w", encoding="utf-8"
|
||||
) as tmp_html:
|
||||
tmp_html.write(html)
|
||||
tmp_html_path = tmp_html.name
|
||||
|
||||
tmp_pdf_path = tmp_html_path.replace(".html", ".pdf")
|
||||
|
||||
try:
|
||||
cmd = [
|
||||
self._get_chromium(),
|
||||
"--headless",
|
||||
f"--print-to-pdf={tmp_pdf_path}",
|
||||
"--no-margins",
|
||||
"--print-background",
|
||||
"--no-pdf-header-footer",
|
||||
"--virtual-time-budget=5000",
|
||||
"--paper-width=8.5",
|
||||
"--paper-height=11",
|
||||
"--disable-gpu",
|
||||
"--no-sandbox",
|
||||
tmp_html_path,
|
||||
]
|
||||
|
||||
result = subprocess.run(
|
||||
cmd, capture_output=True, text=True, timeout=30
|
||||
)
|
||||
|
||||
if result.returncode != 0:
|
||||
logger.error("Chromium PDF generation failed: %s", result.stderr)
|
||||
raise FlyerError(f"PDF generation failed: {result.stderr[:200]}")
|
||||
|
||||
if not os.path.exists(tmp_pdf_path):
|
||||
raise FlyerError("PDF generation failed: no output file created")
|
||||
|
||||
with open(tmp_pdf_path, "rb") as f:
|
||||
return f.read()
|
||||
|
||||
finally:
|
||||
for path in [tmp_html_path, tmp_pdf_path]:
|
||||
try:
|
||||
os.unlink(path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
_flyer_service = None
|
||||
|
||||
|
||||
def get_flyer_service() -> FlyerService:
|
||||
global _flyer_service
|
||||
if _flyer_service is None:
|
||||
_flyer_service = FlyerService()
|
||||
return _flyer_service
|
||||
@@ -1,72 +0,0 @@
|
||||
# GCE Encoding Worker
|
||||
|
||||
HTTP API service that runs on the GCE encoding worker VM for video encoding jobs.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides HTTP endpoints for submitting and monitoring video encoding jobs. It runs as a systemd service on the `encoding-worker` GCE VM instance.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/encode` | POST | Submit a full encoding job |
|
||||
| `/encode-preview` | POST | Submit a preview encoding job (480x270, fast) |
|
||||
| `/status/{job_id}` | GET | Get job status and progress |
|
||||
| `/health` | GET | Health check |
|
||||
|
||||
## Authentication
|
||||
|
||||
All endpoints (except `/health`) require an API key via the `X-API-Key` header. The key is stored in Secret Manager (`encoding-worker-api-key`).
|
||||
|
||||
## Encoding Process
|
||||
|
||||
### Full Encoding (`/encode`)
|
||||
1. Downloads input files from GCS (title screen, karaoke video, end screen, instrumental audio)
|
||||
2. Uses `LocalEncodingService` from the karaoke-gen wheel to produce:
|
||||
- Lossless 4K MP4
|
||||
- Lossy 4K MP4
|
||||
- Lossless MKV
|
||||
- 720p MP4
|
||||
3. Uploads results to GCS
|
||||
4. Reports progress via status endpoint
|
||||
|
||||
### Preview Encoding (`/encode-preview`)
|
||||
1. Downloads ASS subtitle and audio files from GCS
|
||||
2. Runs FFmpeg to create a quick 480x270 preview video
|
||||
3. Uploads result to GCS
|
||||
4. Used for real-time preview in the web UI
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **FFmpeg**: Static build from John Van Sickle
|
||||
- **Python 3.13**: Built from source (required for karaoke-gen)
|
||||
- **karaoke-gen wheel**: Downloaded from GCS at job start (enables hot updates)
|
||||
|
||||
## Deployment
|
||||
|
||||
This service is deployed via the VM's startup script. The script:
|
||||
1. Installs system dependencies (FFmpeg, fonts, Python build deps)
|
||||
2. Builds Python 3.13 from source
|
||||
3. Creates venv and installs dependencies
|
||||
4. Writes `main.py` to `/opt/encoding-worker/`
|
||||
5. Creates systemd service
|
||||
6. Starts the service on port 8080
|
||||
|
||||
## Local Development
|
||||
|
||||
This code is extracted from the infrastructure for maintainability. To test locally:
|
||||
|
||||
```bash
|
||||
cd backend/services/gce_encoding
|
||||
pip install -r requirements.txt
|
||||
pip install ../../../ # Install karaoke-gen wheel
|
||||
ENCODING_API_KEY=test python main.py
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- **Hot code updates**: The karaoke-gen wheel is re-downloaded at the start of each job, allowing code updates without VM restart
|
||||
- **In-memory job tracking**: Job state is stored in memory. Restart clears queue.
|
||||
- **Parallel encoding**: ThreadPoolExecutor with 4 workers
|
||||
- **GCS integration**: Direct download/upload via google-cloud-storage client
|
||||
@@ -1,22 +0,0 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This package contains the FastAPI application that runs on the GCE encoding worker VM.
|
||||
It provides HTTP endpoints for video encoding jobs.
|
||||
|
||||
The service:
|
||||
- Downloads input files from GCS
|
||||
- Runs FFmpeg encoding via LocalEncodingService
|
||||
- Uploads results back to GCS
|
||||
- Provides job status tracking
|
||||
|
||||
Endpoints:
|
||||
- POST /encode - Submit a full encoding job
|
||||
- POST /encode-preview - Submit a preview encoding job (fast, low-res)
|
||||
- GET /status/{job_id} - Get job status
|
||||
- GET /health - Health check
|
||||
"""
|
||||
|
||||
from .main import app
|
||||
|
||||
__all__ = ["app"]
|
||||
Loaded 100 of 5334 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user