mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-21 21:38:07 +09:00
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1245e20fe4 |
No files matched your search
@@ -1,109 +0,0 @@
|
||||
# Plan: Admin Feedback Visibility
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1454-feedback-email-debug
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
User feedback submissions are stored in Firestore (`user_feedback` collection) but are completely invisible to the admin. The admin panel only shows a boolean "Feedback Submitted: Yes/No" badge per user. This plan adds:
|
||||
|
||||
1. **Admin UI page** to browse and review all submitted feedback
|
||||
2. **Email notification** to admin@nomadkaraoke.com on each new submission
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Admin page at `/admin/feedback` listing all feedback with ratings, text, timestamps, and user info
|
||||
- [ ] Sortable by date (newest first by default) and filterable by search (email)
|
||||
- [ ] Star ratings rendered visually (not just numbers)
|
||||
- [ ] Email sent to admin@nomadkaraoke.com immediately when feedback is submitted
|
||||
- [ ] Email includes all feedback content (ratings, text fields, user email)
|
||||
- [ ] Test data filtering works (exclude_test toggle)
|
||||
- [ ] Sidebar navigation includes new "Feedback" item
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Admin API Endpoint
|
||||
Add `GET /api/admin/feedback` to `backend/api/routes/admin.py`. Follows the standard admin endpoint pattern: `require_admin` dependency, pagination with `limit`/`offset`/`search`/`exclude_test`, returns Pydantic response model. Queries `user_feedback` Firestore collection.
|
||||
|
||||
### Admin UI Page
|
||||
New page at `frontend/app/admin/feedback/page.tsx`. Follows existing admin page patterns (Card, Table, Badge, pagination, search input, loading states). Uses `adminApi.listFeedback()` to fetch data.
|
||||
|
||||
### Email Notification
|
||||
Add a new method to `EmailService` (or call `send_email` directly) from the `submit_user_feedback` endpoint in `users.py`. Uses the existing SendGrid infrastructure — no new template service templates needed since this is an internal admin notification (simple HTML table with the feedback data).
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Admin feedback list endpoint
|
||||
|
||||
1. [ ] Add `AdminFeedbackItem` and `AdminFeedbackListResponse` Pydantic models to `admin.py`
|
||||
2. [ ] Add `GET /api/admin/feedback` endpoint with:
|
||||
- Pagination: `limit=50`, `offset=0`
|
||||
- Search by user email
|
||||
- `exclude_test` filter (reuse `is_test_email`)
|
||||
- Sort by `created_at` descending
|
||||
- Return: items, total, offset, limit, has_more
|
||||
3. [ ] Add aggregate stats in response: average ratings, total count
|
||||
|
||||
### Step 2: Backend — Email notification on submission
|
||||
|
||||
4. [ ] Add `send_feedback_notification` method to `EmailService` that sends an HTML email to `admin@nomadkaraoke.com` containing:
|
||||
- User email
|
||||
- All 4 star ratings
|
||||
- All text fields
|
||||
- Would recommend / would use again
|
||||
- Submitted timestamp
|
||||
- Link to admin feedback page
|
||||
5. [ ] Call `send_feedback_notification` from `submit_user_feedback` in `users.py` (fire-and-forget, don't block the user response on email success)
|
||||
|
||||
### Step 3: Frontend — Admin API client
|
||||
|
||||
6. [ ] Add `listFeedback` method to `adminApi` in `frontend/lib/api.ts`
|
||||
7. [ ] Add TypeScript types for `AdminFeedbackItem` and `AdminFeedbackListResponse`
|
||||
|
||||
### Step 4: Frontend — Admin feedback page
|
||||
|
||||
8. [ ] Create `frontend/app/admin/feedback/page.tsx` with:
|
||||
- Table: user email, date, overall rating (stars), ease of use, lyrics accuracy, correction experience, would recommend, would use again
|
||||
- Expandable rows or click-to-detail for text fields (what went well, what could improve, additional comments)
|
||||
- Search by email
|
||||
- Pagination (50 per page)
|
||||
- Refresh button
|
||||
- Test data toggle (via `useAdminSettings`)
|
||||
9. [ ] Add "Feedback" item to admin sidebar in `frontend/components/admin/admin-sidebar.tsx` with `MessageSquare` icon
|
||||
|
||||
### Step 5: Testing
|
||||
|
||||
10. [ ] Backend unit test for `GET /api/admin/feedback` endpoint (pagination, search, exclude_test)
|
||||
11. [ ] Backend unit test for `send_feedback_notification` email method
|
||||
12. [ ] Frontend component test for feedback page (render, empty state)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add feedback list endpoint + response models |
|
||||
| `backend/api/routes/users.py` | Modify | Add email notification call in `submit_user_feedback` |
|
||||
| `backend/services/email_service.py` | Modify | Add `send_feedback_notification` method |
|
||||
| `frontend/lib/api.ts` | Modify | Add `adminApi.listFeedback()` method + types |
|
||||
| `frontend/app/admin/feedback/page.tsx` | Create | Admin feedback review page |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add "Feedback" nav item |
|
||||
| `backend/tests/test_admin_feedback.py` | Create | Tests for admin feedback endpoint |
|
||||
| `backend/tests/test_feedback_email.py` | Create | Tests for feedback email notification |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Admin endpoint (pagination, filtering, sorting), email method (content formatting, send call)
|
||||
- **Integration**: Submit feedback → verify email service called
|
||||
- **Manual verification**: Check email arrives in admin@nomadkaraoke.com inbox after submission
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should the admin page show feedback from the old `beta_feedback` collection too, or just `user_feedback`? (Recommendation: just `user_feedback` — the beta collection is legacy)
|
||||
- [ ] Should there be any spam/quality detection flagging? (Some submissions appear to game the system for credits)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Backend: Remove endpoint from admin.py, revert email call in users.py
|
||||
- Frontend: Delete feedback page, remove sidebar item
|
||||
- No data migration needed — Firestore collection unchanged
|
||||
@@ -1,393 +0,0 @@
|
||||
# Plan: Admin Job Details Page
|
||||
|
||||
**Created:** 2026-01-09
|
||||
**Branch:** feat/sess-20260109-1738-admin-job-details
|
||||
**Status:** In Progress (Phase 1 Complete)
|
||||
|
||||
## Overview
|
||||
|
||||
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
|
||||
|
||||
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
|
||||
|
||||
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Must Have
|
||||
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
|
||||
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
|
||||
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
|
||||
- [ ] Download links for all existing files (generate signed URLs)
|
||||
- [ ] View worker logs with filtering by level/worker
|
||||
- [ ] Restart job from beginning (reset to PENDING)
|
||||
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
|
||||
|
||||
### Should Have
|
||||
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
|
||||
- [ ] View/edit raw JSON fields (state_data, file_urls)
|
||||
- [ ] Quick actions: send completion email, clear cache, impersonate user
|
||||
|
||||
### Nice to Have
|
||||
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
|
||||
- [ ] Diff view when editing fields
|
||||
- [ ] Audit log of admin changes
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Routing Fix (COMPLETED)
|
||||
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
|
||||
|
||||
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
|
||||
- Uses `useSearchParams()` to detect `?id=` parameter
|
||||
- Shows detail view when `selectedJobId` is present, list view otherwise
|
||||
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
|
||||
- Deleted old `/admin/jobs/detail/` directory
|
||||
|
||||
### Backend Enhancements
|
||||
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
|
||||
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
|
||||
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
|
||||
|
||||
### Frontend Structure
|
||||
The page will have collapsible sections:
|
||||
1. **Header**: Job ID, status badge, artist/title, quick actions
|
||||
2. **Overview Cards**: User, dates, progress, source info
|
||||
3. **Stage Timeline**: Visual breakdown of each stage with times
|
||||
4. **Files**: Downloadable links for all GCS files
|
||||
5. **Logs**: Filterable worker logs
|
||||
6. **Configuration**: Editable job settings
|
||||
7. **Raw Data**: JSON views of state_data, file_urls, full job
|
||||
|
||||
### Job Reset Logic
|
||||
Reset to specific states requires:
|
||||
- Clearing downstream state_data keys
|
||||
- Resetting status
|
||||
- Optionally clearing associated files
|
||||
- Adding timeline event for the reset
|
||||
|
||||
Valid reset targets:
|
||||
| Target State | Clears | Use Case |
|
||||
|--------------|--------|----------|
|
||||
| PENDING | Everything except input | Full restart |
|
||||
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
|
||||
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
|
||||
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
|
||||
|
||||
### Phase 1: Fix Routing (COMPLETED)
|
||||
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
|
||||
2. [x] Combined list and detail views in single component
|
||||
3. [x] Use `useSearchParams()` to detect `?id=` parameter
|
||||
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
|
||||
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
|
||||
6. [x] Verified: All 9 E2E tests pass, build succeeds
|
||||
|
||||
### Phase 2: Backend - File Downloads with Signed URLs
|
||||
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
|
||||
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
|
||||
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
|
||||
10. [ ] Traverse job.file_urls recursively
|
||||
11. [ ] Generate signed URLs for each GCS path
|
||||
12. [ ] Return structured response with file metadata
|
||||
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
|
||||
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
|
||||
|
||||
### Phase 3: Frontend - Enhanced File Display
|
||||
15. [ ] Create "Files" accordion section with grouped downloads
|
||||
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
|
||||
17. [ ] Show file name, type, download button
|
||||
18. [ ] Handle missing/unavailable files gracefully
|
||||
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
|
||||
20. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 4: Backend - Job Update Endpoint
|
||||
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
|
||||
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
|
||||
23. [ ] Define editable fields whitelist
|
||||
24. [ ] Validate field values (e.g., status must be valid enum)
|
||||
25. [ ] Log admin changes for audit trail
|
||||
26. [ ] Verify: Backend tests pass
|
||||
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
|
||||
|
||||
### Phase 5: Frontend - Editable Fields
|
||||
28. [ ] Create "Configuration" section with editable fields
|
||||
29. [ ] Use inline edit pattern (click to edit, enter to save)
|
||||
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
|
||||
31. [ ] Show validation errors inline
|
||||
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
|
||||
33. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 6: Backend - Job Reset Endpoint
|
||||
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
|
||||
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
|
||||
36. [ ] Accept `target_state` parameter
|
||||
37. [ ] Implement reset logic for each target state
|
||||
38. [ ] Clear appropriate state_data keys
|
||||
39. [ ] Optionally trigger workers after reset
|
||||
40. [ ] Verify: Backend tests pass
|
||||
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
|
||||
|
||||
### Phase 7: Frontend - Reset Actions
|
||||
42. [ ] Add "Reset Job" dropdown/menu in header
|
||||
43. [ ] Show available reset targets based on current state
|
||||
44. [ ] Confirm dialog explaining what will be cleared
|
||||
45. [ ] Show success/error toast after reset
|
||||
46. [ ] **E2E Test**: Add reset confirmation dialog test
|
||||
47. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 8: Enhanced Stage Timeline
|
||||
48. [ ] Parse timeline events to calculate stage durations
|
||||
49. [ ] Display stages as cards or timeline items
|
||||
50. [ ] Show duration for each stage
|
||||
51. [ ] Highlight current/blocking stages
|
||||
52. [ ] **E2E Test**: Add timeline display test
|
||||
|
||||
### Phase 9: Final Verification
|
||||
53. [ ] Run full backend test suite: `make test-backend`
|
||||
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
|
||||
55. [ ] Manual production testing checklist (see Testing Strategy)
|
||||
56. [ ] Verify CI passes on PR
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
|
||||
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
|
||||
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
|
||||
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
|
||||
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
|
||||
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Based on `docs/TESTING.md` guidelines, this feature requires:
|
||||
|
||||
### Test Types Required
|
||||
|
||||
| Type | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
|
||||
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
|
||||
| Manual Production E2E | N/A | Final validation on staging/prod |
|
||||
|
||||
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
|
||||
|
||||
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
|
||||
|
||||
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
|
||||
```python
|
||||
def test_get_job_files_returns_signed_urls_for_all_files():
|
||||
"""Test that all files in file_urls get signed URLs."""
|
||||
|
||||
def test_get_job_files_handles_nested_file_urls():
|
||||
"""Test recursive traversal of stems.instrumental_clean, etc."""
|
||||
|
||||
def test_get_job_files_handles_missing_files_gracefully():
|
||||
"""Test files that don't exist in GCS are omitted or marked."""
|
||||
|
||||
def test_get_job_files_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_get_job_files_returns_404_for_missing_job():
|
||||
"""Test invalid job_id returns 404."""
|
||||
```
|
||||
|
||||
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
|
||||
```python
|
||||
def test_update_job_allows_editable_fields():
|
||||
"""Test artist, title, user_email, theme_id can be updated."""
|
||||
|
||||
def test_update_job_rejects_non_editable_fields():
|
||||
"""Test job_id, created_at, etc. cannot be updated."""
|
||||
|
||||
def test_update_job_validates_status_enum():
|
||||
"""Test invalid status value returns 422."""
|
||||
|
||||
def test_update_job_validates_theme_id_exists():
|
||||
"""Test non-existent theme_id returns 422."""
|
||||
|
||||
def test_update_job_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_update_job_returns_updated_job():
|
||||
"""Test response includes updated job data."""
|
||||
```
|
||||
|
||||
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
|
||||
```python
|
||||
def test_reset_to_pending_clears_all_state_data():
|
||||
"""Test full reset clears everything except input."""
|
||||
|
||||
def test_reset_to_audio_selection_clears_search_results():
|
||||
"""Test audio selection reset clears audio_search_results."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_review_data():
|
||||
"""Test review reset clears screens, videos, review data."""
|
||||
|
||||
def test_reset_to_instrumental_selection_clears_finals():
|
||||
"""Test instrumental reset clears encoding results."""
|
||||
|
||||
def test_reset_adds_timeline_event():
|
||||
"""Test reset adds admin action to timeline."""
|
||||
|
||||
def test_reset_invalid_target_returns_422():
|
||||
"""Test invalid target_state returns validation error."""
|
||||
|
||||
def test_reset_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_reset_from_complete_to_pending_works():
|
||||
"""Test can reset completed job back to start."""
|
||||
```
|
||||
|
||||
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
|
||||
|
||||
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
|
||||
|
||||
test.describe('Admin Job Detail Page', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await setAdminAuthToken(page);
|
||||
});
|
||||
|
||||
test('navigates from jobs list to detail page', async ({ page }) => {
|
||||
// Mock jobs list and single job endpoints
|
||||
await page.route('**/api/admin/jobs', route => route.fulfill({
|
||||
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
|
||||
}));
|
||||
await page.route('**/api/jobs/test123', route => route.fulfill({
|
||||
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
|
||||
}));
|
||||
|
||||
await page.goto('/admin/jobs');
|
||||
await page.click('text=test123');
|
||||
await expect(page).toHaveURL('/admin/jobs/test123');
|
||||
await expect(page.getByText('Job test123')).toBeVisible();
|
||||
});
|
||||
|
||||
test('displays all job metadata sections', async ({ page }) => {
|
||||
// Test overview cards, timeline, logs accordion, etc.
|
||||
});
|
||||
|
||||
test('files section shows downloadable links', async ({ page }) => {
|
||||
// Mock files endpoint, verify download buttons appear
|
||||
});
|
||||
|
||||
test('edit field inline and save', async ({ page }) => {
|
||||
// Click edit, change value, press Enter, verify PATCH called
|
||||
});
|
||||
|
||||
test('reset job shows confirmation dialog', async ({ page }) => {
|
||||
// Click reset, verify dialog, confirm, verify POST called
|
||||
});
|
||||
|
||||
test('handles API errors gracefully', async ({ page }) => {
|
||||
// Mock 500 response, verify error toast shown
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test File Organization
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── test_admin_job_detail.py # NEW - all backend tests for this feature
|
||||
├── test_admin_api.py # Existing admin tests (keep separate)
|
||||
└── conftest.py # Shared fixtures
|
||||
|
||||
frontend/e2e/
|
||||
├── fixtures/
|
||||
│ ├── test-helper.ts # Add setAdminAuthToken helper
|
||||
│ └── mock-responses.ts # Add admin job detail mocks
|
||||
└── regression/
|
||||
└── admin-job-detail.spec.ts # NEW - E2E regression tests
|
||||
```
|
||||
|
||||
### Mocking Guidelines
|
||||
|
||||
**Backend - What to mock:**
|
||||
- `StorageService.generate_signed_url()` - return predictable test URLs
|
||||
- `StorageService.file_exists()` - control which files "exist"
|
||||
- Firestore operations via existing `mock_firestore` fixture
|
||||
|
||||
**Backend - What NOT to mock:**
|
||||
- `JobManager` logic - test the real reset/update logic
|
||||
- Validation logic - test real Pydantic validation
|
||||
- Authorization checks - test real `require_admin` dependency
|
||||
|
||||
**Frontend - What to mock:**
|
||||
- All API responses via Playwright route interception
|
||||
- Time (if testing token expiry)
|
||||
|
||||
**Frontend - What NOT to mock:**
|
||||
- React component behavior
|
||||
- User interactions (click, type, etc.)
|
||||
|
||||
### CI Requirements
|
||||
|
||||
This feature will be validated by existing CI checks:
|
||||
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
|
||||
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
|
||||
- `frontend-build` - Must pass (verifies new page compiles)
|
||||
|
||||
### Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Backend tests only (for this feature)
|
||||
poetry run pytest backend/tests/test_admin_job_detail.py -v
|
||||
|
||||
# All backend tests
|
||||
make test-backend
|
||||
|
||||
# Frontend E2E regression (includes new tests)
|
||||
cd frontend && npm run test:e2e
|
||||
|
||||
# Single E2E test file
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
|
||||
|
||||
# With browser visible for debugging
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
|
||||
```
|
||||
|
||||
### Manual Production Testing Checklist
|
||||
|
||||
After deployment, manually verify on production:
|
||||
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
|
||||
- [ ] All file download links work (valid signed URLs)
|
||||
- [ ] Edit artist/title → refresh → changes persisted
|
||||
- [ ] Reset job to pending → job status shows "pending"
|
||||
- [ ] Logs section loads and filters work
|
||||
- [ ] Error states handled (invalid job ID shows friendly error)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
|
||||
- **Recommendation**: Delete it. No external links depend on it.
|
||||
|
||||
2. [ ] Should job reset delete GCS files or just clear state_data references?
|
||||
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
|
||||
|
||||
3. [ ] Should edits be immediately saved or require explicit "Save" button?
|
||||
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
|
||||
2. Backend: New endpoints are additive, won't affect existing functionality
|
||||
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
|
||||
|
||||
## Notes
|
||||
|
||||
- The existing detail page has good foundational code - we're enhancing rather than replacing
|
||||
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
|
||||
- Job reset is powerful - consider adding confirmation dialogs and logging
|
||||
@@ -1,120 +0,0 @@
|
||||
# Plan: Admin Lyrics Review Dashboard
|
||||
|
||||
**Created:** 2026-03-05
|
||||
**Branch:** feat/sess-20260305-0002-admin-lyrics-review
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add an "Edit Reviews" section to the admin dashboard for reviewing completed lyrics reviews. Three goals:
|
||||
|
||||
1. **View what users saw** - Jump into a read-only lyrics review UI showing original vs corrected lyrics
|
||||
2. **Replay user edits** - Step through edit log entries to see exactly what each user changed
|
||||
3. **Review raw data** - View human edits data to evaluate quality for AudioShake data sharing
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] New admin sidebar item "Edit Reviews" linking to `/admin/edit-reviews`
|
||||
- [ ] Backend endpoint to list recent jobs that have completed lyrics review (with edit logs)
|
||||
- [ ] Backend endpoint to serve immutable correction data for a job (read-only, cannot modify job)
|
||||
- [ ] Admin page showing list of recently reviewed jobs with key metrics
|
||||
- [ ] Ability to open the existing LyricsAnalyzer in read-only mode with snapshot data
|
||||
- [ ] Edit log replay/stepper UI to walk through user changes one by one
|
||||
- [ ] Raw data viewer for edit logs, annotations, and correction diffs
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Backend: 2 new admin endpoints
|
||||
|
||||
1. **`GET /api/admin/edit-reviews`** - List jobs with completed reviews
|
||||
- Query Firestore for jobs with `state_data.last_edit_log_path` set (indicating review was performed)
|
||||
- Filter by status (completed jobs, or optionally all post-review statuses)
|
||||
- Return summary: job_id, artist, title, user_email, review_completed_at, edit_count, feedback_count
|
||||
- Support pagination, search, exclude_test filtering
|
||||
|
||||
2. **`GET /api/admin/edit-reviews/{job_id}`** - Get full review snapshot
|
||||
- Returns immutable copy of:
|
||||
- Original corrections (`corrections.json` from GCS)
|
||||
- Updated corrections (`corrections_updated.json` from GCS)
|
||||
- Edit log (`edit_log_{session_id}.json` from GCS)
|
||||
- Annotations (`annotations.json` from GCS)
|
||||
- Job metadata (artist, title, user_email, timestamps)
|
||||
- Audio URL (signed, for playback in review UI)
|
||||
- This is a **read-only** endpoint - no job state mutations
|
||||
- Uses `require_admin` dependency (not `require_review_auth`)
|
||||
- Does NOT transition job status or modify any state
|
||||
|
||||
### Frontend: New admin page + Read-only review mode
|
||||
|
||||
1. **`/admin/edit-reviews/page.tsx`** - List page
|
||||
- Table of recently reviewed jobs
|
||||
- Columns: Artist/Title, User, Review Date, Edits Made, Feedback Given, Actions
|
||||
- "View Review" button → opens read-only lyrics review
|
||||
- "View Edit Log" button → opens edit replay view
|
||||
- "View Raw Data" button → shows raw JSON
|
||||
|
||||
2. **Read-only review viewer** - Reuse LyricsAnalyzer
|
||||
- Load data from the new admin endpoint (not the regular review endpoint)
|
||||
- Pass `isReadOnly={true}` and `apiClient={null}` to LyricsAnalyzer
|
||||
- Header shows "Admin Review - Read Only" indicator
|
||||
- No submit/save buttons, no state transitions
|
||||
- Audio playback still works (via signed URL)
|
||||
|
||||
3. **Edit log replay** - New component
|
||||
- Timeline/stepper showing each edit entry
|
||||
- For each entry: operation type, text before → text after, timestamp
|
||||
- Step forward/backward through edits
|
||||
- Show feedback reason if user provided one
|
||||
- Summary stats: total edits, edits with feedback, correction types breakdown
|
||||
|
||||
4. **Raw data viewer** - JSON viewer tabs
|
||||
- Tab 1: Edit Log (formatted JSON with syntax highlighting)
|
||||
- Tab 2: Annotations
|
||||
- Tab 3: Corrections diff (original vs updated)
|
||||
|
||||
### Key safety guarantee
|
||||
|
||||
The admin review endpoint serves a **snapshot** of the data. It reads from GCS files that were already written. The LyricsAnalyzer receives `apiClient={null}` so it physically cannot call any mutation endpoints. Even if code changes in the future, the admin endpoint itself never writes to Firestore or GCS.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Backend: Add `GET /api/admin/edit-reviews` endpoint** - List reviewed jobs with edit log metadata
|
||||
2. [ ] **Backend: Add `GET /api/admin/edit-reviews/{job_id}` endpoint** - Serve full immutable review snapshot
|
||||
3. [ ] **Frontend: Add admin API client methods** - `adminApi.listEditReviews()`, `adminApi.getEditReview(jobId)`
|
||||
4. [ ] **Frontend: Create `/admin/edit-reviews/page.tsx`** - List page with table
|
||||
5. [ ] **Frontend: Add sidebar nav item** - "Edit Reviews" with BookOpen icon
|
||||
6. [ ] **Frontend: Create read-only review viewer** - Load snapshot data into LyricsAnalyzer with isReadOnly
|
||||
7. [ ] **Frontend: Create edit log replay component** - Step-through UI for edit entries
|
||||
8. [ ] **Frontend: Create raw data viewer** - JSON viewer with tabs
|
||||
9. [ ] **Tests: Backend unit tests** for new admin endpoints
|
||||
10. [ ] **Tests: Frontend component tests** for new page
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/admin.py` | Modify | Add edit-reviews endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for edit reviews |
|
||||
| `frontend/app/admin/edit-reviews/page.tsx` | Create | Main edit reviews list page |
|
||||
| `frontend/components/admin/edit-review-viewer.tsx` | Create | Read-only review wrapper using LyricsAnalyzer |
|
||||
| `frontend/components/admin/edit-log-replay.tsx` | Create | Edit log stepper/replay component |
|
||||
| `frontend/components/admin/raw-data-viewer.tsx` | Create | JSON viewer with tabs |
|
||||
| `frontend/components/admin/admin-sidebar.tsx` | Modify | Add Edit Reviews nav item |
|
||||
| `backend/tests/test_admin_edit_reviews.py` | Create | Backend tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for backend endpoints (mock Firestore + GCS)
|
||||
- Verify admin auth is required on all new endpoints
|
||||
- Verify no write operations occur on read-only endpoints
|
||||
- Frontend component tests for key interactions
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we show ALL jobs that went through review, or only ones with edit logs? (Recommendation: only those with edit logs, since those are the interesting ones)
|
||||
- [ ] Do we need audio playback in the admin review viewer? (Recommendation: yes, it helps understand what the user heard)
|
||||
- [ ] Should the raw data be downloadable? (Nice-to-have, can add later)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new endpoints, new pages). Removing the sidebar link effectively hides the feature. No existing behavior is modified.
|
||||
@@ -1,151 +0,0 @@
|
||||
# Plan: AI-Powered Credit Gating
|
||||
|
||||
**Created:** 2026-03-23
|
||||
**Branch:** feat/sess-20260323-1716-ai-credit-gating
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Replace the automatic welcome/feedback credit grants with an AI-powered evaluation step. When a user triggers a credit grant (email verification or feedback submission), instead of immediately granting credits, we collect abuse signals and ask Gemini to make a grant/deny decision. Suspected abusers get a friendly rejection email (CC'd to Andrew) instead of free credits.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Welcome credit grants go through AI evaluation before granting
|
||||
- [ ] Feedback credit grants go through AI evaluation before granting
|
||||
- [ ] AI evaluates: fingerprint correlations, IP correlations, IP geo info, user agent, signup recency patterns
|
||||
- [ ] Approved users get credits as before (seamless experience)
|
||||
- [ ] Rejected users get a friendly-but-firm email explaining the situation, CC'd to andrew@beveridge.uk
|
||||
- [ ] Rejection email invites them to reply if it's a mistake
|
||||
- [ ] Decision is logged to Firestore for audit trail
|
||||
- [ ] Fail-open: if AI evaluation fails, grant credits (don't punish users for our errors)
|
||||
- [ ] Admin can override decisions via existing admin credit grant endpoint
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
User verifies email / submits feedback
|
||||
↓
|
||||
Collect abuse signals (sync, fast):
|
||||
- User's fingerprint + IP
|
||||
- Other accounts with same fingerprint (from correlations)
|
||||
- Other accounts with same IP
|
||||
- IP geolocation info
|
||||
- Recent signup count from same IP/fingerprint (last 24h)
|
||||
- User agent string
|
||||
- For feedback: the feedback content itself
|
||||
↓
|
||||
Call Gemini (vertexai/gemini-2.5-pro-preview) with structured prompt:
|
||||
- Present all signals
|
||||
- Ask for JSON response: {decision: "grant" | "deny", reasoning: string, confidence: number}
|
||||
↓
|
||||
If "grant": grant credits normally, log decision
|
||||
If "deny": skip credit grant, send rejection email, log decision
|
||||
If AI call fails: grant credits (fail-open), log error
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Two-phase verification**: The magic link verify endpoint returns immediately with session + a `credit_status: "evaluating"` field. The frontend shows a "preparing your account" loading screen, then polls/waits for the evaluation result via a new lightweight endpoint. This keeps the auth flow snappy while the AI evaluation runs in the background of the same request or via a short follow-up call. The frontend then shows either a celebratory "2 free credits!" interstitial or a "sorry, couldn't grant credits" interstitial before navigating to the dashboard.
|
||||
|
||||
2. **Fail-open**: If Gemini is down or returns an error, we grant credits. Better to let an occasional abuser through than block legitimate users.
|
||||
|
||||
3. **Use google-genai directly**: The project already has `langchain-google-genai` installed. We can use the simpler `google.generativeai` library directly for a focused prompt/response, or use the existing LangChain setup. We'll use `google.generativeai` for simplicity since this is a single-turn evaluation, not an agentic workflow.
|
||||
|
||||
4. **Structured JSON output**: Ask Gemini for structured output with decision, reasoning, and confidence to make decisions parseable and auditable.
|
||||
|
||||
5. **Log all decisions**: Store every evaluation in a `credit_evaluations` Firestore collection for audit trail and to tune the prompt over time.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add google-generativeai dependency
|
||||
- Add `google-generativeai` to pyproject.toml (lightweight, direct Gemini access)
|
||||
- Verify it works with Vertex AI auth in the nomadkaraoke project
|
||||
|
||||
### Step 2: Create credit evaluation service
|
||||
- New file: `backend/services/credit_evaluation_service.py`
|
||||
- `CreditEvaluationService` class with:
|
||||
- `evaluate_credit_grant(email, grant_type, feedback_content=None) -> CreditEvaluation`
|
||||
- Collects all abuse signals by calling existing services (user_service, ip_geolocation_service)
|
||||
- Builds prompt with signals
|
||||
- Calls Gemini for decision
|
||||
- Logs decision to Firestore
|
||||
- Returns structured result
|
||||
|
||||
### Step 3: Build the Gemini prompt
|
||||
- System prompt explaining the role: "You are an anti-abuse evaluator for a karaoke video service..."
|
||||
- Present all signals in a structured format
|
||||
- Request JSON response: `{decision: "grant"|"deny", reasoning: "...", confidence: 0.0-1.0}`
|
||||
- Include clear guidelines: same fingerprint across accounts = very suspicious, same IP = somewhat suspicious, etc.
|
||||
|
||||
### Step 4: Create rejection email template
|
||||
- Add `send_credit_denied_email(email, reasoning)` to EmailService
|
||||
- Friendly-but-firm tone
|
||||
- CC andrew@beveridge.uk
|
||||
- Invite reply if it's a mistake
|
||||
- Use existing email template infrastructure (branding, styling)
|
||||
|
||||
### Step 5: Integrate into welcome credit flow
|
||||
- Modify `grant_welcome_credits_if_eligible()` in user_service.py
|
||||
- Before granting: call `CreditEvaluationService.evaluate_credit_grant(email, "welcome")`
|
||||
- If denied: send rejection email, return False
|
||||
- If error: log warning, grant anyway (fail-open)
|
||||
|
||||
### Step 6: Integrate into feedback credit flow
|
||||
- Modify feedback endpoint in users.py
|
||||
- Before granting credits: evaluate with feedback content included
|
||||
- If denied: return different response message, send rejection email
|
||||
- If error: grant anyway
|
||||
|
||||
### Step 7: Add Firestore collection for audit trail
|
||||
- Collection: `credit_evaluations`
|
||||
- Document fields: email, grant_type, decision, reasoning, confidence, signals_snapshot, model_used, evaluated_at
|
||||
- TTL: keep indefinitely (small volume, high audit value)
|
||||
|
||||
### Step 8: Tests
|
||||
- Unit tests for signal collection
|
||||
- Unit tests for prompt building
|
||||
- Unit tests for decision parsing (grant/deny/error cases)
|
||||
- Integration test for the evaluation service with mocked Gemini
|
||||
- Integration test for welcome credit flow with evaluation
|
||||
- Integration test for feedback credit flow with evaluation
|
||||
- Test fail-open behavior
|
||||
|
||||
### Step 9: Configuration
|
||||
- Add env vars: `CREDIT_EVAL_MODEL` (default: "gemini-2.5-pro-preview"), `CREDIT_EVAL_ENABLED` (default: true)
|
||||
- Add to config.py
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `pyproject.toml` | Modify | Add google-generativeai dependency |
|
||||
| `backend/config.py` | Modify | Add credit evaluation config vars |
|
||||
| `backend/services/credit_evaluation_service.py` | Create | New service: collect signals, call Gemini, log decision |
|
||||
| `backend/services/user_service.py` | Modify | Integrate evaluation into welcome credit grant |
|
||||
| `backend/api/routes/users.py` | Modify | Integrate evaluation into feedback credit grant |
|
||||
| `backend/services/email_service.py` | Modify | Add credit denied email template |
|
||||
| `backend/tests/test_credit_evaluation.py` | Create | Tests for evaluation service |
|
||||
| `backend/tests/test_anti_abuse.py` | Modify | Update welcome credit tests for evaluation |
|
||||
| `frontend/app/auth/verify/page.tsx` | Modify | Add "preparing account" loading + credit result interstitial |
|
||||
| `frontend/lib/api.ts` | Modify | Add credit evaluation status endpoint |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Signal collection, prompt building, response parsing, fail-open behavior
|
||||
- **Integration tests**: Full evaluation flow with mocked Gemini responses
|
||||
- **E2E tests**: Manual verification in production with test accounts
|
||||
- **Edge cases**: Gemini timeout, malformed response, missing signals, first-ever user (no correlations)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Gemini model**: Using `gemini-3.1-pro-preview` as specified — already available in the nomadkaraoke GCP project.
|
||||
|
||||
2. **Latency UX**: Frontend shows a friendly "preparing your account" loading screen while evaluation runs. Then transitions to either a celebratory "2 free credits!" message (with gentle reminder not to abuse) or a "sorry, couldn't grant free credits" message with a buy credits CTA. User clicks through to dashboard from either.
|
||||
|
||||
3. **Should we also gate the existing rate limiter?** Currently `is_signup_rate_limited()` silently blocks at 2 accounts per IP/fingerprint per 24h. The AI evaluation is complementary — rate limiter blocks obvious rapid abuse, AI evaluation catches more subtle patterns.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Set `CREDIT_EVAL_ENABLED=false` in Cloud Run env vars → immediate rollback to auto-grant behavior. No data model changes needed. The evaluation service is purely additive.
|
||||
@@ -1,150 +0,0 @@
|
||||
# Plan: Block Disposable Email Registration
|
||||
|
||||
**Created:** 2026-03-19
|
||||
**Branch:** feat/sess-20260319-1332-disposable-email-abuse
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Users are signing up with disposable email addresses (yopmail.com, emailhook.site, etc.) to get unlimited free credits (2 per account). The `EmailValidationService` already exists with 160+ disposable domains but is **never called during signup**. We need to wire it into the magic link flow and show a friendly rejection message.
|
||||
|
||||
## Context
|
||||
|
||||
- The `EmailValidationService` (`backend/services/email_validation_service.py`) is fully implemented with disposable domain detection, email/IP blocklists, and admin management APIs
|
||||
- The admin blocklist management routes exist (`backend/api/routes/rate_limits.py`) and work
|
||||
- The magic link endpoint (`POST /api/users/auth/magic-link` in `backend/api/routes/users.py:133`) **never calls** the validation service
|
||||
- Currently 1 yopmail user and 1 emailhook.site user in `gen_users`
|
||||
- Each new account gets 2 free welcome credits automatically
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Reject disposable email domains at the magic link endpoint (backend)
|
||||
- [x] Return a **specific, distinguishable error** so the frontend can show a friendly message (not the generic "email enumeration" response)
|
||||
- [x] Show a warm, friendly message in the AuthDialog explaining why and encouraging real email use
|
||||
- [x] Add `emailhook.site` to the default disposable domains list
|
||||
- [x] Add backend unit tests for the integration
|
||||
- [x] Add frontend test coverage for the new error state
|
||||
|
||||
## Design Decision: Explicit Rejection vs Silent Failure
|
||||
|
||||
The current magic link endpoint returns a generic success message to prevent email enumeration. For disposable emails, we will **explicitly reject** with a 422 status and a specific error message. This is the right trade-off because:
|
||||
|
||||
1. We *want* the user to know their disposable email won't work (so they can use a real one)
|
||||
2. Disposable domain detection is not sensitive information — the blocklist is public knowledge
|
||||
3. The friendly message serves our goal of converting them to real email users
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Backend — Add disposable email check to magic link endpoint
|
||||
|
||||
**File:** `backend/api/routes/users.py`
|
||||
|
||||
Add the `EmailValidationService` check **after** email lowercasing (line 158) and **before** tenant validation (line 164). Return a 422 with a friendly message.
|
||||
|
||||
```python
|
||||
# Check for disposable email domains
|
||||
email_validation = get_email_validation_service()
|
||||
if email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"Blocked disposable email signup attempt: {_mask_email(email)}")
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail="disposable_email_not_allowed"
|
||||
)
|
||||
```
|
||||
|
||||
Use a machine-readable error code (`disposable_email_not_allowed`) so the frontend can match on it and show the custom friendly message. This keeps the backend response lean and puts the UX copy in the frontend where it belongs.
|
||||
|
||||
### Step 2: Backend — Also check email and IP blocklists
|
||||
|
||||
In the same location, add checks for explicitly blocked emails and IPs:
|
||||
|
||||
```python
|
||||
if email_validation.is_email_blocked(email):
|
||||
logger.warning(f"Blocked email signup attempt: {_mask_email(email)}")
|
||||
# Silent reject — return generic success to prevent enumeration
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
|
||||
ip_address = http_request.client.host if http_request.client else None
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"Blocked IP signup attempt: {ip_address}")
|
||||
return SendMagicLinkResponse(
|
||||
status="success",
|
||||
message="If this email is registered, you will receive a sign-in link shortly."
|
||||
)
|
||||
```
|
||||
|
||||
Note: Blocked emails/IPs get silent rejection (anti-enumeration). Disposable domains get explicit rejection (we want the user to switch to a real email).
|
||||
|
||||
### Step 3: Backend — Add `emailhook.site` to default disposable domains
|
||||
|
||||
**File:** `backend/services/email_validation_service.py`
|
||||
|
||||
Add `emailhook.site` to `DEFAULT_DISPOSABLE_DOMAINS` set.
|
||||
|
||||
### Step 4: Frontend — Handle the disposable email error in AuthDialog
|
||||
|
||||
**File:** `frontend/components/auth/AuthDialog.tsx`
|
||||
|
||||
The `sendMagicLink` call in `auth.ts` already catches errors and sets `error` from the API response message. The `handleResponse` in `api.ts` throws an `ApiError` with the `detail` string for non-200 responses.
|
||||
|
||||
Update `handleSendMagicLink` to detect the `disposable_email_not_allowed` error code and show a friendly multi-line message instead of the terse error string:
|
||||
|
||||
```tsx
|
||||
const success = await sendMagicLink(email.trim().toLowerCase())
|
||||
if (success) {
|
||||
setStep("sent")
|
||||
}
|
||||
// If error contains our specific code, override with friendly message
|
||||
// (the error state is already set by the auth store)
|
||||
```
|
||||
|
||||
We need to detect the specific error. The cleanest approach: check the error string in the `displayError` rendering. If it matches `disposable_email_not_allowed`, render a custom friendly block instead of the red error text.
|
||||
|
||||
The friendly message:
|
||||
|
||||
> **We don't support disposable email addresses**
|
||||
>
|
||||
> We'd love to help you create karaoke videos! Please use your regular email address so we can send you a notification when your video is ready.
|
||||
>
|
||||
> We promise — no spam, no marketing emails, ever. We only email you about your karaoke videos.
|
||||
|
||||
### Step 5: Backend tests
|
||||
|
||||
**File:** `backend/tests/test_email_validation_integration.py` (new)
|
||||
|
||||
Test cases:
|
||||
1. Magic link with disposable email returns 422 with `disposable_email_not_allowed`
|
||||
2. Magic link with legitimate email proceeds normally
|
||||
3. Magic link with blocked email returns silent success (no actual email sent)
|
||||
4. Magic link with blocked IP returns silent success
|
||||
|
||||
### Step 6: Frontend tests
|
||||
|
||||
**File:** `frontend/components/__tests__/AuthDialog.test.tsx` (new or extend existing)
|
||||
|
||||
Test cases:
|
||||
1. Submitting a disposable email shows the friendly rejection message
|
||||
2. The friendly message contains the key copy (no spam promise, use real email)
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/users.py` | Modify | Add email validation checks to magic link endpoint |
|
||||
| `backend/services/email_validation_service.py` | Modify | Add `emailhook.site` to default domains |
|
||||
| `frontend/components/auth/AuthDialog.tsx` | Modify | Handle disposable email error with friendly message |
|
||||
| `backend/tests/test_email_validation_integration.py` | Create | Integration tests for the validation in signup flow |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests:** Validate `is_disposable_domain` catches yopmail, emailhook.site (already covered in existing tests)
|
||||
- **Integration tests:** New test file verifying the magic link endpoint rejects disposable emails with 422
|
||||
- **Frontend tests:** Verify the AuthDialog shows friendly message on disposable email error
|
||||
- **Manual verification:** Try signing up with `test@yopmail.com` on dev server to see the UX
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Remove the validation check from `send_magic_link()` in `users.py`. The `EmailValidationService` itself remains unchanged (it's already deployed and used by admin routes).
|
||||
@@ -1,236 +0,0 @@
|
||||
# Plan: Combined Lyrics + Instrumental Review Phase
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
|
||||
|
||||
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
|
||||
|
||||
### Key Insight
|
||||
|
||||
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
|
||||
|
||||
### Benefits
|
||||
|
||||
1. **Better UX** - One interaction point instead of two
|
||||
2. **Faster turnaround** - Users complete both reviews in one session
|
||||
3. **Reduced drop-off** - Users less likely to abandon between phases
|
||||
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
|
||||
5. **Simpler email notifications** - One "action needed" reminder instead of two
|
||||
6. **Cleaner codebase** - No conditional paths or legacy state handling
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
|
||||
- [x] Functional: Single email notification for "review needed" instead of two separate ones
|
||||
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
|
||||
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
|
||||
- [x] Functional: Admin "reset to review" action works with combined flow
|
||||
- [x] Non-functional: No regression in existing test coverage
|
||||
- [x] Non-functional: Clean removal of old code paths (no dead code)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Strategy: "Combined Review - Clean Cutover"
|
||||
|
||||
**State Machine - Before:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
**State Machine - After:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
|
||||
|
||||
### Key Changes
|
||||
|
||||
1. **Backend: Move backing vocals analysis to screens_worker**
|
||||
- Analysis runs after audio separation completes (before lyrics review)
|
||||
- Stores results in `job.state_data['backing_vocals_analysis']`
|
||||
- Remove analysis from render_video_worker entirely
|
||||
|
||||
2. **Backend: Replace review completion endpoint**
|
||||
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
|
||||
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
|
||||
- Remove the old endpoint that didn't require instrumental
|
||||
|
||||
3. **Backend: Simplify render_video_worker**
|
||||
- Always expect `instrumental_selection` in state_data
|
||||
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
|
||||
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
|
||||
|
||||
4. **Frontend: Single combined review page**
|
||||
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
|
||||
- Remove separate `/instrumental` route entirely
|
||||
- Single "Complete Review" button submits both
|
||||
|
||||
5. **Backend: Remove instrumental-only endpoints**
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-options`
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
|
||||
- Remove `POST /api/jobs/{job_id}/select-instrumental`
|
||||
- Keep custom instrumental creation for future advanced use
|
||||
|
||||
6. **Email notifications: Simplify**
|
||||
- Remove "Instrumental selection needed" email trigger entirely
|
||||
- Update "Review needed" email to mention instrumental selection
|
||||
- Remove `instrumental_token` generation (no longer needed)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Move backing vocals analysis to screens_worker**
|
||||
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
|
||||
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
|
||||
- Remove analysis code from render_video_worker.py entirely
|
||||
|
||||
2. [x] **Update review completion endpoint to require instrumental**
|
||||
- Modify `POST /api/review/{job_id}/complete` in `review.py`
|
||||
- Add required `instrumental_selection` field to request body
|
||||
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
|
||||
|
||||
3. [x] **Add instrumental data to correction-data endpoint**
|
||||
- Modify `GET /api/review/{job_id}/correction-data` to include:
|
||||
- `instrumental_options`: List of {id, label, audio_url}
|
||||
- `backing_vocals_analysis`: Analysis data from state_data
|
||||
|
||||
4. [x] **Simplify render_video_worker**
|
||||
- Remove conditional check for instrumental pre-selection
|
||||
- Always transition directly to `INSTRUMENTAL_SELECTED`
|
||||
- Always trigger video worker immediately after render
|
||||
|
||||
5. [x] **Update state machine**
|
||||
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
|
||||
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
|
||||
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
|
||||
|
||||
6. [x] **Remove instrumental-only endpoints**
|
||||
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
|
||||
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
|
||||
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
|
||||
- Remove `require_instrumental_auth` dependency
|
||||
- Remove `instrumental_token` generation in job_manager.py
|
||||
|
||||
### Phase 2: Frontend Changes
|
||||
|
||||
7. [x] **Create combined review page**
|
||||
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
|
||||
- Instrumental selection UI appears after preview video section
|
||||
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
|
||||
|
||||
8. [x] **Create InstrumentalSelectorEmbedded component**
|
||||
- Created embedded version of InstrumentalSelector
|
||||
- Accepts options, analysis, value, onChange props
|
||||
- Supports compact mode for modal embedding
|
||||
|
||||
9. [x] **Update review completion flow**
|
||||
- "Complete Review" button requires instrumental selection
|
||||
- Validation error shown if no instrumental selected
|
||||
- Single API call submits both lyrics and instrumental selection
|
||||
|
||||
10. [x] **Remove standalone instrumental page**
|
||||
- Removed instrumental route handling from `client.tsx`
|
||||
- Removed `getExpectedStates` case for "instrumental"
|
||||
- InstrumentalSelector component stubbed out (deprecated)
|
||||
- Kept `selectInstrumental` API for finalise-only jobs
|
||||
|
||||
11. [x] **Update lib/api.ts**
|
||||
- Modified `completeReview()` to require instrumental selection
|
||||
- Kept `selectInstrumental()` for finalise-only jobs
|
||||
- Removed other instrumental-only functions
|
||||
|
||||
### Phase 3: Email & Notification Cleanup
|
||||
|
||||
12. [x] **Remove instrumental notification trigger**
|
||||
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
|
||||
- Updated docstrings to clarify finalise-only vs normal flow
|
||||
- Instrumental reminder scheduling still works for finalise-only jobs
|
||||
|
||||
13. [x] **Update review email template**
|
||||
- Update copy to mention "review lyrics and select instrumental"
|
||||
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
|
||||
|
||||
### Phase 4: Testing & Cleanup
|
||||
|
||||
14. [x] **Update/remove tests for old flow**
|
||||
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
|
||||
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
|
||||
- Updated worker tests to import from screens_worker (where analysis moved)
|
||||
- Fixed conftest.py mocks to return AuthResult objects
|
||||
|
||||
15. [x] **Add tests for combined flow**
|
||||
- Existing tests already cover combined completion endpoint
|
||||
- Test infrastructure validates combined review flow works
|
||||
- 1741 tests pass (only 1 pre-existing unrelated failure)
|
||||
|
||||
16. [x] **Update documentation**
|
||||
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
|
||||
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
|
||||
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
|
||||
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
|
||||
|
||||
17. [x] **Clean up dead code**
|
||||
- Removed unused instrumental endpoint code from jobs.py
|
||||
- Stubbed InstrumentalSelector component (deprecated)
|
||||
- Kept select-instrumental endpoint for finalise-only jobs
|
||||
|
||||
## Files to Modify/Delete
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
|
||||
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
|
||||
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
|
||||
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
|
||||
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
|
||||
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
|
||||
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
|
||||
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
|
||||
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
|
||||
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
|
||||
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
|
||||
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
|
||||
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
|
||||
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
|
||||
|
||||
### Integration Tests
|
||||
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
|
||||
- `test_non_interactive_job`: CLI jobs skip review entirely as before
|
||||
|
||||
### E2E Tests
|
||||
- `test_combined_review_ui`: User completes both reviews in single session
|
||||
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
|
||||
| CLI regression | Test `non_interactive` flag behavior explicitly |
|
||||
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
|
||||
|
||||
## Migration Notes
|
||||
|
||||
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
|
||||
1. Admin resets job to `AWAITING_REVIEW` via admin panel
|
||||
2. User completes combined review
|
||||
3. Job proceeds normally
|
||||
|
||||
This is acceptable because:
|
||||
- Very few jobs should be in this state at any time
|
||||
- Admin reset is already a supported operation
|
||||
- Clean cutover is worth the minor migration friction
|
||||
@@ -1,321 +0,0 @@
|
||||
# Plan: Consolidate YouTube Download Paths
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Status:** Planning
|
||||
|
||||
## Problem Statement
|
||||
|
||||
YouTube downloads currently have **3 different code paths** with different behaviors:
|
||||
|
||||
1. **Audio Search → Select YouTube** (`audio_search.py`)
|
||||
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
|
||||
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
|
||||
|
||||
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
|
||||
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
|
||||
- Runs on Cloud Run, blocked by YouTube bot detection
|
||||
- Never uses remote flacfetch even when configured
|
||||
|
||||
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
|
||||
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
|
||||
- Works fine locally, not relevant to cloud issues
|
||||
|
||||
### Current Issues
|
||||
|
||||
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
|
||||
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
|
||||
3. **Inconsistent error handling** - Different paths handle errors differently
|
||||
4. **Testing gaps** - No integration tests verifying the full flow
|
||||
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
|
||||
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
|
||||
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
|
||||
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
|
||||
5. **Remove dead code** - Clean up redundant implementations
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Entry Points │
|
||||
├──────────────────────┬──────────────────────┬───────────────────────┤
|
||||
│ Audio Search Select │ Direct URL Submit │ File Upload │
|
||||
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
|
||||
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
┌──────────────────────────────────────────────┐ │
|
||||
│ YouTubeDownloadService │ │
|
||||
│ (NEW: backend/services/youtube_service.py) │ │
|
||||
│ │ │
|
||||
│ - download_by_url(url, job_id) → gcs_path │ │
|
||||
│ - download_by_id(video_id, job_id) → gcs_path│ │
|
||||
│ │ │
|
||||
│ Internally: │ │
|
||||
│ 1. Check if remote flacfetch enabled │ │
|
||||
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
|
||||
│ 3. If no: Use local yt_dlp with clear error │ │
|
||||
│ messaging about bot detection risk │ │
|
||||
└──────────────────┬───────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GCS │
|
||||
│ uploads/{job_id}/audio/ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Create YouTubeDownloadService
|
||||
|
||||
**File:** `backend/services/youtube_download_service.py`
|
||||
|
||||
```python
|
||||
class YouTubeDownloadService:
|
||||
"""
|
||||
Single point of entry for all YouTube downloads in the cloud backend.
|
||||
|
||||
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
|
||||
on the flacfetch VM which has YouTube cookies and avoids bot detection.
|
||||
|
||||
When remote is not configured, falls back to local yt_dlp with a warning
|
||||
that downloads may be blocked.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._flacfetch_client = get_flacfetch_client()
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio and upload to GCS.
|
||||
|
||||
Args:
|
||||
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (not gs:// prefix, just the path portion)
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
video_id = self._extract_video_id(url)
|
||||
|
||||
if self._flacfetch_client:
|
||||
return await self._download_remote(video_id, job_id, artist, title)
|
||||
else:
|
||||
logger.warning(
|
||||
"Remote flacfetch not configured. Attempting local download, "
|
||||
"but this may fail due to YouTube bot detection on Cloud Run IPs."
|
||||
)
|
||||
return await self._download_local(url, job_id, artist, title)
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Download by video ID instead of URL."""
|
||||
url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
return await self.download(url, job_id, artist, title)
|
||||
```
|
||||
|
||||
### Phase 2: Refactor Audio Search Path
|
||||
|
||||
**File:** `backend/api/routes/audio_search.py`
|
||||
|
||||
Changes to `_download_and_start_processing()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Complex branching with multiple paths
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Torrent path...
|
||||
elif is_remote_enabled and source_id and source_name:
|
||||
# YouTube remote path (buggy source_id extraction)...
|
||||
else:
|
||||
# Local path with another YouTube branch...
|
||||
|
||||
# AFTER: Clean separation by source type
|
||||
if is_torrent_source:
|
||||
result = await _download_torrent(source_name, source_id, target_file, job_id)
|
||||
elif source_name == 'YouTube':
|
||||
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
|
||||
elif source_name == 'Spotify':
|
||||
result = await _download_spotify(source_id, job_id, artist, title)
|
||||
else:
|
||||
raise DownloadError(f"Unknown source: {source_name}")
|
||||
```
|
||||
|
||||
### Phase 3: Refactor Direct URL Path
|
||||
|
||||
**File:** `backend/api/routes/file_upload.py`
|
||||
|
||||
Changes to `create_job_from_url()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
|
||||
# This always runs on Cloud Run and gets blocked
|
||||
|
||||
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
|
||||
if is_youtube_url(body.url):
|
||||
youtube_service = get_youtube_download_service()
|
||||
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
|
||||
# Update job with audio already downloaded
|
||||
job.input_media_gcs_path = gcs_path
|
||||
# Skip audio download in worker
|
||||
```
|
||||
|
||||
**File:** `backend/workers/audio_worker.py`
|
||||
|
||||
Changes to `download_audio()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Has YouTube download logic with download_from_url()
|
||||
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
|
||||
|
||||
async def download_audio(job_id: str, job: Job) -> str:
|
||||
"""Download audio from GCS (already uploaded by service layer)."""
|
||||
if not job.input_media_gcs_path:
|
||||
raise Exception("No audio path - this shouldn't happen for URL jobs")
|
||||
|
||||
# Download from GCS to local for processing
|
||||
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
|
||||
storage_service.download_file(job.input_media_gcs_path, local_path)
|
||||
return local_path
|
||||
```
|
||||
|
||||
### Phase 4: Clean Up Dead Code
|
||||
|
||||
Remove or consolidate:
|
||||
|
||||
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
|
||||
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
|
||||
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
|
||||
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
|
||||
|
||||
### Phase 5: Comprehensive Testing
|
||||
|
||||
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadService:
|
||||
def test_download_uses_remote_when_configured(self):
|
||||
"""When FLACFETCH_API_URL is set, should use remote client."""
|
||||
|
||||
def test_download_falls_back_to_local_when_remote_disabled(self):
|
||||
"""When remote not configured, should use local yt_dlp."""
|
||||
|
||||
def test_download_extracts_video_id_from_various_url_formats(self):
|
||||
"""Should handle youtube.com, youtu.be, shorts, etc."""
|
||||
|
||||
def test_download_returns_correct_gcs_path_format(self):
|
||||
"""Should return path without gs:// prefix."""
|
||||
|
||||
def test_download_handles_remote_failure_gracefully(self):
|
||||
"""If remote fails, should raise clear error, not fall back silently."""
|
||||
```
|
||||
|
||||
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadFlow:
|
||||
@pytest.mark.integration
|
||||
async def test_audio_search_youtube_selection_downloads_correctly(self):
|
||||
"""Full flow: search → select YouTube → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_direct_url_submission_downloads_correctly(self):
|
||||
"""Full flow: submit YouTube URL → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_youtube_download_with_remote_flacfetch(self):
|
||||
"""Verify remote flacfetch integration works end-to-end."""
|
||||
```
|
||||
|
||||
#### Flacfetch Tests (in flacfetch repo)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadByIdEndpoint:
|
||||
def test_download_by_id_with_youtube_source(self):
|
||||
"""POST /download-by-id with source_name=YouTube works."""
|
||||
|
||||
def test_download_by_id_uploads_to_gcs_when_requested(self):
|
||||
"""When gcs_path provided, should upload and return gs:// path."""
|
||||
|
||||
def test_download_by_id_handles_bot_detection_error(self):
|
||||
"""Should return clear error if YouTube blocks the request."""
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
|
||||
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
|
||||
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
|
||||
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
|
||||
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
|
||||
2. **Phase 3** changes the URL submission flow - may need careful rollout
|
||||
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
|
||||
4. **Phase 5** tests should be written alongside each phase
|
||||
|
||||
## Quick Fix vs Full Refactor
|
||||
|
||||
### Quick Fix (do now)
|
||||
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
|
||||
- This unblocks the "search → select YouTube" flow
|
||||
|
||||
### Full Refactor (this plan)
|
||||
- Consolidate all YouTube downloads into `YouTubeDownloadService`
|
||||
- Fix the "direct URL submission" flow to use remote flacfetch
|
||||
- Remove dead code and simplify branching
|
||||
- Add comprehensive tests
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing flows | Comprehensive tests before deployment |
|
||||
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
|
||||
| YouTube bot detection | Remote flacfetch with cookies is the solution |
|
||||
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **One code path** for YouTube downloads in cloud backend
|
||||
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
|
||||
3. **All tests pass** with >80% coverage on new code
|
||||
4. **Production verified** with real YouTube downloads
|
||||
5. **No dead code** remaining in the codebase
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
|
||||
2. Should direct URL submission be synchronous (wait for download) or async (background)?
|
||||
3. How do we handle YouTube cookies management for remote flacfetch?
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
|
||||
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
|
||||
@@ -1,111 +0,0 @@
|
||||
# Plan: Credit Refund on Job Deletion and Cancellation
|
||||
|
||||
**Created:** 2026-03-04
|
||||
**Branch:** feat/sess-20260304-1308-investigate-user-credits
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
When a user deletes or cancels a job, their credit is not refunded. This was discovered when user john2011@sky.com deleted a job 16 seconds after creation (accidental duplicate) and lost the credit. Job *failures* already auto-refund via `mark_job_failed()` → `refund_credit()`, but deletions and cancellations do not.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Job failures already refund credits (existing behavior, no change needed)
|
||||
- [ ] Job cancellation refunds 1 credit
|
||||
- [ ] Job deletion refunds 1 credit (if not already completed)
|
||||
- [ ] No refund for completed jobs (user got their video)
|
||||
- [ ] No refund for admin-owned jobs (consistent with failure refund logic)
|
||||
- [ ] Idempotency: prevent double-refund if a job is cancelled then deleted, or fails then deleted
|
||||
- [ ] Credit transaction records the reason clearly ("job_cancelled", "job_deleted")
|
||||
- [ ] Refund failure must not block the cancel/delete operation
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Key Insight: Idempotency via Job Field
|
||||
|
||||
The current `refund_credit()` has no guard against double-refund for the same job. Since a job can transition through cancelled → deleted, or failed → deleted, we need a way to track that a refund was already issued.
|
||||
|
||||
**Approach**: Add a `credit_refunded` boolean field to the Job model. Check it before refunding. This is simpler and more reliable than scanning the user's transaction history.
|
||||
|
||||
### What Should and Shouldn't Be Refunded
|
||||
|
||||
| Job Status at Time of Action | Delete | Cancel | Notes |
|
||||
|------------------------------|--------|--------|-------|
|
||||
| PENDING through pre-COMPLETE | Refund | Refund | Job didn't deliver value |
|
||||
| COMPLETE / PREP_COMPLETE | No refund | N/A (can't cancel terminal) | User got their output |
|
||||
| FAILED | No refund (already refunded) | N/A (can't cancel terminal) | Already refunded by fail_job |
|
||||
| CANCELLED | No refund (already refunded) | N/A (already cancelled) | Already refunded by cancel |
|
||||
|
||||
For deletion: we need to read the job status *before* deleting it to decide whether to refund. The delete endpoint already fetches the job for ownership check, so this is straightforward.
|
||||
|
||||
### Refund Helper
|
||||
|
||||
Extract the refund logic (currently in `mark_job_failed`) into a reusable private method `_refund_credit_for_job()` that:
|
||||
1. Checks `job.credit_refunded` — skip if already refunded
|
||||
2. Checks `job.user_email` — skip if no user
|
||||
3. Checks if admin — skip if admin job
|
||||
4. Calls `user_service.refund_credit()` with appropriate reason
|
||||
5. Sets `credit_refunded = True` on the job (for cancel; for delete the job is about to be removed but we set it anyway for the brief window)
|
||||
6. Wraps everything in try/except so failures don't block the parent operation
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [ ] **Add `credit_refunded` field to Job model** (`backend/models/job.py`)
|
||||
- Add `credit_refunded: bool = False` to the Job model
|
||||
- No migration needed — Firestore is schemaless, defaults to False for existing docs
|
||||
|
||||
2. [ ] **Extract refund helper in JobManager** (`backend/services/job_manager.py`)
|
||||
- Create `_refund_credit_for_job(self, job_id: str, job: Job, reason: str) -> bool`
|
||||
- Move the refund logic from `mark_job_failed` (lines 821-839) into this helper
|
||||
- Add `credit_refunded` check at the top
|
||||
- Update `mark_job_failed` to call the helper with `reason="job_failed"`
|
||||
|
||||
3. [ ] **Add refund to `cancel_job()`** (`backend/services/job_manager.py`)
|
||||
- After setting status to CANCELLED, call `_refund_credit_for_job(job_id, job, reason="job_cancelled")`
|
||||
- The job object is already fetched at line 1017
|
||||
|
||||
4. [ ] **Add refund to `delete_job()`** (`backend/services/job_manager.py`)
|
||||
- Need to fetch the job (already done when `delete_files=True`, but not always)
|
||||
- Before deletion, check if job status is NOT in `[COMPLETE, PREP_COMPLETE, FAILED, CANCELLED]`
|
||||
- If eligible, call `_refund_credit_for_job(job_id, job, reason="job_deleted")`
|
||||
- For FAILED/CANCELLED: skip because refund was already issued during that transition
|
||||
- For COMPLETE/PREP_COMPLETE: skip because user received their output
|
||||
|
||||
5. [ ] **Update `mark_job_failed` to set `credit_refunded`**
|
||||
- The helper handles this, but also update the Firestore doc
|
||||
|
||||
6. [ ] **Add unit tests** (`backend/tests/test_credit_enforcement.py`)
|
||||
- Test: cancel_job refunds credit for non-admin user
|
||||
- Test: cancel_job skips refund for admin jobs
|
||||
- Test: delete_job refunds credit for non-complete, non-terminal jobs
|
||||
- Test: delete_job does NOT refund for completed jobs
|
||||
- Test: delete_job does NOT double-refund for already-failed jobs
|
||||
- Test: delete_job does NOT double-refund for already-cancelled jobs
|
||||
- Test: refund failure doesn't block cancel/delete
|
||||
- Test: credit_refunded flag prevents double refund
|
||||
|
||||
7. [ ] **Backfill: Refund john2011@sky.com** (manual one-time)
|
||||
- Grant 1 credit for the deleted job 40dcca89 via admin API or direct Firestore update
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `credit_refunded: bool = False` field |
|
||||
| `backend/services/job_manager.py` | Modify | Extract `_refund_credit_for_job()` helper, add refund to `cancel_job()` and `delete_job()` |
|
||||
| `backend/tests/test_credit_enforcement.py` | Modify | Add tests for cancel/delete refund scenarios |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: Extend `test_credit_enforcement.py` with new test classes `TestCreditRefundOnJobCancellation` and `TestCreditRefundOnJobDeletion`, following the exact pattern of existing `TestCreditRefundOnJobFailure`
|
||||
- **Manual verification**: After deploy, cancel and delete test jobs and verify credit balance updates
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Should we notify the user (email/in-app) when a credit is refunded? (Suggest: no, keep it simple for now — the credit just appears back in their balance)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- The `credit_refunded` field is additive (defaults to False for old jobs)
|
||||
- If issues arise, revert the PR — old behavior (no refund on cancel/delete) is restored
|
||||
- Any refunds already issued are just bonus credits to users, no harm done
|
||||
@@ -1,204 +0,0 @@
|
||||
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
|
||||
|
||||
**Created:** 2026-01-19
|
||||
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
|
||||
|
||||
This plan covers:
|
||||
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
|
||||
2. Archiving the GitHub repo
|
||||
3. Deprecating the PyPI package (via README update published as final version)
|
||||
4. Reorganizing vendored code to a permanent location in karaoke-gen
|
||||
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
|
||||
- [ ] python-lyrics-transcriber GitHub repo archived
|
||||
- [ ] Final PyPI release with deprecation notice in README
|
||||
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
|
||||
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
|
||||
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
|
||||
- [ ] Tests pass after reorganization
|
||||
- [ ] karaoke-gen pyproject.toml updated for new path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### New Path Structure
|
||||
|
||||
Move the vendored code to `libs/lyrics_transcriber/`:
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── libs/
|
||||
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
|
||||
│ ├── __init__.py
|
||||
│ ├── cli/
|
||||
│ ├── core/
|
||||
│ ├── correction/
|
||||
│ ├── lyrics/
|
||||
│ ├── output/
|
||||
│ ├── review/
|
||||
│ ├── storage/
|
||||
│ ├── transcribers/
|
||||
│ ├── types.py
|
||||
│ └── utils/
|
||||
├── karaoke_gen/
|
||||
├── backend/
|
||||
├── frontend/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Why `libs/lyrics_transcriber/` instead of alternatives?
|
||||
|
||||
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
|
||||
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
|
||||
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
|
||||
|
||||
### pyproject.toml Changes
|
||||
|
||||
```toml
|
||||
packages = [
|
||||
{ include = "karaoke_gen" },
|
||||
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
|
||||
{ include = "backend" }
|
||||
]
|
||||
include = [
|
||||
"karaoke_gen/nextjs_frontend/out",
|
||||
"karaoke_gen/nextjs_frontend/out/**/*",
|
||||
"libs/lyrics_transcriber/output/fonts/*", # Updated path
|
||||
"libs/lyrics_transcriber/output/cdgmaker/images/*",
|
||||
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
|
||||
]
|
||||
```
|
||||
|
||||
### Files to Remove from Vendored Code
|
||||
|
||||
From current `lyrics_transcriber_temp/`:
|
||||
- `pyproject.toml` - Package config for standalone release
|
||||
- `poetry.lock` - Lock file for standalone package
|
||||
- `README.md` - Will be replaced by deprecation notice
|
||||
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
|
||||
- `Dockerfile` - Standalone container build
|
||||
- `.github/` - CI/CD for standalone package
|
||||
- `.cursor/` - Cursor AI config
|
||||
- `.specify/` - Specify config
|
||||
- `.gitignore` - Not needed (karaoke-gen has its own)
|
||||
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
|
||||
- `scripts/` - Development scripts for standalone package
|
||||
- `specs/` - Specification files
|
||||
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
|
||||
|
||||
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update and Archive External Project (Manual Steps)
|
||||
|
||||
1. [ ] Clone python-lyrics-transcriber repo locally
|
||||
2. [ ] Create deprecation notice README:
|
||||
```markdown
|
||||
# python-lyrics-transcriber (DEPRECATED)
|
||||
|
||||
**This project has been deprecated and archived.**
|
||||
|
||||
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
|
||||
|
||||
## Migration
|
||||
|
||||
For karaoke video generation with synchronized lyrics:
|
||||
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
|
||||
- Web app: https://gen.nomadkaraoke.com
|
||||
|
||||
## Historical Context
|
||||
|
||||
This library was originally developed as a standalone tool for creating synchronized
|
||||
lyrics files. It has now been integrated into the karaoke-gen platform which provides
|
||||
a complete end-to-end solution for karaoke video generation.
|
||||
|
||||
The last standalone version was 0.81.0. No further releases will be made to PyPI.
|
||||
```
|
||||
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
|
||||
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
|
||||
|
||||
### Phase 2: Reorganize Vendored Code in karaoke-gen
|
||||
|
||||
5. [ ] Create `libs/` directory
|
||||
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
|
||||
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
|
||||
8. [ ] Update `pyproject.toml` packages and include paths
|
||||
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
|
||||
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
|
||||
|
||||
### Phase 3: Verify and Test
|
||||
|
||||
10. [ ] Run `poetry install` to verify package configuration
|
||||
11. [ ] Run `make test` to ensure all tests pass
|
||||
12. [ ] Verify imports work:
|
||||
```python
|
||||
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
```
|
||||
|
||||
### Phase 4: Handle Tests from lyrics_transcriber
|
||||
|
||||
13. [ ] Evaluate lyrics_transcriber tests:
|
||||
- **74 test files** in `lyrics_transcriber_temp/tests/`
|
||||
- These test the internal lyrics_transcriber library functionality
|
||||
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
|
||||
- Skip integration/performance/manual tests that were for standalone development
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
|
||||
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
|
||||
| `pyproject.toml` | Modify | Update package paths |
|
||||
| `scripts/benchmark_encoding.py` | Modify | Update import path |
|
||||
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Immediate Testing
|
||||
- Run `make test` after reorganization - all existing tests must pass
|
||||
- Verify Poetry can build/install the package correctly
|
||||
|
||||
### Test Migration Decision
|
||||
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
|
||||
- `unit/correction/` - Correction logic tests (important)
|
||||
- `unit/lyrics/` - Lyrics provider tests
|
||||
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
|
||||
- `integration/` - Integration tests
|
||||
- `performance/` - Performance benchmarks
|
||||
|
||||
**Recommendation:** Migrate a subset of critical unit tests, particularly:
|
||||
- `test_corrector.py` - Core correction logic
|
||||
- `test_anchor_sequence*.py` - Anchor sequence matching
|
||||
- Handler tests in `handlers/`
|
||||
|
||||
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
|
||||
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
|
||||
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Git revert the consolidation commits
|
||||
2. `lyrics_transcriber_temp/` can be restored from git history
|
||||
3. The external python-lyrics-transcriber repo can be unarchived if needed
|
||||
|
||||
## Notes
|
||||
|
||||
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
|
||||
- Only the internal project structure changes, not the API
|
||||
- This is primarily a cleanup/organization task with some external repo admin work
|
||||
@@ -1,110 +0,0 @@
|
||||
# Plan: Disable Auto-Correction (Agentic and Heuristic)
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1914-disable-auto-correction
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Disable agentic AI correction (LLM-based via Gemini)
|
||||
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
|
||||
- [x] Preserve the transcription → human review → video generation flow
|
||||
- [x] Minimal code changes (configuration-based if possible)
|
||||
- [x] Easy to re-enable in the future if correction quality improves
|
||||
|
||||
## Technical Approach
|
||||
|
||||
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
|
||||
|
||||
**Current behavior:**
|
||||
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
|
||||
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
|
||||
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
|
||||
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
|
||||
|
||||
**Proposed changes:**
|
||||
|
||||
### Option A: Add a new `skip_correction` config (Recommended)
|
||||
Add a single boolean `skip_correction` setting that:
|
||||
- When `true`: Skips the entire correction process, returning raw transcription
|
||||
- When `false` (future): Falls back to existing agentic/heuristic behavior
|
||||
|
||||
This is cleaner than disabling both systems separately because:
|
||||
1. Single config controls both
|
||||
2. Clear intent ("skip correction" vs "disable two different systems")
|
||||
3. Easy to understand for future maintainers
|
||||
|
||||
### Option B: Just disable both individually
|
||||
- Set `use_agentic_ai` default to `false`
|
||||
- Disable default heuristic handlers by setting `enabled_handlers=[]`
|
||||
|
||||
Option A is cleaner, but Option B requires fewer code changes.
|
||||
|
||||
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Add `skip_correction` config to backend/config.py**
|
||||
- Add `skip_correction: bool` setting, default `true`
|
||||
- Comment explaining this disables both agentic and heuristic correction
|
||||
|
||||
2. [x] **Update lyrics_worker.py to pass skip_correction**
|
||||
- Pass the config value through to LyricsProcessor/LyricsTranscriber
|
||||
|
||||
3. [x] **Update corrector.py to support skip_correction**
|
||||
- When `skip_correction=True`, return immediately with uncorrected segments
|
||||
- No handler processing, no agentic AI
|
||||
- Still need to do anchor/gap finding? No - skip entirely
|
||||
|
||||
4. [x] **Update LyricsProcessor to support skip_correction**
|
||||
- Pass through to LyricsTranscriber
|
||||
|
||||
5. [x] **Update LyricsTranscriber controller to support skip_correction**
|
||||
- Pass through to corrector or skip corrector.run() entirely
|
||||
|
||||
6. [x] **Update docs/README.md**
|
||||
- Note that auto-correction is disabled
|
||||
- Update "Agentic AI correction" status to "Disabled"
|
||||
|
||||
7. [x] **Run tests to verify no regressions**
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
|
||||
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
|
||||
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
|
||||
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
|
||||
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
|
||||
| `docs/README.md` | Modify | Update status to reflect disabled correction |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: No new tests needed - this is a config change
|
||||
- **Existing tests**: Run `make test` to ensure no regressions
|
||||
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the approach is straightforward.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
To re-enable auto-correction:
|
||||
1. Set `SKIP_CORRECTION=false` environment variable, OR
|
||||
2. Change the default back to `false` in `backend/config.py`
|
||||
|
||||
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
|
||||
|
||||
## Alternative Considered
|
||||
|
||||
Instead of adding a new config, we could:
|
||||
1. Set `USE_AGENTIC_AI=false` (disables agentic)
|
||||
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
|
||||
|
||||
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
|
||||
@@ -1,99 +0,0 @@
|
||||
# Plan: Encoding Worker Immutable Deployment Pattern
|
||||
|
||||
## Status: In Progress
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The GCE encoding worker has recurring version mismatch issues because:
|
||||
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
|
||||
2. Wheel selection uses `sort -V` which can fail if image is outdated
|
||||
3. CI only restarts service - doesn't verify version strictly
|
||||
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
|
||||
5. 51 old wheel files accumulate in GCS bucket
|
||||
|
||||
## Solution: Immutable Deployment Pattern
|
||||
|
||||
### Core Principles
|
||||
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
|
||||
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
|
||||
3. **Version Manifest**: CI writes expected version, service MUST match
|
||||
4. **Strict Verification**: CI FAILS if deployed version doesn't match
|
||||
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: GCS Structure & Scripts
|
||||
|
||||
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
|
||||
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
|
||||
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
|
||||
- [x] **Step 1.4**: Create README.md with documentation
|
||||
|
||||
### Phase 2: CI Workflow Updates
|
||||
|
||||
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
|
||||
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
|
||||
- [x] **Step 2.3**: Update CI to write version.txt manifest
|
||||
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
|
||||
|
||||
### Phase 3: Packer Image Updates
|
||||
|
||||
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
|
||||
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
|
||||
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
|
||||
|
||||
### Phase 4: Documentation & Cleanup
|
||||
|
||||
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
|
||||
- [x] **Step 4.2**: Update infrastructure/packer/README.md
|
||||
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
|
||||
|
||||
## New GCS Structure
|
||||
|
||||
```
|
||||
gs://karaoke-gen-storage-nomadkaraoke/
|
||||
├── wheels/
|
||||
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
|
||||
│ └── karaoke_gen-{version}.whl # Versioned for rollback
|
||||
│
|
||||
└── encoding-worker/
|
||||
├── bootstrap.sh # Reference copy
|
||||
├── startup.sh # CI-managed, downloaded on boot
|
||||
└── version.txt # Expected version number
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Purpose |
|
||||
|------|--------|---------|
|
||||
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
|
||||
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
|
||||
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
|
||||
| `.github/workflows/ci.yml` | Modify | New deployment flow |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit test**: Verify startup.sh logic locally
|
||||
2. **Integration test**: After merge, verify:
|
||||
- CI uploads files to correct GCS paths
|
||||
- Service starts with correct version
|
||||
- Health endpoint returns expected version
|
||||
3. **Manual verification**: Check VM logs after first deployment
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
|
||||
2. Previous versioned wheels remain in GCS for manual rollback
|
||||
3. Old Packer images remain for VM rollback
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - design is complete.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bootstrap.sh is ~10 lines and should never need changes
|
||||
- All logic changes go through startup.sh in GCS
|
||||
- Version verification is strict - CI fails on mismatch
|
||||
@@ -1,166 +0,0 @@
|
||||
# Plan: Fix YouTube Remote Download Missing GCS Path
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
|
||||
|
||||
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
|
||||
|
||||
## Root Cause
|
||||
|
||||
In `backend/api/routes/audio_search.py` lines 354-363:
|
||||
|
||||
```python
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
# <-- MISSING: gcs_path parameter
|
||||
)
|
||||
```
|
||||
|
||||
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
|
||||
- [x] Backend should receive a `gs://` path from flacfetch
|
||||
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
|
||||
- [x] Maintain backwards compatibility (local-only deployments should still work)
|
||||
- [x] Add test coverage for this code path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
|
||||
|
||||
The current code structure has two branches:
|
||||
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
|
||||
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
|
||||
|
||||
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
|
||||
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
|
||||
- If remote disabled: download locally and upload to GCS from Cloud Run
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
|
||||
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
|
||||
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
|
||||
4. [x] Update logging to clarify the download path being used
|
||||
5. [x] Bump version in `pyproject.toml`
|
||||
6. [x] Run tests to verify no regressions
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
|
||||
| `pyproject.toml` | Modify | Bump version |
|
||||
|
||||
## Detailed Code Changes
|
||||
|
||||
### audio_search.py (lines 349-393)
|
||||
|
||||
Current structure:
|
||||
```python
|
||||
else:
|
||||
# Local download (YouTube or fallback) <-- WRONG: not always local!
|
||||
temp_dir = tempfile.mkdtemp(...)
|
||||
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
result = audio_search_service.download_by_id(
|
||||
...
|
||||
output_dir=temp_dir, # <-- BUG: unused for remote
|
||||
# Missing gcs_path!
|
||||
)
|
||||
```
|
||||
|
||||
Fixed structure:
|
||||
```python
|
||||
else:
|
||||
# Non-torrent download (YouTube or fallback)
|
||||
|
||||
if is_remote_enabled and source_id and source_name:
|
||||
# Remote download via flacfetch VM - upload directly to GCS
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination, # <-- FIX: Add this!
|
||||
)
|
||||
|
||||
# Handle GCS path response (same logic as torrent branch)
|
||||
if result.filepath.startswith("gs://"):
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: upload manually
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (remote disabled or no source_id)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
# ... existing local download logic ...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
|
||||
|
||||
```python
|
||||
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads via remote flacfetch include GCS path.
|
||||
|
||||
This is the same bug pattern as the torrent download bug fixed in 2025-12:
|
||||
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
|
||||
"""
|
||||
# Setup: YouTube source with remote enabled
|
||||
# Verify: download_by_id called with gcs_path parameter
|
||||
```
|
||||
|
||||
### Manual Testing (post-deploy)
|
||||
1. Create a job and select a YouTube audio source
|
||||
2. Verify job completes successfully
|
||||
3. Check GCS path is properly set in job data
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the fix is straightforward and follows the pattern established for torrent sources.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
This is a minor code change. If issues occur:
|
||||
1. Revert the PR
|
||||
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
@@ -1,407 +0,0 @@
|
||||
# Plan: Add Retry Logic for Flacfetch Download Failures
|
||||
|
||||
**Created:** 2026-02-10
|
||||
**Branch:** feat/sess-20260210-1158-investigate-job-749141f8
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Job 749141f8 failed with error: "Audio download failed: Remote download by ID failed: Download by ID request failed:"
|
||||
|
||||
Investigation revealed **multiple related issues** when flacfetch network calls fail:
|
||||
|
||||
1. **No retry logic** - A single network hiccup causes immediate job failure
|
||||
2. **Missing source info** - When download fails, source_id/source_name aren't saved to job
|
||||
3. **Broken retry button** - Can't retry because no source info was persisted
|
||||
4. **Silent UI failures** - Frontend doesn't show retry error messages to user
|
||||
|
||||
### Root Cause
|
||||
|
||||
Audio search flow saves selection info to `state_data`, but doesn't save download parameters (source_id, source_name, etc.) to the job document until AFTER download succeeds. If the flacfetch call fails (network error), the job fails with no way to retry.
|
||||
|
||||
### Solution
|
||||
|
||||
This plan fixes all four issues:
|
||||
- Add retry logic to flacfetch client (network resilience)
|
||||
- Persist download info before attempting download (enables retry)
|
||||
- Update retry endpoint to handle audio search jobs
|
||||
- Show error messages in frontend when retry fails
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements - Flacfetch Retry Logic
|
||||
- [x] Retry `download_by_id()` on `httpx.RequestError` (network failures)
|
||||
- [x] Use exponential backoff spread over ~5 minutes
|
||||
- [x] Configurable max retries via environment variable
|
||||
- [x] Log each retry attempt with context
|
||||
- [x] Do NOT retry on HTTP 4xx errors (client errors - immediate failure)
|
||||
- [x] Do retry on HTTP 5xx errors (server errors - transient)
|
||||
|
||||
### Functional Requirements - Persist Download Info
|
||||
- [x] Save source_id, source_name, target_file, download_url to job document
|
||||
- [x] Save BEFORE attempting download (not after)
|
||||
- [x] Store in top-level fields for easy retry access
|
||||
- [x] Preserve existing state_data fields (selected_audio_index, etc.)
|
||||
|
||||
### Functional Requirements - Retry Endpoint
|
||||
- [x] Add case for audio search jobs in `/jobs/{id}/retry` endpoint
|
||||
- [x] Check for saved download params (source_id, source_name)
|
||||
- [x] Re-download using saved params
|
||||
- [x] Clear error state before retrying
|
||||
- [x] Trigger appropriate workers after successful download
|
||||
|
||||
### Functional Requirements - Frontend Error Handling
|
||||
- [x] Show toast notification when retry fails
|
||||
- [x] Display actual error message from API
|
||||
- [x] Don't just console.error - make it visible to user
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [x] Minimal code changes (use existing patterns)
|
||||
- [x] Clear error messages that include retry context
|
||||
- [x] Backwards compatible (no breaking changes)
|
||||
- [x] Should not significantly increase total timeout duration
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decisions
|
||||
|
||||
**Option 1: Tenacity library (Recommended)**
|
||||
- ✅ Industry standard, well-tested
|
||||
- ✅ Declarative retry configuration
|
||||
- ✅ Built-in exponential backoff
|
||||
- ✅ Already used in project (check `poetry.lock`)
|
||||
- ❌ New dependency if not already present
|
||||
|
||||
**Option 2: Manual retry loop**
|
||||
- ✅ No new dependencies
|
||||
- ✅ Full control over logic
|
||||
- ❌ More code to maintain
|
||||
- ❌ Easy to get wrong (edge cases)
|
||||
|
||||
**Decision:** Use **tenacity** if already in dependencies, otherwise implement manual retry loop.
|
||||
|
||||
### Retry Strategy
|
||||
|
||||
```python
|
||||
# Retry on network errors only (RequestError)
|
||||
# Do NOT retry on:
|
||||
# - HTTP 4xx (client errors like 400, 401, 404)
|
||||
# - Already completed downloads
|
||||
# Do retry on:
|
||||
# - httpx.RequestError (connection timeout, refused, DNS failure)
|
||||
# - HTTP 5xx (server errors - service temporarily down)
|
||||
|
||||
Max retries: 8 (total 9 attempts)
|
||||
Backoff: Exponential with cap, multiplier=2, min=10s, max=60s
|
||||
- Attempt 1: immediate
|
||||
- Attempt 2: wait 10s
|
||||
- Attempt 3: wait 20s
|
||||
- Attempt 4: wait 40s
|
||||
- Attempt 5: wait 60s (capped)
|
||||
- Attempt 6: wait 60s (capped)
|
||||
- Attempt 7: wait 60s (capped)
|
||||
- Attempt 8: wait 60s (capped)
|
||||
- Attempt 9: wait 60s (capped)
|
||||
Total max time: ~370s (6+ minutes) of backoff + request timeouts
|
||||
|
||||
This gives the flacfetch service time to recover from restarts,
|
||||
network issues, or temporary VM problems.
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Add to `backend/config.py`:
|
||||
```python
|
||||
# Flacfetch client retry settings
|
||||
flacfetch_retry_max_attempts: int = int(os.getenv("FLACFETCH_RETRY_MAX_ATTEMPTS", "9")) # Total attempts (1 initial + 8 retries)
|
||||
flacfetch_retry_min_wait: float = float(os.getenv("FLACFETCH_RETRY_MIN_WAIT", "10.0")) # seconds
|
||||
flacfetch_retry_max_wait: float = float(os.getenv("FLACFETCH_RETRY_MAX_WAIT", "60.0")) # seconds
|
||||
```
|
||||
|
||||
### Error Message Improvements
|
||||
|
||||
Change from:
|
||||
```
|
||||
Download by ID request failed:
|
||||
```
|
||||
|
||||
To:
|
||||
```
|
||||
Download by ID request failed after 4 attempts: [ConnectTimeout] Connection to http://10.0.0.5:8080 timed out
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Check Dependencies
|
||||
1. [ ] Check if `tenacity` is in `pyproject.toml` dependencies
|
||||
2. [ ] If not present, add `tenacity = "^8.2.0"` to dependencies
|
||||
3. [ ] Run `poetry lock` if dependencies changed
|
||||
|
||||
### Phase 2: Add Configuration
|
||||
4. [ ] Add retry configuration to `backend/config.py`
|
||||
- `flacfetch_retry_max_attempts = 9`
|
||||
- `flacfetch_retry_min_wait = 10.0`
|
||||
- `flacfetch_retry_max_wait = 60.0`
|
||||
|
||||
### Phase 3: Implement Flacfetch Retry Logic
|
||||
5. [ ] Import tenacity in `backend/services/flacfetch_client.py`
|
||||
6. [ ] Create retry decorator with:
|
||||
- Retry on `httpx.RequestError`
|
||||
- Retry on `httpx.HTTPStatusError` with 5xx status
|
||||
- Stop on `httpx.HTTPStatusError` with 4xx status
|
||||
- Exponential backoff with cap at 60s
|
||||
- Before-sleep logging callback
|
||||
7. [ ] Apply decorator to `download_by_id()` method
|
||||
8. [ ] Improve error messages to include:
|
||||
- Number of attempts made
|
||||
- Original exception type and message
|
||||
|
||||
### Phase 4: Persist Download Info Before Download
|
||||
9. [ ] Update `audio_search.py:_download_and_start_processing()`
|
||||
10. [ ] After getting source info from selected result (line 307-311)
|
||||
11. [ ] **Save to job document BEFORE download attempt:**
|
||||
```python
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_source_type': 'audio_search',
|
||||
'source_name': source_name,
|
||||
'source_id': source_id,
|
||||
'target_file': target_file,
|
||||
'download_url': download_url,
|
||||
})
|
||||
```
|
||||
12. [ ] Then proceed with download (existing code)
|
||||
13. [ ] Add to Job model if needed (check if fields exist)
|
||||
|
||||
### Phase 5: Update Retry Endpoint for Audio Search
|
||||
14. [ ] Update `backend/api/routes/jobs.py:retry_job()`
|
||||
15. [ ] Add new case before "no input audio" error (line ~1088)
|
||||
16. [ ] Check if job has saved download params:
|
||||
```python
|
||||
elif job.audio_source_type == 'audio_search' and job.source_id:
|
||||
# Has audio search download info - retry download
|
||||
```
|
||||
17. [ ] Re-call audio search download logic with saved params
|
||||
18. [ ] Or refactor download into reusable function
|
||||
|
||||
### Phase 6: Frontend Error Handling
|
||||
19. [ ] Update `frontend/components/job/JobActions.tsx`
|
||||
20. [ ] Add toast notification library (check if exists)
|
||||
21. [ ] In `handleRetry()` catch block, show toast:
|
||||
```typescript
|
||||
catch (error) {
|
||||
const message = error.response?.data?.detail || "Failed to retry job"
|
||||
toast.error(message)
|
||||
console.error("Failed to retry job:", error)
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 7: Testing - Flacfetch Retry
|
||||
22. [ ] Add unit tests to `backend/tests/test_flacfetch_client.py`
|
||||
23. [ ] Test retry on RequestError (mock fails N times then succeeds)
|
||||
24. [ ] Test no retry on 4xx errors
|
||||
25. [ ] Test retry on 5xx errors
|
||||
26. [ ] Test max retries exhausted
|
||||
27. [ ] Verify logging output includes attempt numbers
|
||||
|
||||
### Phase 8: Testing - End-to-End
|
||||
28. [ ] Test audio search job with flacfetch down
|
||||
29. [ ] Verify source info is saved even when download fails
|
||||
30. [ ] Verify retry button works after source info saved
|
||||
31. [ ] Test retry succeeds when flacfetch comes back up
|
||||
32. [ ] Verify error toast appears when retry fails
|
||||
|
||||
### Phase 9: Documentation
|
||||
33. [ ] Update `backend/services/flacfetch_client.py` docstring
|
||||
34. [ ] Add to `docs/LESSONS-LEARNED.md`:
|
||||
- "Always add retries for external service calls"
|
||||
- "Save operation parameters before attempting operations"
|
||||
- "Persist enough state to enable retry from any failure point"
|
||||
35. [ ] Update `docs/ARCHITECTURE.md` if needed
|
||||
36. [ ] Document the new job fields in API.md
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add retry configuration variables |
|
||||
| `backend/services/flacfetch_client.py` | Modify | Add retry logic to `download_by_id()` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Save download info before attempting download |
|
||||
| `backend/api/routes/jobs.py` | Modify | Add audio search case to retry endpoint |
|
||||
| `backend/models/job.py` | Modify (maybe) | Add fields: source_name, source_id, target_file, download_url |
|
||||
| `frontend/components/job/JobActions.tsx` | Modify | Show error toast when retry fails |
|
||||
| `backend/tests/test_flacfetch_client.py` | Modify | Add retry tests |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Test download info persistence |
|
||||
| `backend/tests/test_jobs.py` | Modify | Test retry with audio search jobs |
|
||||
| `pyproject.toml` | Modify (maybe) | Add tenacity if not present |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document patterns |
|
||||
| `docs/API.md` | Modify | Document new job fields |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (`backend/tests/test_flacfetch_client.py`)
|
||||
|
||||
1. **Test retry on RequestError**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_network_error():
|
||||
"""Should retry 3 times on connection errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
# Fail 2 times, then succeed
|
||||
mock_post.side_effect = [
|
||||
httpx.RequestError("Connection timeout"),
|
||||
httpx.RequestError("Connection refused"),
|
||||
Mock(json=lambda: {"download_id": "test123"}, status_code=200),
|
||||
]
|
||||
|
||||
result = await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert result == "test123"
|
||||
assert mock_post.call_count == 3
|
||||
```
|
||||
|
||||
2. **Test no retry on 4xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_no_retry_on_client_error():
|
||||
"""Should NOT retry on 400/401/404 errors."""
|
||||
client = FlacfetchClient(base_url="http://test", api_key="key")
|
||||
|
||||
with patch("httpx.AsyncClient.post") as mock_post:
|
||||
response = Mock(status_code=404, text="Not found")
|
||||
response.raise_for_status.side_effect = httpx.HTTPStatusError(
|
||||
"Not found", request=Mock(), response=response
|
||||
)
|
||||
mock_post.return_value = response
|
||||
|
||||
with pytest.raises(FlacfetchServiceError, match="404"):
|
||||
await client.download_by_id("YouTube", "abc123")
|
||||
|
||||
assert mock_post.call_count == 1 # No retries
|
||||
```
|
||||
|
||||
3. **Test retry on 5xx errors**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_retries_on_server_error():
|
||||
"""Should retry on 500/503 errors."""
|
||||
# Similar to test 1 but with HTTP 503 errors
|
||||
```
|
||||
|
||||
4. **Test max retries exhausted**
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_download_by_id_fails_after_max_retries():
|
||||
"""Should fail with clear message after max retries."""
|
||||
# All attempts fail, verify error message includes attempt count
|
||||
```
|
||||
|
||||
### Integration Tests
|
||||
|
||||
5. **Test with actual flacfetch service** (if available in test environment)
|
||||
- Verify retry doesn't break normal operation
|
||||
- Test with deliberate network issues
|
||||
|
||||
### Manual Testing
|
||||
|
||||
6. **Simulate network failure**
|
||||
```bash
|
||||
# Temporarily stop flacfetch VM
|
||||
gcloud compute instances stop flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Submit a test job
|
||||
# Observe logs showing retry attempts
|
||||
|
||||
# Restart VM during retries
|
||||
gcloud compute instances start flacfetch-service --zone=us-central1-a
|
||||
|
||||
# Job should succeed
|
||||
```
|
||||
|
||||
7. **Test with intentional delays**
|
||||
- Add sleep in flacfetch API handler
|
||||
- Verify retries don't cause issues
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] ~~Is tenacity already in dependencies?~~ (Will check during implementation)
|
||||
- [x] ~~What's the appropriate total timeout budget?~~ - 9 attempts over ~6 minutes
|
||||
- [x] ~~Should we fix the Retry button UX in this PR?~~ - YES, it's the same root cause
|
||||
- [x] ~~Why did job 749141f8 fail before saving audio source info?~~ - Download info saved AFTER download, not before
|
||||
- [ ] Should we also add retries to `wait_for_download()`?
|
||||
- Probably NO - that already has its own polling/timeout logic
|
||||
- [ ] Should we add circuit breaker pattern?
|
||||
- Probably NO for now - keep it simple
|
||||
- Could add in future if we see cascading failures
|
||||
- [ ] Should retry logic be applied to other flacfetch methods?
|
||||
- `start_download()` - probably YES (same pattern)
|
||||
- `search()` - maybe YES (search can also have network issues)
|
||||
- `get_download_status()` - probably NO (polling already handles this)
|
||||
- **Decision**: Apply to `start_download()` too, skip `search()` for now
|
||||
- [ ] Which toast library should frontend use?
|
||||
- Check if already exists (react-hot-toast, sonner, etc.)
|
||||
- If not, add lightweight library
|
||||
- [ ] Should we save download params to state_data or top-level fields?
|
||||
- Top-level fields more visible and easier to query
|
||||
- **Decision**: Top-level fields (source_id, source_name, etc.)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If this causes issues:
|
||||
|
||||
1. **Quick rollback**: Set env var `FLACFETCH_RETRY_MAX_ATTEMPTS=1` in Cloud Run
|
||||
- Disables retries without code change
|
||||
- Can deploy in ~2 minutes
|
||||
|
||||
2. **Full rollback**: Revert the PR
|
||||
- Git revert the merge commit
|
||||
- Deploy previous version
|
||||
|
||||
3. **Monitoring**: Watch for:
|
||||
- Increased request duration to flacfetch
|
||||
- Jobs timing out at higher layer (Cloud Tasks, Cloud Run)
|
||||
- Duplicate download attempts causing issues
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Jobs with transient flacfetch failures succeed instead of failing
|
||||
- [ ] Error messages are more informative (show attempts and root cause)
|
||||
- [ ] No increase in false positives (jobs that should fail, now hang)
|
||||
- [ ] All tests pass
|
||||
- [ ] No impact on successful jobs (retry overhead is minimal)
|
||||
|
||||
## Related Issues
|
||||
|
||||
### Job 749141f8 - The Incident
|
||||
|
||||
**What happened:**
|
||||
- User searched for audio via audio search
|
||||
- Selected "Elizabeth Moen - Songbird"
|
||||
- Backend tried to download via flacfetch
|
||||
- Flacfetch call failed with network error (connection timeout/refused)
|
||||
- Job immediately failed
|
||||
- No source info saved → Retry button broken
|
||||
- User sees "Retry" button but clicking does nothing (silent 400 error)
|
||||
|
||||
**How this fix helps:**
|
||||
1. ✅ Flacfetch retry logic → Download would have succeeded after retry
|
||||
2. ✅ Persist download info → Even if all retries fail, user can retry later
|
||||
3. ✅ Retry endpoint fix → Retry button will actually work
|
||||
4. ✅ Frontend toast → User sees helpful error if retry still fails
|
||||
|
||||
**Note:** Job 749141f8 cannot be fixed retroactively (no source info saved). But future jobs with same issue will be fixable via retry button.
|
||||
|
||||
### Future Work (Not in This PR)
|
||||
|
||||
Similar pattern should be applied to other external service calls:
|
||||
- AudioShake API (lyrics transcription)
|
||||
- Modal API (audio separation)
|
||||
- YouTube direct API calls (not via flacfetch)
|
||||
- Email sending (SendGrid)
|
||||
|
||||
## References
|
||||
|
||||
- [Tenacity documentation](https://tenacity.readthedocs.io/)
|
||||
- [LESSONS-LEARNED.md](docs/LESSONS-LEARNED.md) - External service response format mismatches
|
||||
- [flacfetch README](../../flacfetch/README.md) - Understanding the service
|
||||
@@ -1,149 +0,0 @@
|
||||
# Plan: Improve Job Observability
|
||||
|
||||
**Created:** 2026-03-07
|
||||
**Branch:** feat/sess-20260307-2216-improve-job-observability
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
After shipping the "edit completed tracks" feature, we found it very difficult to trace what actually happened to edited jobs. Timeline events are too sparse (no output details), cleanup operations aren't logged to Cloud Logging or Firestore, state_data gets fully overwritten during re-processing (losing history of previous outputs), and API endpoints don't write to the job's Firestore log subcollection.
|
||||
|
||||
The goal is to make it trivially easy for future agents and humans to reconstruct the complete lifecycle of any job by reading its timeline and Firestore logs.
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
### What we found investigating two edited jobs:
|
||||
|
||||
1. **Timeline events only have `status`, `timestamp`, `progress`, `message`** — no output details like YouTube video IDs, brand codes, Dropbox links, or GDrive file IDs
|
||||
2. **Edit endpoint cleanup results never logged** — the `cleanup_results` dict is built and returned in the HTTP response but never logged to Cloud Logging or the job's Firestore log subcollection
|
||||
3. **Distribution details incomplete in logs** — YouTube logs the URL but not video_id separately; Dropbox doesn't log folder path; GDrive doesn't log file IDs
|
||||
4. **`state_data` fully overwritten on re-processing** — `video_worker.py:342-354` does `{**job.state_data, ...}`, so any `DELETE_FIELD` operations from the edit endpoint get lost when old keys are re-merged from the in-memory copy
|
||||
5. **`outputs_deleted_at` cleared by video worker** — intentionally, but loses audit trail
|
||||
6. **No Firestore job logs from API endpoints** — edit/admin endpoints only log to Cloud Run stdout, not to `jobs/{job_id}/logs/`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Add `metadata` dict to `TimelineEvent` for structured output details
|
||||
- [ ] Log all cleanup operations (success AND failure) to both Cloud Logging and Firestore job logs
|
||||
- [ ] Log distribution output details (YouTube video ID, Dropbox path, brand code, GDrive file IDs) in timeline events and Firestore logs
|
||||
- [ ] Preserve previous output details in timeline when a job is edited (snapshot before cleanup)
|
||||
- [ ] Add helper to write Firestore job logs from API endpoints (not just workers)
|
||||
- [ ] Ensure edit/admin delete endpoints write to Firestore job logs
|
||||
- [ ] No breaking changes to existing timeline format (additive only)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Add `metadata` field to `TimelineEvent`
|
||||
|
||||
The `TimelineEvent` model gains an optional `metadata: Optional[Dict[str, Any]]` field. This allows any timeline event to carry structured data without changing the schema. The field is additive — existing events without metadata continue to work.
|
||||
|
||||
Key events that gain metadata:
|
||||
- **`complete`**: `{youtube_url, youtube_video_id, dropbox_link, brand_code, gdrive_file_ids}`
|
||||
- **Edit initiated**: `{previous_outputs: {youtube_url, dropbox_link, brand_code, gdrive_files}, cleanup_results: {...}}`
|
||||
- **Outputs deleted**: `{deleted_outputs: {...}, cleanup_results: {...}}`
|
||||
|
||||
### 2. Write Firestore job logs from API endpoints
|
||||
|
||||
Create a simple helper function that API route handlers can call to append logs to the job's Firestore subcollection. Workers already have `JobLogAdapter` — we just need a lightweight equivalent for API routes:
|
||||
|
||||
```python
|
||||
def log_to_job(job_id: str, worker: str, level: str, message: str, metadata: dict = None):
|
||||
"""Write a log entry to the job's Firestore log subcollection."""
|
||||
entry = WorkerLogEntry.create(job_id, worker, level, message, metadata)
|
||||
FirestoreService().append_log_to_subcollection(job_id, entry)
|
||||
```
|
||||
|
||||
### 3. Enhance distribution logging in video_worker_orchestrator
|
||||
|
||||
After each distribution upload succeeds, log the details with structured metadata:
|
||||
- YouTube: video_id, video_url, channel
|
||||
- Dropbox: folder_path, sharing_link
|
||||
- GDrive: file_ids dict
|
||||
- Brand code: code, prefix, allocated vs recycled
|
||||
|
||||
### 4. Snapshot previous outputs on edit
|
||||
|
||||
Before the edit endpoint clears state_data distribution keys, capture the current values in the timeline event's metadata. This preserves the history permanently in the timeline array (which is append-only via `ArrayUnion`).
|
||||
|
||||
### 5. Log cleanup results from edit/admin endpoints
|
||||
|
||||
After cleanup runs, log each step's result to both Cloud Logging and the Firestore job log subcollection with structured metadata.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add `metadata` to `TimelineEvent` model
|
||||
- [ ] Add `metadata: Optional[Dict[str, Any]] = None` to `TimelineEvent` in `backend/models/job.py`
|
||||
- [ ] No migration needed — Firestore is schemaless, existing events just won't have the field
|
||||
|
||||
### Step 2: Create `log_to_job` helper
|
||||
- [ ] Add `log_to_job()` function in `backend/services/firestore_service.py` (or a small utility module)
|
||||
- [ ] Takes `job_id`, `worker`, `level`, `message`, optional `metadata`
|
||||
- [ ] Wraps `WorkerLogEntry.create()` + `append_log_to_subcollection()`
|
||||
- [ ] Catches and suppresses errors (logging should never break the caller)
|
||||
|
||||
### Step 3: Update `transition_to_state` to accept metadata
|
||||
- [ ] Add optional `timeline_metadata: Optional[Dict[str, Any]] = None` parameter to `job_manager.transition_to_state()`
|
||||
- [ ] Pass through to `update_job_status()` → `TimelineEvent` construction
|
||||
- [ ] Update `firestore_service.update_job_status()` to accept and pass `timeline_metadata`
|
||||
|
||||
### Step 4: Enhance edit endpoint logging
|
||||
- [ ] Before cleanup: snapshot current distribution outputs (`youtube_url`, `dropbox_link`, `brand_code`, `gdrive_files`) into `previous_outputs` dict
|
||||
- [ ] After each cleanup step: log result to Cloud Logging AND call `log_to_job()`
|
||||
- [ ] After all cleanup: include `previous_outputs` and `cleanup_results` in the timeline event metadata
|
||||
- [ ] The "Track edit initiated" timeline event gains metadata: `{initiated_by, previous_outputs, cleanup_results}`
|
||||
|
||||
### Step 5: Enhance admin delete-outputs endpoint logging
|
||||
- [ ] Same pattern as edit endpoint: snapshot outputs, log each cleanup step, include in timeline metadata
|
||||
- [ ] The "Outputs deleted by admin" timeline event gains metadata: `{deleted_by, deleted_outputs, cleanup_results}`
|
||||
|
||||
### Step 6: Enhance distribution logging in video_worker_orchestrator
|
||||
- [ ] YouTube upload success: log video_id and video_url with metadata `{video_id, video_url}`
|
||||
- [ ] Dropbox upload success: log folder_path and sharing_link with metadata `{folder_path, sharing_link}`
|
||||
- [ ] GDrive upload success: log file_ids with metadata `{file_ids: {...}}`
|
||||
- [ ] Brand code allocation: log with metadata `{brand_code, source: "allocated"|"recycled"}`
|
||||
|
||||
### Step 7: Add output details to `complete` timeline event
|
||||
- [ ] When `video_worker.py` calls `transition_to_state(COMPLETE)`, pass `timeline_metadata` with distribution output details
|
||||
- [ ] Include: `youtube_url`, `youtube_video_id`, `dropbox_link`, `brand_code`, `gdrive_file_ids`, `edit_count`
|
||||
|
||||
### Step 8: Tests
|
||||
- [ ] Unit test: `TimelineEvent` with metadata serializes correctly
|
||||
- [ ] Unit test: `log_to_job` writes to subcollection
|
||||
- [ ] Unit test: edit endpoint includes `previous_outputs` and `cleanup_results` in timeline event
|
||||
- [ ] Unit test: `complete` transition includes output metadata
|
||||
- [ ] Verify existing tests still pass
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `metadata` field to `TimelineEvent` |
|
||||
| `backend/services/firestore_service.py` | Modify | Add `log_to_job()` helper, update `update_job_status()` to accept timeline metadata |
|
||||
| `backend/services/job_manager.py` | Modify | Add `timeline_metadata` param to `transition_to_state()` |
|
||||
| `backend/api/routes/jobs.py` | Modify | Enhance edit endpoint with cleanup logging, output snapshots, timeline metadata |
|
||||
| `backend/api/routes/admin.py` | Modify | Enhance delete-outputs endpoint similarly |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Add structured metadata to distribution log messages |
|
||||
| `backend/workers/video_worker.py` | Modify | Pass output metadata in `complete` transition (both orchestrator and legacy paths) |
|
||||
| `backend/tests/test_edit_completed_track.py` | Modify | Verify timeline metadata in edit tests |
|
||||
| `backend/tests/test_observability.py` | Create | Tests for TimelineEvent metadata and log_to_job |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: TimelineEvent metadata serialization, log_to_job helper, edit endpoint timeline metadata assertions
|
||||
- **Existing test verification**: `make test` must pass — changes are additive so existing tests should be unaffected
|
||||
- **No E2E tests needed**: This is purely backend logging/observability — no UI changes
|
||||
|
||||
## What This Does NOT Include
|
||||
|
||||
- **Dedicated admin UI for viewing job logs** — existing admin page already shows timeline
|
||||
- **Log aggregation/search UI** — Cloud Logging and Firestore viewer already serve this purpose
|
||||
- **Changing state_data overwrite behavior** — the `{**job.state_data, ...}` pattern in video_worker.py is intentional; instead we preserve history in timeline events
|
||||
- **Changing outputs_deleted_at clearing** — video worker clearing it after re-processing is correct behavior; the edit history lives in timeline metadata instead
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
All changes are additive (new optional field on TimelineEvent, additional log writes). Rolling back just means deploying the previous version — no data migration needed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None — the approach is straightforward and low-risk.
|
||||
@@ -1,218 +0,0 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -1,106 +0,0 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
@@ -1,279 +0,0 @@
|
||||
# Plan: Private (Non-Published) Tracks
|
||||
|
||||
**Created:** 2026-02-24
|
||||
**Branch:** feat/sess-20260224-1149-private-tracks
|
||||
**Status:** Implemented
|
||||
|
||||
## Overview
|
||||
|
||||
Add a `is_private` flag to jobs that changes distribution behavior:
|
||||
- **Dropbox**: Uses `Tracks-NonPublished` folder (not `Tracks-Organized`) with brand code `NOMADNP` (not `NOMAD`)
|
||||
- **YouTube**: Skipped entirely
|
||||
- **Google Drive**: Skipped entirely
|
||||
- **Email**: Completion email includes Dropbox link but no YouTube section
|
||||
- **Discord**: Still notified (for internal tracking)
|
||||
|
||||
Available to all users at job creation and toggleable by admin on existing jobs.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] All users can mark a job as "Private (Non-Published)" when creating it
|
||||
- [ ] Admin can toggle `is_private` on any existing job via admin dashboard
|
||||
- [ ] Private jobs upload to Dropbox at `Tracks-NonPublished` with `NOMADNP-XXXX` brand codes
|
||||
- [ ] Private jobs skip YouTube and Google Drive uploads entirely
|
||||
- [ ] Completion emails omit YouTube section when no YouTube URL (private jobs + any other edge case)
|
||||
- [ ] When admin toggles a completed job to private, existing outputs are auto-deleted
|
||||
- [ ] Discord notifications still fire for private jobs
|
||||
- [ ] Existing (non-private) jobs are completely unaffected (backward compatible)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Core Design: Single Boolean Flag with Runtime Overrides
|
||||
|
||||
Store `is_private: bool = False` on the job model. At distribution time, a helper function returns effective distribution parameters based on this flag. This means:
|
||||
|
||||
- **Toggling is_private automatically changes all distribution behavior** - no need to manually update dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id
|
||||
- **Job fields still store the "normal" defaults** - the override happens at the config-to-orchestrator boundary
|
||||
- **Admin can flip one field** and everything adapts
|
||||
|
||||
### Key Helper Function
|
||||
|
||||
```python
|
||||
# In job_defaults_service.py
|
||||
def get_effective_distribution_for_job(job) -> EffectiveDistributionSettings:
|
||||
"""Return distribution settings, applying private overrides if is_private=True."""
|
||||
if getattr(job, 'is_private', False):
|
||||
settings = get_settings()
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=settings.default_private_dropbox_path,
|
||||
brand_prefix=settings.default_private_brand_prefix,
|
||||
enable_youtube_upload=False,
|
||||
gdrive_folder_id=None,
|
||||
# Keep other settings from job (discord, youtube_description, etc.)
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=None,
|
||||
)
|
||||
# Non-private: return job's own settings
|
||||
return EffectiveDistributionSettings(
|
||||
dropbox_path=getattr(job, 'dropbox_path', None),
|
||||
brand_prefix=getattr(job, 'brand_prefix', None),
|
||||
enable_youtube_upload=getattr(job, 'enable_youtube_upload', False),
|
||||
gdrive_folder_id=getattr(job, 'gdrive_folder_id', None),
|
||||
discord_webhook_url=getattr(job, 'discord_webhook_url', None),
|
||||
youtube_description=getattr(job, 'youtube_description_template', None),
|
||||
)
|
||||
```
|
||||
|
||||
This helper is used in both distribution paths (orchestrator and native fallback).
|
||||
|
||||
### Distribution Paths That Need Changes
|
||||
|
||||
Two code paths handle distribution (see LESSONS-LEARNED: "Fix Both Sides of Dual Code Paths"):
|
||||
|
||||
1. **Orchestrator path** (`video_worker_orchestrator.py`): Primary path. `create_orchestrator_config_from_job()` uses the helper to build config.
|
||||
2. **Native distribution path** (`video_worker.py` → `_handle_native_distribution()`): Fallback path. Also uses the helper instead of reading job fields directly.
|
||||
|
||||
### Email Template Change
|
||||
|
||||
Add conditional YouTube section removal in `render_template()`, matching the existing pattern for `feedback_url`:
|
||||
|
||||
```python
|
||||
# Remove YouTube section if no URL provided
|
||||
if not variables.get("youtube_url"):
|
||||
result = re.sub(
|
||||
r"\n*Here's the link for the karaoke video published to YouTube:\n\{youtube_url\}\n*",
|
||||
'\n',
|
||||
result,
|
||||
flags=re.DOTALL
|
||||
)
|
||||
variables["youtube_url"] = ""
|
||||
```
|
||||
|
||||
And change `render_job_completion()` to pass `None` instead of `"[YouTube URL not available]"` when youtube_url is empty.
|
||||
|
||||
### Admin Toggle Auto-Delete
|
||||
|
||||
When admin PATCH updates `is_private` from False to True on a job that has existing outputs, reuse the existing delete-outputs logic to remove YouTube + GDrive + Dropbox outputs. Admin can then "Reprocess" to re-distribute to the private Dropbox path.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Model & Config (foundation)
|
||||
|
||||
1. [ ] **Add `is_private` to Job model** (`backend/models/job.py`)
|
||||
- Add `is_private: bool = False` field after `made_for_you` field
|
||||
- Add to `JobCreate` model as well
|
||||
|
||||
2. [ ] **Add private distribution config** (`backend/config.py`)
|
||||
- Add `default_private_dropbox_path: Optional[str] = os.getenv("DEFAULT_PRIVATE_DROPBOX_PATH")`
|
||||
- Add `default_private_brand_prefix: Optional[str] = os.getenv("DEFAULT_PRIVATE_BRAND_PREFIX", "NOMADNP")`
|
||||
|
||||
3. [ ] **Add helper function** (`backend/services/job_defaults_service.py`)
|
||||
- Add `get_effective_distribution_for_job(job)` function that returns distribution settings with private overrides applied
|
||||
|
||||
### Phase 2: Backend API (job creation + admin)
|
||||
|
||||
4. [ ] **Add `is_private` to request models** (`backend/models/requests.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `URLSubmissionRequest`
|
||||
- Add `is_private: Optional[bool] = None` to `AudioSearchRequest`
|
||||
|
||||
5. [ ] **Update URL creation endpoint** (`backend/api/routes/jobs.py`)
|
||||
- Accept and pass through `is_private` field to `JobCreate`
|
||||
|
||||
6. [ ] **Update file upload endpoint** (`backend/api/routes/file_upload.py`)
|
||||
- Accept `is_private` Form parameter
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
7. [ ] **Update audio search endpoint** (`backend/api/routes/audio_search.py`)
|
||||
- Accept `is_private` in `AudioSearchRequest`
|
||||
- Pass through to `JobCreate`
|
||||
|
||||
8. [ ] **Update admin endpoints** (`backend/api/routes/admin.py`)
|
||||
- Add `is_private: Optional[bool] = None` to `JobUpdateRequest`
|
||||
- Add `"is_private"` to `EDITABLE_JOB_FIELDS`
|
||||
- In the PATCH handler: detect when `is_private` changes from False→True on a completed job, auto-delete existing outputs (call existing delete-outputs service logic)
|
||||
|
||||
### Phase 3: Distribution Pipeline
|
||||
|
||||
9. [ ] **Update orchestrator config creation** (`backend/workers/video_worker_orchestrator.py`)
|
||||
- In `create_orchestrator_config_from_job()`: Use `get_effective_distribution_for_job()` helper to determine dropbox_path, brand_prefix, enable_youtube_upload, gdrive_folder_id instead of reading directly from job
|
||||
|
||||
10. [ ] **Update native distribution fallback** (`backend/workers/video_worker.py`)
|
||||
- In `_handle_native_distribution()`: Use `get_effective_distribution_for_job()` helper instead of reading directly from job fields
|
||||
|
||||
### Phase 4: Email Template
|
||||
|
||||
11. [ ] **Add YouTube section conditional removal** (`backend/services/template_service.py`)
|
||||
- In `render_template()`: Add regex to remove YouTube paragraph when `youtube_url` is empty (matching existing `feedback_url` pattern)
|
||||
- In `render_job_completion()`: Pass `None` for youtube_url when empty (instead of `"[YouTube URL not available]"`)
|
||||
|
||||
### Phase 5: Frontend
|
||||
|
||||
12. [ ] **Add private checkbox to job creation form** (`frontend/components/job/JobSubmission.tsx`)
|
||||
- Add "Private (Non-Published)" checkbox below the existing "Skip lyrics review" checkbox
|
||||
- Show for all users (not just admin)
|
||||
- Include brief description text: "Private tracks go to Dropbox only, not YouTube or Google Drive"
|
||||
|
||||
13. [ ] **Update API interfaces** (`frontend/lib/api.ts`)
|
||||
- Add `is_private?: boolean` to all job creation option interfaces
|
||||
- Add `is_private` to `JobUpdateRequest` interface
|
||||
- Pass through in `uploadJob()`, `createJobFromUrl()`, `searchAudio()` methods
|
||||
|
||||
14. [ ] **Add private toggle to admin job detail** (`frontend/app/admin/jobs/page.tsx`)
|
||||
- Add `is_private` as an inline-editable field in the job overview section (use Switch component)
|
||||
- Show current state with visual indicator (e.g., lock icon for private jobs)
|
||||
- When toggling to private on a completed job, show confirmation dialog about auto-deleting outputs
|
||||
|
||||
### Phase 6: Environment Configuration
|
||||
|
||||
15. [ ] **Update CI/deployment config** (`.github/workflows/ci.yml` or Cloud Run env)
|
||||
- Add `DEFAULT_PRIVATE_DROPBOX_PATH` environment variable (e.g., `/Karaoke/Tracks-NonPublished`)
|
||||
- Add `DEFAULT_PRIVATE_BRAND_PREFIX` environment variable (e.g., `NOMADNP`)
|
||||
- These should be configured in Cloud Run service and CI test environment
|
||||
|
||||
### Phase 7: Testing
|
||||
|
||||
16. [ ] **Unit tests for private distribution logic** (`backend/tests/test_private_tracks.py`)
|
||||
- Test `get_effective_distribution_for_job()` with private and non-private jobs
|
||||
- Test `create_orchestrator_config_from_job()` applies private overrides correctly
|
||||
- Test `_run_distribution()` skips YouTube and GDrive when private config applied
|
||||
- Test `_upload_to_dropbox()` uses private path and brand prefix
|
||||
- Test brand code generation with NOMADNP prefix in Tracks-NonPublished
|
||||
- Test `_handle_native_distribution()` respects private overrides
|
||||
- Test non-private jobs are completely unaffected (regression tests)
|
||||
|
||||
17. [ ] **Unit tests for email changes** (add to `backend/tests/test_email_service.py` or `test_template_service.py`)
|
||||
- Test YouTube section removed from email when youtube_url is None
|
||||
- Test YouTube section present when youtube_url is provided
|
||||
- Test Dropbox section still present for private jobs
|
||||
- Test full email rendering for private job scenario
|
||||
|
||||
18. [ ] **Unit tests for admin toggle** (add to `backend/tests/test_admin.py` or new file)
|
||||
- Test admin can update is_private field
|
||||
- Test toggling is_private to True on completed job triggers output deletion
|
||||
- Test toggling is_private to True on non-completed job does NOT delete outputs
|
||||
- Test toggling is_private to False does nothing special
|
||||
|
||||
19. [ ] **Integration test** (`backend/tests/emulator/` or `test_distribution_services.py`)
|
||||
- Test full distribution pipeline with is_private=True: verify Dropbox path, brand prefix, YouTube skip, GDrive skip
|
||||
|
||||
20. [ ] **Frontend E2E regression test** (`frontend/e2e/regression/`)
|
||||
- Test private checkbox appears in job creation form
|
||||
- Test private toggle in admin dashboard
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Add `is_private: bool = False` to Job and JobCreate |
|
||||
| `backend/config.py` | Modify | Add `default_private_dropbox_path` and `default_private_brand_prefix` |
|
||||
| `backend/services/job_defaults_service.py` | Modify | Add `get_effective_distribution_for_job()` helper |
|
||||
| `backend/models/requests.py` | Modify | Add `is_private` to URL and AudioSearch request models |
|
||||
| `backend/api/routes/jobs.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Accept and pass `is_private` |
|
||||
| `backend/api/routes/admin.py` | Modify | Add `is_private` to editable fields + auto-delete logic |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Modify | Use helper in `create_orchestrator_config_from_job()` |
|
||||
| `backend/workers/video_worker.py` | Modify | Use helper in `_handle_native_distribution()` |
|
||||
| `backend/services/template_service.py` | Modify | Conditional YouTube section removal |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Add private checkbox |
|
||||
| `frontend/lib/api.ts` | Modify | Add `is_private` to interfaces and API calls |
|
||||
| `frontend/app/admin/jobs/page.tsx` | Modify | Add private toggle to admin detail |
|
||||
| `backend/tests/test_private_tracks.py` | Create | Comprehensive unit tests for private track logic |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests (~25-30 tests in `test_private_tracks.py`)
|
||||
- `get_effective_distribution_for_job()` with private=True and False
|
||||
- `create_orchestrator_config_from_job()` private overrides (YouTube=False, GDrive=None, private Dropbox path, NOMADNP brand)
|
||||
- Distribution stage skips: YouTube not called, GDrive not called, Dropbox IS called with correct path
|
||||
- Brand code generation uses NOMADNP prefix and Tracks-NonPublished path
|
||||
- Native distribution fallback applies same overrides
|
||||
- **Regression tests**: Non-private jobs have unchanged behavior (same paths, same brand code, YouTube+GDrive+Dropbox all fire)
|
||||
|
||||
### Email Template Tests (~5 tests)
|
||||
- YouTube section stripped when youtube_url is None
|
||||
- YouTube section preserved when youtube_url has value
|
||||
- Dropbox section always present
|
||||
- Full private job email rendering (Dropbox + no YouTube)
|
||||
- Full non-private job email rendering (both present) - regression
|
||||
|
||||
### Admin Toggle Tests (~5 tests)
|
||||
- is_private editable via PATCH
|
||||
- Toggle to private on completed job deletes outputs
|
||||
- Toggle to private on pending/in-progress job does NOT delete
|
||||
- Toggle to non-private does nothing special
|
||||
|
||||
### Frontend Tests
|
||||
- E2E: Private checkbox visible and functional in job creation
|
||||
- E2E: Admin toggle visible and includes confirmation dialog
|
||||
|
||||
### Manual Verification
|
||||
- Create a private job end-to-end, verify Dropbox folder in Tracks-NonPublished with NOMADNP code
|
||||
- Verify completion email has Dropbox link but no YouTube section
|
||||
- Toggle existing completed job to private, verify outputs deleted
|
||||
|
||||
## Regression Risk Analysis
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| Non-private jobs affected | `is_private` defaults to `False`; helper returns unchanged values for non-private. Explicit regression tests. |
|
||||
| Dual code paths (orchestrator vs native) | Both paths use same helper function. Tests cover both paths. (Lesson: "Fix Both Sides of Dual Code Paths") |
|
||||
| Email template regex breaks other content | Regex targets specific YouTube paragraph text. Tests verify non-YouTube content untouched. |
|
||||
| Brand code namespace collision | NOMADNP codes are in a completely separate Dropbox folder (Tracks-NonPublished), no overlap with NOMAD codes in Tracks-Organized |
|
||||
| Admin auto-delete on wrong job | Only triggers when `is_private` changes from False→True AND job has terminal status (complete). Confirmation dialog in frontend. |
|
||||
| Made-for-you orders | MFY webhook doesn't set is_private (defaults False). Admin can toggle after. No change to MFY flow. |
|
||||
| Existing Firestore documents | Missing `is_private` field defaults to `False` via model default. No migration needed. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Who sees private toggle? → **Everyone** (all users at creation, admin on existing)
|
||||
- [x] Auto-delete on toggle? → **Yes**, auto-delete all outputs when toggling to private on completed job
|
||||
- [x] Email format? → **Remove YouTube section** when no YouTube URL (improves all edge cases)
|
||||
- [x] Discord? → **Still notify** (useful for internal tracking)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Feature flag**: The `is_private` field defaults to `False`, so setting `DEFAULT_PRIVATE_DROPBOX_PATH` to empty/None effectively disables private distribution
|
||||
2. **No schema migration needed**: Missing `is_private` field on existing Firestore documents defaults to `False`
|
||||
3. **Email change is backward-compatible**: The YouTube section removal only happens when youtube_url is None, which is already an edge case showing "[YouTube URL not available]"
|
||||
4. **Frontend**: Private checkbox is additive UI, removing it just means users can't create new private jobs
|
||||
@@ -1,239 +0,0 @@
|
||||
# Plan: Song Lookup & Autocomplete + Community Version Detection
|
||||
|
||||
**Created:** 2026-03-02
|
||||
**Branch:** feat/sess-20260302-2324-song-lookup-autocomplete
|
||||
**Status:** Implementation Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Add two features to the karaoke-gen job creation flow:
|
||||
|
||||
1. **Artist/title autocomplete** using karaoke-decide's MusicBrainz + Spotify catalog data, so users get properly-styled artist/title text (e.g. "Fleetwood Mac" instead of "fleetwood mac")
|
||||
2. **Community karaoke version detection** that warns users when a high-quality community karaoke version already exists on YouTube (via karaokenerds data), saving them credits and time
|
||||
|
||||
Both features are **optional and non-blocking** — users can still type whatever they want and proceed even if a community version exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] Artist field autocomplete across all 3 job creation tabs (Search, Upload, URL)
|
||||
- [ ] Title field autocomplete (optionally filtered by selected artist) across all 3 tabs
|
||||
- [ ] Autocomplete uses canonical MusicBrainz/Spotify text styling
|
||||
- [ ] Autocomplete is optional — users can ignore suggestions and type freely
|
||||
- [ ] Community version check triggers when both artist and title are filled
|
||||
- [ ] If community karaoke version exists, show a dismissible banner with YouTube link(s)
|
||||
- [ ] Community check does not block job submission — just informs the user
|
||||
- [ ] Graceful degradation if karaoke-decide API or karaokenerds.com is down
|
||||
|
||||
## Architecture Decisions
|
||||
|
||||
### How to access karaoke-decide catalog data
|
||||
|
||||
**Decision: Proxy through karaoke-gen backend**
|
||||
|
||||
Both services share the same GCP project (`nomadkaraoke`), but the frontend at `gen.nomadkaraoke.com` can't directly call `decide.nomadkaraoke.com/api/` due to CORS (different origins). Options considered:
|
||||
|
||||
| Option | Pros | Cons |
|
||||
|--------|------|------|
|
||||
| **Proxy via karaoke-gen backend** | No CORS issues, no changes to karaoke-decide, frontend already calls api.nomadkaraoke.com | Added latency (extra hop), karaoke-decide dependency |
|
||||
| Direct frontend calls + CORS | Simpler, no proxy code | Requires karaoke-decide CORS changes, cross-project coupling |
|
||||
| Query BigQuery directly from karaoke-gen | No dependency on karaoke-decide service | Duplicates query logic, BigQuery setup needed, higher latency |
|
||||
|
||||
The proxy approach is cleanest: karaoke-gen frontend → karaoke-gen backend → karaoke-decide API. Response caching in the proxy minimizes latency impact.
|
||||
|
||||
### How to detect community karaoke versions
|
||||
|
||||
**Decision: Scrape karaokenerds.com from karaoke-gen backend** (same approach as kjbox)
|
||||
|
||||
The `karaokenerds_raw` BigQuery table has the catalog but NOT community status. Community status is determined by an `<img class="check">` badge in karaokenerds.com's HTML, only available via web scraping. The kjbox codebase has proven, well-tested scraping logic we can port directly.
|
||||
|
||||
Search URL: `https://karaokenerds.com/Search?query={artist}+{title}&webFilter=OnlyWeb`
|
||||
|
||||
Returns: songs with tracks, each track having `brand_name`, `brand_code`, `youtube_url`, `is_community`.
|
||||
|
||||
### Autocomplete UI component
|
||||
|
||||
**Decision: Use existing shadcn Command (cmdk) + Popover components**
|
||||
|
||||
The frontend already has `command.tsx` and `popover.tsx` from shadcn/ui. The standard combobox pattern (Popover + Command) provides:
|
||||
- Keyboard navigation (arrow keys, enter to select)
|
||||
- Debounced search-as-you-type
|
||||
- Clean dropdown with loading states
|
||||
- Works inline within forms (not modal)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Data flow
|
||||
|
||||
```
|
||||
User types in artist field
|
||||
→ debounced (300ms) API call to GET /api/catalog/artists?q=fle
|
||||
→ karaoke-gen backend proxies to decide.nomadkaraoke.com/api/catalog/artists?q=fle
|
||||
→ returns [{ name: "Fleetwood Mac", mbid: "...", popularity: 85 }, ...]
|
||||
→ user selects "Fleetwood Mac" → field populated with canonical name
|
||||
|
||||
User types in title field
|
||||
→ debounced (300ms) API call to GET /api/catalog/tracks?q=dreams&artist=Fleetwood+Mac
|
||||
→ returns [{ track_name: "Dreams", artist_name: "Fleetwood Mac", popularity: 82 }, ...]
|
||||
→ user selects "Dreams" → field populated with canonical title
|
||||
|
||||
Both fields filled → community check fires
|
||||
→ POST /api/catalog/community-check { artist: "Fleetwood Mac", title: "Dreams" }
|
||||
→ backend scrapes karaokenerds.com for "Fleetwood Mac Dreams" with webFilter=OnlyWeb
|
||||
→ if community tracks found with YouTube URLs, return them
|
||||
→ frontend shows banner: "A community karaoke version already exists! [Watch on YouTube →]"
|
||||
```
|
||||
|
||||
### Caching strategy
|
||||
|
||||
| Endpoint | Cache TTL | Scope |
|
||||
|----------|-----------|-------|
|
||||
| Artist search proxy | 5 min | In-memory (backend) |
|
||||
| Track search proxy | 5 min | In-memory (backend) |
|
||||
| Community check | 1 hour | In-memory (backend) |
|
||||
|
||||
### karaoke-decide API endpoints used
|
||||
|
||||
These are all **public, no auth required**:
|
||||
|
||||
- `GET /api/catalog/artists?q={query}&limit=10` — MBID-first artist search
|
||||
- Returns: `[{ mbid, name, disambiguation, artist_type, tags, spotify_id, popularity, genres }]`
|
||||
- `GET /api/catalog/tracks?q={query}&limit=10` — Track search (Spotify data)
|
||||
- Returns: `[{ track_id, track_name, artist_name, artist_id, popularity, duration_ms, explicit }]`
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend — Catalog proxy + community check endpoints
|
||||
|
||||
1. [ ] **Add catalog proxy service** — `backend/services/catalog_proxy_service.py`
|
||||
- HTTP client (httpx async) calling karaoke-decide API
|
||||
- In-memory TTL cache for responses (5 min)
|
||||
- Configurable karaoke-decide base URL
|
||||
- Graceful error handling (return empty results on failure)
|
||||
|
||||
2. [ ] **Port karaokenerds scraping logic** — `backend/services/karaokenerds_service.py`
|
||||
- Port `search()`, `parse_results()`, `_parse_tracks()`, `_parse_single_track()`, `_clean_youtube_url()` from `kjbox/kj-controller/karaoke_nerds.py`
|
||||
- Add async support (httpx instead of requests)
|
||||
- In-memory TTL cache (1 hour)
|
||||
- Return only community tracks with YouTube URLs
|
||||
|
||||
3. [ ] **Add catalog routes** — `backend/api/routes/catalog.py`
|
||||
- `GET /api/catalog/artists?q=&limit=10` — proxy to karaoke-decide
|
||||
- `GET /api/catalog/tracks?q=&artist=&limit=10` — proxy to karaoke-decide
|
||||
- `POST /api/catalog/community-check` — scrape karaokenerds.com
|
||||
- All endpoints require auth (same as job creation)
|
||||
- Pydantic request/response models
|
||||
|
||||
4. [ ] **Register routes in main.py** — add `catalog.router` to app
|
||||
|
||||
5. [ ] **Add config** — `backend/config.py`
|
||||
- `karaoke_decide_api_url` setting (default: `https://decide.nomadkaraoke.com`)
|
||||
|
||||
6. [ ] **Add `beautifulsoup4` + `httpx` dependencies** to `pyproject.toml` (if not already present)
|
||||
|
||||
### Phase 2: Frontend — Autocomplete component
|
||||
|
||||
7. [ ] **Create `AutocompleteInput` component** — `frontend/components/ui/autocomplete-input.tsx`
|
||||
- Wraps an Input field with a Popover + Command dropdown
|
||||
- Props: `value`, `onChange`, `onSelect`, `fetchSuggestions(query) => Promise<Suggestion[]>`, `placeholder`, `disabled`, `renderItem`
|
||||
- Debounced search (300ms) triggered on input change
|
||||
- Shows loading spinner while fetching
|
||||
- Shows "No results" when search returns empty
|
||||
- Keyboard navigation (up/down arrows, enter to select, escape to close)
|
||||
- Clicking outside closes dropdown
|
||||
- When user selects a suggestion, calls `onSelect(suggestion)` and populates input
|
||||
- When user types without selecting, uses their raw text (autocomplete is optional)
|
||||
|
||||
8. [ ] **Add catalog API functions** — `frontend/lib/api.ts`
|
||||
- `searchArtists(query: string, limit?: number)` → `GET /api/catalog/artists`
|
||||
- `searchTracks(query: string, artist?: string, limit?: number)` → `GET /api/catalog/tracks`
|
||||
- `checkCommunityVersions(artist: string, title: string)` → `POST /api/catalog/community-check`
|
||||
|
||||
### Phase 3: Frontend — Integrate autocomplete into job creation
|
||||
|
||||
9. [ ] **Replace artist Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchArtist`, `uploadArtist`, `youtubeArtist`
|
||||
- Fetch suggestions from `api.searchArtists(query)`
|
||||
- On select: populate artist field with canonical name
|
||||
- Show artist name + optional disambiguation in dropdown items
|
||||
|
||||
10. [ ] **Replace title Input fields with AutocompleteInput** in `JobSubmission.tsx`
|
||||
- All 3 tabs: `searchTitle`, `uploadTitle`, `youtubeTitle`
|
||||
- Fetch suggestions from `api.searchTracks(query, selectedArtist)`
|
||||
- On select: populate title field with canonical name
|
||||
- Show track name + artist name + duration in dropdown items
|
||||
|
||||
### Phase 4: Frontend — Community version detection
|
||||
|
||||
11. [ ] **Create `CommunityVersionBanner` component** — `frontend/components/job/CommunityVersionBanner.tsx`
|
||||
- Shows when community karaoke version(s) found
|
||||
- Displays: "A karaoke version of this song already exists!"
|
||||
- Lists community tracks with brand name and YouTube link
|
||||
- "Watch on YouTube" button for top community version
|
||||
- Dismissible (X button or "Continue anyway" link)
|
||||
- Styled with green border (matching kjbox community track styling)
|
||||
|
||||
12. [ ] **Add community check logic to `JobSubmission.tsx`** (Search tab only)
|
||||
- Debounced check (500ms) when both artist AND title fields have values
|
||||
- Only triggers when values change (not on every keystroke)
|
||||
- Shows `CommunityVersionBanner` between the form fields and the submit button
|
||||
- Does NOT block form submission
|
||||
- Clears when artist or title changes
|
||||
|
||||
### Phase 5: Testing
|
||||
|
||||
13. [ ] **Backend unit tests** — `backend/tests/unit/test_catalog_routes.py`
|
||||
- Test proxy endpoints with mocked karaoke-decide responses
|
||||
- Test community check with mocked karaokenerds HTML
|
||||
- Test error handling (karaoke-decide down, karaokenerds down)
|
||||
- Test caching behavior
|
||||
|
||||
14. [ ] **Backend unit tests** — `backend/tests/unit/test_karaokenerds_service.py`
|
||||
- Port relevant tests from `kjbox/kj-controller/tests/unit/test_karaoke_nerds.py`
|
||||
- Test HTML parsing, community detection, YouTube URL cleanup
|
||||
|
||||
15. [ ] **Frontend component tests** — `frontend/components/ui/__tests__/autocomplete-input.test.tsx`
|
||||
- Test suggestion fetching and display
|
||||
- Test keyboard navigation
|
||||
- Test selection behavior
|
||||
- Test free-text input (no selection)
|
||||
|
||||
16. [ ] **E2E tests** — `frontend/e2e/job-creation-autocomplete.spec.ts`
|
||||
- Test autocomplete appears on typing
|
||||
- Test selecting from autocomplete populates field
|
||||
- Test community banner appears for known songs
|
||||
- Test form submission works with and without autocomplete selection
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/catalog_proxy_service.py` | Create | HTTP client for karaoke-decide API with caching |
|
||||
| `backend/services/karaokenerds_service.py` | Create | Karaokenerds.com scraper (ported from kjbox) |
|
||||
| `backend/api/routes/catalog.py` | Create | Proxy endpoints for artist/track search + community check |
|
||||
| `backend/main.py` | Modify | Register catalog router |
|
||||
| `backend/config.py` | Modify | Add karaoke_decide_api_url setting |
|
||||
| `pyproject.toml` | Modify | Add beautifulsoup4 dependency (if missing) |
|
||||
| `frontend/components/ui/autocomplete-input.tsx` | Create | Reusable autocomplete input component |
|
||||
| `frontend/components/job/CommunityVersionBanner.tsx` | Create | Community version warning banner |
|
||||
| `frontend/components/job/JobSubmission.tsx` | Modify | Replace Input with AutocompleteInput, add community check |
|
||||
| `frontend/lib/api.ts` | Modify | Add catalog API functions |
|
||||
| `backend/tests/unit/test_catalog_routes.py` | Create | Backend route tests |
|
||||
| `backend/tests/unit/test_karaokenerds_service.py` | Create | Scraper unit tests |
|
||||
| `frontend/components/ui/__tests__/autocomplete-input.test.tsx` | Create | Component tests |
|
||||
| `frontend/e2e/job-creation-autocomplete.spec.ts` | Create | E2E tests |
|
||||
|
||||
## Decisions (Resolved)
|
||||
|
||||
- **Track search scope**: All Spotify tracks (256M) — better coverage for new/obscure songs
|
||||
- **Community check scope**: Search tab only (Upload/URL users already have their audio)
|
||||
- **Rate limiting**: 20 requests/min per user on catalog proxy endpoints (prevents abuse of karaokenerds scraping). All endpoints require auth.
|
||||
- **"Display As" fields**: No autocomplete — they're for deliberate overrides like soundtrack names
|
||||
- **CORS**: Proxy approach (no changes needed to karaoke-decide)
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
- Feature is entirely additive — no existing functionality is modified
|
||||
- Autocomplete fields fall back to plain text Input if API calls fail
|
||||
- Community check is non-blocking — if it fails, form works normally
|
||||
- All new endpoints are behind the existing auth middleware
|
||||
- Can disable by reverting the frontend changes (backend proxy is harmless if unused)
|
||||
@@ -1,329 +0,0 @@
|
||||
# Plan: Unified Combined Review for Local CLI
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
|
||||
|
||||
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
|
||||
- Review code lives inside `correct_lyrics()` method in controller.py
|
||||
- When auto-correction is disabled, `correct_lyrics()` is never called
|
||||
- Users never get a chance to review lyrics
|
||||
|
||||
2. **Standalone instrumental UI still exists for CLI**
|
||||
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
|
||||
- CLI launches this standalone server after lyrics review
|
||||
- The combined review flow we built only works for the web app
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
|
||||
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
|
||||
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
|
||||
4. **Keep it simple** - Minimal changes to achieve the goal
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Current State (Broken)
|
||||
|
||||
```
|
||||
CLI Flow:
|
||||
controller.py::process()
|
||||
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
|
||||
→ ReviewServer starts (lyrics only)
|
||||
→ User reviews
|
||||
→ ReviewServer stops
|
||||
→ gen_cli.py::run_instrumental_review()
|
||||
→ InstrumentalReviewServer starts (instrumental only)
|
||||
→ User selects instrumental
|
||||
→ InstrumentalReviewServer stops
|
||||
|
||||
Cloud Web App Flow:
|
||||
Job enters AWAITING_REVIEW
|
||||
→ User visits /app/jobs/{id}/review
|
||||
→ Combined UI: lyrics + instrumental
|
||||
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
|
||||
```
|
||||
|
||||
### Target State
|
||||
|
||||
```
|
||||
CLI Flow (and Cloud CLI):
|
||||
controller.py::process()
|
||||
→ _run_human_review() [NEW - always called when review enabled]
|
||||
→ ReviewServer starts (combined: lyrics + instrumental)
|
||||
→ User reviews lyrics AND selects instrumental
|
||||
→ ReviewServer stops, returns both results
|
||||
→ Continue with selected instrumental
|
||||
|
||||
Cloud Web App Flow: (unchanged)
|
||||
Same as before
|
||||
```
|
||||
|
||||
## Key Insight
|
||||
|
||||
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
|
||||
|
||||
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
|
||||
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
|
||||
3. Call the review flow from the right place (not inside `correct_lyrics()`)
|
||||
4. Delete `InstrumentalReviewServer` entirely
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Fix ReviewServer to Support Instrumental Selection
|
||||
|
||||
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None, # NEW
|
||||
backing_vocals_analysis: dict | None = None, # NEW
|
||||
...
|
||||
):
|
||||
```
|
||||
|
||||
Update `_correction_data()` endpoint to include these in response:
|
||||
|
||||
```python
|
||||
@app.get("/api/review/{job_id}/correction-data")
|
||||
def _correction_data(job_id: str):
|
||||
return {
|
||||
"lyrics_data": self.correction_result.to_dict(),
|
||||
"instrumental_options": self.instrumental_options, # NEW
|
||||
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
|
||||
}
|
||||
```
|
||||
|
||||
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
|
||||
|
||||
```python
|
||||
@app.post("/api/review/{job_id}/complete-review")
|
||||
def _complete_review(job_id: str, request: ReviewCompleteRequest):
|
||||
# Existing: store corrected lyrics
|
||||
self.corrected_lyrics = request.corrected_lyrics
|
||||
|
||||
# NEW: store instrumental selection
|
||||
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
|
||||
|
||||
self.review_completed = True
|
||||
```
|
||||
|
||||
**1.3. Expose instrumental selection after review completes**
|
||||
|
||||
The calling code needs to get the selection. Update the return value or add an attribute:
|
||||
|
||||
```python
|
||||
# After review loop completes:
|
||||
return {
|
||||
"corrected_lyrics": self.corrected_lyrics,
|
||||
"instrumental_selection": self.instrumental_selection
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Fix controller.py to Call Review at the Right Time
|
||||
|
||||
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Create new method `_run_human_review()` that can be called independently:
|
||||
|
||||
```python
|
||||
def _run_human_review(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None,
|
||||
backing_vocals_analysis: dict | None = None
|
||||
) -> tuple[CorrectionResult, str | None]:
|
||||
"""Run human review server and return corrected result + instrumental selection."""
|
||||
|
||||
if not self.output_config.review_enabled:
|
||||
return correction_result, None
|
||||
|
||||
server = ReviewServer(
|
||||
correction_result=correction_result,
|
||||
instrumental_options=instrumental_options,
|
||||
backing_vocals_analysis=backing_vocals_analysis,
|
||||
...
|
||||
)
|
||||
|
||||
result = server.start() # Blocks until review complete
|
||||
|
||||
return result["corrected_lyrics"], result.get("instrumental_selection")
|
||||
```
|
||||
|
||||
**2.2. Update `process()` to always call review when appropriate**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Current flow (simplified):
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
if self.output_config.run_correction:
|
||||
self.correct_lyrics() # Review is inside here!
|
||||
# ...
|
||||
```
|
||||
|
||||
New flow:
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
|
||||
# Build correction result (with or without auto-correction)
|
||||
if self.output_config.run_correction:
|
||||
correction_result = self._run_auto_correction()
|
||||
else:
|
||||
correction_result = self._create_uncorrected_result()
|
||||
|
||||
# Human review is separate from auto-correction
|
||||
if self.output_config.review_enabled:
|
||||
instrumental_options = self._get_instrumental_options()
|
||||
backing_vocals_analysis = self._analyze_backing_vocals()
|
||||
|
||||
corrected, instrumental_selection = self._run_human_review(
|
||||
correction_result,
|
||||
instrumental_options,
|
||||
backing_vocals_analysis
|
||||
)
|
||||
|
||||
self.results.correction_result = corrected
|
||||
self.results.instrumental_selection = instrumental_selection
|
||||
```
|
||||
|
||||
### Phase 3: Update gen_cli.py to Use Combined Review
|
||||
|
||||
**3.1. Remove standalone instrumental review call**
|
||||
|
||||
File: `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
|
||||
|
||||
Before:
|
||||
```python
|
||||
# After lyrics processing
|
||||
if not args.non_interactive:
|
||||
instrumental_selection = run_instrumental_review(...)
|
||||
```
|
||||
|
||||
After:
|
||||
```python
|
||||
# Instrumental selection comes from controller.results.instrumental_selection
|
||||
instrumental_selection = controller.results.instrumental_selection
|
||||
```
|
||||
|
||||
**3.2. Pass instrumental options to controller**
|
||||
|
||||
The controller needs instrumental options to display in review. These come from the separated audio stems:
|
||||
|
||||
```python
|
||||
controller = LyricsTranscriber(
|
||||
# ... existing params ...
|
||||
)
|
||||
|
||||
# Before calling process(), set up instrumental options
|
||||
# These are typically discovered during audio separation
|
||||
```
|
||||
|
||||
### Phase 4: Delete Standalone InstrumentalReviewServer
|
||||
|
||||
**4.1. Delete files**
|
||||
|
||||
- Delete `karaoke_gen/instrumental_review/server.py`
|
||||
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
|
||||
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
**4.2. Clean up imports**
|
||||
|
||||
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
|
||||
|
||||
### Phase 5: Update Frontend for Local Mode
|
||||
|
||||
**5.1. Verify local mode detection works**
|
||||
|
||||
File: `frontend/lib/api.ts`
|
||||
|
||||
The frontend already has local mode detection based on hostname and port. Verify it works:
|
||||
|
||||
```typescript
|
||||
const isLocalMode = () => {
|
||||
if (typeof window === 'undefined') return false;
|
||||
const hostname = window.location.hostname;
|
||||
const port = parseInt(window.location.port, 10);
|
||||
return hostname === 'localhost' && port >= 8000 && port <= 8770;
|
||||
};
|
||||
```
|
||||
|
||||
**5.2. Verify correction-data endpoint parsing**
|
||||
|
||||
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
|
||||
|
||||
### Phase 6: Testing
|
||||
|
||||
**6.1. Manual testing checklist**
|
||||
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
|
||||
- [ ] Combined UI shows both lyrics editor and instrumental selector
|
||||
- [ ] Selecting instrumental and completing review works
|
||||
- [ ] No standalone instrumental review page appears
|
||||
- [ ] Cloud web app still works (no regression)
|
||||
|
||||
**6.2. Update unit tests**
|
||||
|
||||
- Remove tests for `InstrumentalReviewServer`
|
||||
- Add tests for `ReviewServer` with instrumental options
|
||||
- Add tests for `controller._run_human_review()`
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
|
||||
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
|
||||
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
|
||||
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
|
||||
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
|
||||
| Missing instrumental options in local mode | Fail clearly if options not provided |
|
||||
| Review server port conflicts | Keep existing port allocation logic |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Where does backing vocals analysis run for local CLI?**
|
||||
- Currently runs in `render_video_worker.py` for cloud
|
||||
- For local, need to run it after audio separation, before review
|
||||
- May need to call analysis function from controller
|
||||
|
||||
2. **How are instrumental stems exposed to local review?**
|
||||
- Need to serve audio files from ReviewServer
|
||||
- Or use file:// URLs (may have CORS issues)
|
||||
- Existing approach: ReviewServer already serves static files
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. `SKIP_CORRECTION=true` no longer skips human review
|
||||
2. Single combined review UI appears (no separate instrumental step)
|
||||
3. All platforms (CLI, cloud CLI, web) use same review flow
|
||||
4. No dead code remaining for standalone instrumental review
|
||||
@@ -1,401 +0,0 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -1,317 +0,0 @@
|
||||
# Plan: White Label B2B Karaoke Portals
|
||||
|
||||
**Created:** 2026-01-08
|
||||
**Branch:** feat/sess-20260108-0358-white-label-portal
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
|
||||
|
||||
**First customer: Vocal Star**
|
||||
- Subdomain: `vocalstar.nomadkaraoke.com`
|
||||
- Workflow: Upload original + instrumental audio (no audio search)
|
||||
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
|
||||
- Distribution: Download only (no Dropbox/GDrive/YouTube)
|
||||
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
|
||||
- [ ] Per-tenant frontend branding (logo, colors, metadata)
|
||||
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
|
||||
- [ ] Per-tenant default theme (always use Vocal Star theme)
|
||||
- [ ] Per-tenant allowed email domains for magic link auth
|
||||
- [ ] Per-tenant fixed API token support (for automation)
|
||||
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
|
||||
- [ ] Users scoped to tenant
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Minimal code duplication - single codebase serves all tenants
|
||||
- [ ] Easy to add new tenants via config, not code changes
|
||||
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
|
||||
- [ ] No performance impact on existing single-tenant flow
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decision: Config-Driven Multi-Tenancy
|
||||
|
||||
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
|
||||
|
||||
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
|
||||
2. **Frontend detects subdomain** and fetches tenant config from API
|
||||
3. **Backend middleware** extracts tenant from request and applies config
|
||||
4. **Firestore collections** get `tenant_id` field for isolation
|
||||
|
||||
This approach:
|
||||
- Requires no database migrations for new tenants
|
||||
- Configs can be managed via simple JSON files
|
||||
- Easy to test locally with query param override (`?tenant=vocalstar`)
|
||||
- Can evolve to admin UI later if needed
|
||||
|
||||
### Tenant Config Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vocalstar",
|
||||
"name": "Vocal Star",
|
||||
"subdomain": "vocalstar.nomadkaraoke.com",
|
||||
"branding": {
|
||||
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
|
||||
"primary_color": "#ffff00",
|
||||
"secondary_color": "#006CF9",
|
||||
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
|
||||
"site_title": "Vocal Star Karaoke Generator"
|
||||
},
|
||||
"features": {
|
||||
"audio_search": false,
|
||||
"file_upload": true,
|
||||
"youtube_upload": false,
|
||||
"dropbox_upload": false,
|
||||
"gdrive_upload": false,
|
||||
"theme_selection": false
|
||||
},
|
||||
"defaults": {
|
||||
"theme_id": "vocalstar",
|
||||
"distribution_mode": "download_only"
|
||||
},
|
||||
"auth": {
|
||||
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
|
||||
"fixed_tokens": ["<will be in Secret Manager>"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### 1. Subdomain Detection (Frontend)
|
||||
- Next.js middleware checks `request.headers.host`
|
||||
- If subdomain matches tenant, store tenant ID in cookie/context
|
||||
- Fetch `/api/tenant/config` to get branding/features
|
||||
- Apply CSS variables dynamically for colors
|
||||
- Swap logo component based on config
|
||||
|
||||
#### 2. Tenant Middleware (Backend)
|
||||
- FastAPI middleware extracts tenant from:
|
||||
- `X-Tenant-ID` header (set by frontend)
|
||||
- Or subdomain from `Host` header
|
||||
- Or query param `?tenant=` (dev only)
|
||||
- Attaches `tenant_id` to request state
|
||||
- Tenant config loaded from GCS (cached 5 min like themes)
|
||||
|
||||
#### 3. Auth Enhancement
|
||||
- `AuthService.validate_token_full()` already returns `user_email`
|
||||
- Add tenant validation: check if user's email domain is in tenant's allowed list
|
||||
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
|
||||
- Magic links include tenant context for proper redirect
|
||||
|
||||
#### 4. Data Isolation
|
||||
- Add `tenant_id: Optional[str]` to User and Job models
|
||||
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
|
||||
- Query filters: `where('tenant_id', '==', tenant_id or None)`
|
||||
- No schema migration needed - new field, optional
|
||||
|
||||
#### 5. Theme Integration
|
||||
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
|
||||
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
|
||||
- Tenant config specifies `default_theme_id: "vocalstar"`
|
||||
- When `theme_selection: false`, UI hides theme picker and uses default
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Tenant Foundation (Day 1-2)
|
||||
|
||||
1. [ ] **Create tenant model and config loader**
|
||||
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
|
||||
- Create `backend/services/tenant_service.py` with GCS config loading and caching
|
||||
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
|
||||
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
|
||||
|
||||
2. [ ] **Add tenant middleware**
|
||||
- Create `backend/api/middleware/tenant.py`
|
||||
- Extract tenant from subdomain or header
|
||||
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
|
||||
- Add `get_tenant()` dependency for routes
|
||||
|
||||
3. [ ] **Update data models**
|
||||
- Add `tenant_id: Optional[str] = None` to `User` model
|
||||
- Add `tenant_id: Optional[str] = None` to `Job` model
|
||||
- Update Firestore indexes in Pulumi for tenant-scoped queries
|
||||
|
||||
4. [ ] **Update auth for tenant awareness**
|
||||
- Modify `AuthService` to validate email domain against tenant config
|
||||
- Add tenant context to magic link generation
|
||||
- Support per-tenant fixed tokens in `auth_tokens` collection
|
||||
|
||||
### Phase 2: Upload Vocal Star Theme (Day 2)
|
||||
|
||||
5. [ ] **Prepare Vocal Star theme for GCS**
|
||||
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
|
||||
- Upload background images to `themes/vocalstar/assets/`
|
||||
- Upload Oswald font to `themes/vocalstar/assets/`
|
||||
- Upload `style_params.json` to `themes/vocalstar/`
|
||||
- Create preview image and upload to `themes/vocalstar/preview.png`
|
||||
- Add Vocal Star to `themes/_metadata.json`
|
||||
|
||||
6. [ ] **Create Vocal Star tenant config**
|
||||
- Create `tenants/vocalstar/config.json` in GCS
|
||||
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
|
||||
- Upload favicon to `tenants/vocalstar/favicon.ico`
|
||||
|
||||
### Phase 3: Frontend Tenant Support (Day 3-4)
|
||||
|
||||
7. [ ] **Add tenant context provider**
|
||||
- Create `frontend/lib/tenant.ts` with tenant config types
|
||||
- Create `frontend/hooks/useTenant.ts` for tenant state
|
||||
- Create `TenantProvider` component that fetches config on mount
|
||||
- Detect subdomain in `_app.tsx` or middleware
|
||||
|
||||
8. [ ] **Dynamic branding**
|
||||
- Create `TenantLogo` component that renders tenant or default logo
|
||||
- Update `globals.css` to support CSS variable overrides
|
||||
- Add `applyTenantColors()` function to set CSS variables from config
|
||||
- Update page metadata (title, favicon) from tenant config
|
||||
|
||||
9. [ ] **Feature-flagged UI**
|
||||
- Wrap audio search in `{tenant.features.audio_search && ...}`
|
||||
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
|
||||
- Wrap distribution options based on tenant features
|
||||
- Show simplified form for Vocal Star (just file upload fields)
|
||||
|
||||
10. [ ] **Update magic link flow**
|
||||
- Pass tenant context when requesting magic link
|
||||
- Magic link redirect goes to tenant subdomain
|
||||
- Verify endpoint extracts tenant from link context
|
||||
|
||||
### Phase 4: Backend Feature Enforcement (Day 4-5)
|
||||
|
||||
11. [ ] **Enforce tenant features in API**
|
||||
- Audio search endpoint: return 403 if `audio_search: false`
|
||||
- File upload: check `file_upload: true`
|
||||
- Distribution settings: ignore/error if disabled features requested
|
||||
- Theme selection: use default if `theme_selection: false`
|
||||
|
||||
12. [ ] **Job creation with tenant context**
|
||||
- `JobCreate` handler sets `tenant_id` from request state
|
||||
- Distribution config respects tenant defaults
|
||||
- Theme defaults to tenant's default theme
|
||||
|
||||
13. [ ] **Query scoping**
|
||||
- Update `JobManager.get_jobs()` to filter by tenant_id
|
||||
- Update `UserService` queries to include tenant_id
|
||||
- Admin endpoints bypass tenant filter (see all tenants)
|
||||
|
||||
### Phase 5: Infrastructure & Deployment (Day 5-6)
|
||||
|
||||
14. [ ] **DNS setup**
|
||||
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
|
||||
- Configure Cloudflare Pages to accept wildcard subdomain
|
||||
- Add SSL certificate for subdomain
|
||||
|
||||
15. [ ] **Backend CORS update**
|
||||
- Add `vocalstar.nomadkaraoke.com` to allowed origins
|
||||
- Or use pattern matching for `*.nomadkaraoke.com`
|
||||
|
||||
16. [ ] **Pulumi updates**
|
||||
- Add Firestore indexes for tenant_id queries
|
||||
- Add any new secrets (tenant fixed tokens)
|
||||
|
||||
### Phase 6: Testing & Documentation (Day 6-7)
|
||||
|
||||
17. [ ] **Testing**
|
||||
- Unit tests for TenantService config loading
|
||||
- Unit tests for tenant middleware
|
||||
- E2E test for Vocal Star flow (upload audio, generate, download)
|
||||
- Test magic link works for @vocal-star.com email
|
||||
|
||||
18. [ ] **Documentation**
|
||||
- Update CLAUDE.md with tenant info
|
||||
- Create `docs/WHITE-LABEL.md` with tenant setup guide
|
||||
- Document how to add new tenants
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/tenant.py` | Create | Tenant config models |
|
||||
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
|
||||
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
|
||||
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
|
||||
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
|
||||
| `backend/models/user.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/models/job.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
|
||||
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
|
||||
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/main.py` | Modify | Add tenant middleware |
|
||||
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
|
||||
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
|
||||
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
|
||||
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
|
||||
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
|
||||
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
|
||||
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
|
||||
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
|
||||
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `TenantService`: Config loading, caching, validation
|
||||
- `TenantMiddleware`: Subdomain extraction, header extraction
|
||||
- `AuthService`: Tenant email domain validation
|
||||
- `JobManager`: Tenant-scoped queries
|
||||
|
||||
### Integration Tests (Emulator)
|
||||
- Create job with tenant context
|
||||
- Query jobs filtered by tenant
|
||||
- Auth with tenant-allowed email domain
|
||||
- Auth rejection for wrong email domain
|
||||
|
||||
### E2E Tests
|
||||
- Vocal Star happy path: Upload audio files, review lyrics, download video
|
||||
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
|
||||
- Feature hiding: Verify audio search not shown on Vocal Star portal
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] **Q: Should tenant configs be in GCS or Firestore?**
|
||||
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
|
||||
|
||||
- [x] **Q: How to handle localhost development for specific tenant?**
|
||||
A: Query param override `?tenant=vocalstar` for local testing
|
||||
|
||||
- [x] **Q: Should admin portal be tenant-aware?**
|
||||
A: No - admins see all tenants. Admin routes bypass tenant filter.
|
||||
|
||||
- [x] **Q: Do we need tenant-specific email sender identity?**
|
||||
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
|
||||
|
||||
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
|
||||
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
|
||||
|
||||
1. Revert frontend to not check tenant
|
||||
2. Revert backend middleware to not extract tenant
|
||||
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
|
||||
4. No schema migration to reverse
|
||||
|
||||
For DNS: Simply remove CNAME record if needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
|
||||
- [ ] Portal shows Vocal Star branding (logo, colors)
|
||||
- [ ] Audio search is hidden; only file upload available
|
||||
- [ ] Theme is locked to Vocal Star style
|
||||
- [ ] Distribution options hidden; download only
|
||||
- [ ] Magic link works for @vocal-star.com emails
|
||||
- [ ] Fixed API token works for automation
|
||||
- [ ] Generated videos use Vocal Star theme
|
||||
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
|
||||
@@ -1,222 +0,0 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
description: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Goal: Identify inconsistencies, duplications, ambiguities, and underspecified items across the three core artifacts (`spec.md`, `plan.md`, `tasks.md`) before implementation. This command MUST run only after `/tasks` has successfully produced a complete `tasks.md`.
|
||||
|
||||
STRICTLY READ-ONLY: Do **not** modify any files. Output a structured analysis report. Offer an optional remediation plan (user must explicitly approve before any follow-up editing commands would be invoked manually).
|
||||
|
||||
Constitution Authority: The project constitution (`.specify/memory/constitution.md`) is **non-negotiable** within this analysis scope. Constitution conflicts are automatically CRITICAL and require adjustment of the spec, plan, or tasks—not dilution, reinterpretation, or silent ignoring of the principle. If a principle itself needs to change, that must occur in a separate, explicit constitution update outside `/analyze`.
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` once from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS. Derive absolute paths:
|
||||
- SPEC = FEATURE_DIR/spec.md
|
||||
- PLAN = FEATURE_DIR/plan.md
|
||||
- TASKS = FEATURE_DIR/tasks.md
|
||||
Abort with an error message if any required file is missing (instruct the user to run missing prerequisite command).
|
||||
|
||||
2. Load artifacts:
|
||||
- Parse spec.md sections: Overview/Context, Functional Requirements, Non-Functional Requirements, User Stories, Edge Cases (if present).
|
||||
- Parse plan.md: Architecture/stack choices, Data Model references, Phases, Technical constraints.
|
||||
- Parse tasks.md: Task IDs, descriptions, phase grouping, parallel markers [P], referenced file paths.
|
||||
- Load constitution `.specify/memory/constitution.md` for principle validation.
|
||||
|
||||
3. Build internal semantic models:
|
||||
- Requirements inventory: Each functional + non-functional requirement with a stable key (derive slug based on imperative phrase; e.g., "User can upload file" -> `user-can-upload-file`).
|
||||
- User story/action inventory.
|
||||
- Task coverage mapping: Map each task to one or more requirements or stories (inference by keyword / explicit reference patterns like IDs or key phrases).
|
||||
- Constitution rule set: Extract principle names and any MUST/SHOULD normative statements.
|
||||
|
||||
4. Detection passes:
|
||||
A. Duplication detection:
|
||||
- Identify near-duplicate requirements. Mark lower-quality phrasing for consolidation.
|
||||
B. Ambiguity detection:
|
||||
- Flag vague adjectives (fast, scalable, secure, intuitive, robust) lacking measurable criteria.
|
||||
- Flag unresolved placeholders (TODO, TKTK, ???, <placeholder>, etc.).
|
||||
C. Underspecification:
|
||||
- Requirements with verbs but missing object or measurable outcome.
|
||||
- User stories missing acceptance criteria alignment.
|
||||
- Tasks referencing files or components not defined in spec/plan.
|
||||
D. Constitution alignment:
|
||||
- Any requirement or plan element conflicting with a MUST principle.
|
||||
- Missing mandated sections or quality gates from constitution.
|
||||
E. Coverage gaps:
|
||||
- Requirements with zero associated tasks.
|
||||
- Tasks with no mapped requirement/story.
|
||||
- Non-functional requirements not reflected in tasks (e.g., performance, security).
|
||||
F. Inconsistency:
|
||||
- Terminology drift (same concept named differently across files).
|
||||
- Data entities referenced in plan but absent in spec (or vice versa).
|
||||
- Task ordering contradictions (e.g., integration tasks before foundational setup tasks without dependency note).
|
||||
- Conflicting requirements (e.g., one requires to use Next.js while other says to use Vue as the framework).
|
||||
|
||||
5. Severity assignment heuristic:
|
||||
- CRITICAL: Violates constitution MUST, missing core spec artifact, or requirement with zero coverage that blocks baseline functionality.
|
||||
- HIGH: Duplicate or conflicting requirement, ambiguous security/performance attribute, untestable acceptance criterion.
|
||||
- MEDIUM: Terminology drift, missing non-functional task coverage, underspecified edge case.
|
||||
- LOW: Style/wording improvements, minor redundancy not affecting execution order.
|
||||
|
||||
6. Produce a Markdown report (no file writes) with sections:
|
||||
|
||||
### Specification Analysis Report
|
||||
| ID | Category | Severity | Location(s) | Summary | Recommendation |
|
||||
|----|----------|----------|-------------|---------|----------------|
|
||||
| A1 | Duplication | HIGH | spec.md:L120-134 | Two similar requirements ... | Merge phrasing; keep clearer version |
|
||||
(Add one row per finding; generate stable IDs prefixed by category initial.)
|
||||
|
||||
Additional subsections:
|
||||
- Coverage Summary Table:
|
||||
| Requirement Key | Has Task? | Task IDs | Notes |
|
||||
- Constitution Alignment Issues (if any)
|
||||
- Unmapped Tasks (if any)
|
||||
- Metrics:
|
||||
* Total Requirements
|
||||
* Total Tasks
|
||||
* Coverage % (requirements with >=1 task)
|
||||
* Ambiguity Count
|
||||
* Duplication Count
|
||||
* Critical Issues Count
|
||||
|
||||
7. At end of report, output a concise Next Actions block:
|
||||
- If CRITICAL issues exist: Recommend resolving before `/implement`.
|
||||
- If only LOW/MEDIUM: User may proceed, but provide improvement suggestions.
|
||||
- Provide explicit command suggestions: e.g., "Run /specify with refinement", "Run /plan to adjust architecture", "Manually edit tasks.md to add coverage for 'performance-metrics'".
|
||||
|
||||
8. Ask the user: "Would you like me to suggest concrete remediation edits for the top N issues?" (Do NOT apply them automatically.)
|
||||
|
||||
Behavior rules:
|
||||
- NEVER modify files.
|
||||
- NEVER hallucinate missing sections—if absent, report them.
|
||||
- KEEP findings deterministic: if rerun without changes, produce consistent IDs and counts.
|
||||
- LIMIT total findings in the main table to 50; aggregate remainder in a summarized overflow note.
|
||||
- If zero issues found, emit a success report with coverage statistics and proceed recommendation.
|
||||
|
||||
Context: $ARGUMENTS
|
||||
@@ -1,158 +0,0 @@
|
||||
---
|
||||
description: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Goal: Detect and reduce ambiguity or missing decision points in the active feature specification and record the clarifications directly in the spec file.
|
||||
|
||||
Note: This clarification workflow is expected to run (and be completed) BEFORE invoking `/plan`. If the user explicitly states they are skipping clarification (e.g., exploratory spike), you may proceed, but must warn that downstream rework risk increases.
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --paths-only` from repo root **once** (combined `--json --paths-only` mode / `-Json -PathsOnly`). Parse minimal JSON payload fields:
|
||||
- `FEATURE_DIR`
|
||||
- `FEATURE_SPEC`
|
||||
- (Optionally capture `IMPL_PLAN`, `TASKS` for future chained flows.)
|
||||
- If JSON parsing fails, abort and instruct user to re-run `/specify` or verify feature branch environment.
|
||||
|
||||
2. Load the current spec file. Perform a structured ambiguity & coverage scan using this taxonomy. For each category, mark status: Clear / Partial / Missing. Produce an internal coverage map used for prioritization (do not output raw map unless no questions will be asked).
|
||||
|
||||
Functional Scope & Behavior:
|
||||
- Core user goals & success criteria
|
||||
- Explicit out-of-scope declarations
|
||||
- User roles / personas differentiation
|
||||
|
||||
Domain & Data Model:
|
||||
- Entities, attributes, relationships
|
||||
- Identity & uniqueness rules
|
||||
- Lifecycle/state transitions
|
||||
- Data volume / scale assumptions
|
||||
|
||||
Interaction & UX Flow:
|
||||
- Critical user journeys / sequences
|
||||
- Error/empty/loading states
|
||||
- Accessibility or localization notes
|
||||
|
||||
Non-Functional Quality Attributes:
|
||||
- Performance (latency, throughput targets)
|
||||
- Scalability (horizontal/vertical, limits)
|
||||
- Reliability & availability (uptime, recovery expectations)
|
||||
- Observability (logging, metrics, tracing signals)
|
||||
- Security & privacy (authN/Z, data protection, threat assumptions)
|
||||
- Compliance / regulatory constraints (if any)
|
||||
|
||||
Integration & External Dependencies:
|
||||
- External services/APIs and failure modes
|
||||
- Data import/export formats
|
||||
- Protocol/versioning assumptions
|
||||
|
||||
Edge Cases & Failure Handling:
|
||||
- Negative scenarios
|
||||
- Rate limiting / throttling
|
||||
- Conflict resolution (e.g., concurrent edits)
|
||||
|
||||
Constraints & Tradeoffs:
|
||||
- Technical constraints (language, storage, hosting)
|
||||
- Explicit tradeoffs or rejected alternatives
|
||||
|
||||
Terminology & Consistency:
|
||||
- Canonical glossary terms
|
||||
- Avoided synonyms / deprecated terms
|
||||
|
||||
Completion Signals:
|
||||
- Acceptance criteria testability
|
||||
- Measurable Definition of Done style indicators
|
||||
|
||||
Misc / Placeholders:
|
||||
- TODO markers / unresolved decisions
|
||||
- Ambiguous adjectives ("robust", "intuitive") lacking quantification
|
||||
|
||||
For each category with Partial or Missing status, add a candidate question opportunity unless:
|
||||
- Clarification would not materially change implementation or validation strategy
|
||||
- Information is better deferred to planning phase (note internally)
|
||||
|
||||
3. Generate (internally) a prioritized queue of candidate clarification questions (maximum 5). Do NOT output them all at once. Apply these constraints:
|
||||
- Maximum of 5 total questions across the whole session.
|
||||
- Each question must be answerable with EITHER:
|
||||
* A short multiple‑choice selection (2–5 distinct, mutually exclusive options), OR
|
||||
* A one-word / short‑phrase answer (explicitly constrain: "Answer in <=5 words").
|
||||
- Only include questions whose answers materially impact architecture, data modeling, task decomposition, test design, UX behavior, operational readiness, or compliance validation.
|
||||
- Ensure category coverage balance: attempt to cover the highest impact unresolved categories first; avoid asking two low-impact questions when a single high-impact area (e.g., security posture) is unresolved.
|
||||
- Exclude questions already answered, trivial stylistic preferences, or plan-level execution details (unless blocking correctness).
|
||||
- Favor clarifications that reduce downstream rework risk or prevent misaligned acceptance tests.
|
||||
- If more than 5 categories remain unresolved, select the top 5 by (Impact * Uncertainty) heuristic.
|
||||
|
||||
4. Sequential questioning loop (interactive):
|
||||
- Present EXACTLY ONE question at a time.
|
||||
- For multiple‑choice questions render options as a Markdown table:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| A | <Option A description> |
|
||||
| B | <Option B description> |
|
||||
| C | <Option C description> | (add D/E as needed up to 5)
|
||||
| Short | Provide a different short answer (<=5 words) | (Include only if free-form alternative is appropriate)
|
||||
|
||||
- For short‑answer style (no meaningful discrete options), output a single line after the question: `Format: Short answer (<=5 words)`.
|
||||
- After the user answers:
|
||||
* Validate the answer maps to one option or fits the <=5 word constraint.
|
||||
* If ambiguous, ask for a quick disambiguation (count still belongs to same question; do not advance).
|
||||
* Once satisfactory, record it in working memory (do not yet write to disk) and move to the next queued question.
|
||||
- Stop asking further questions when:
|
||||
* All critical ambiguities resolved early (remaining queued items become unnecessary), OR
|
||||
* User signals completion ("done", "good", "no more"), OR
|
||||
* You reach 5 asked questions.
|
||||
- Never reveal future queued questions in advance.
|
||||
- If no valid questions exist at start, immediately report no critical ambiguities.
|
||||
|
||||
5. Integration after EACH accepted answer (incremental update approach):
|
||||
- Maintain in-memory representation of the spec (loaded once at start) plus the raw file contents.
|
||||
- For the first integrated answer in this session:
|
||||
* Ensure a `## Clarifications` section exists (create it just after the highest-level contextual/overview section per the spec template if missing).
|
||||
* Under it, create (if not present) a `### Session YYYY-MM-DD` subheading for today.
|
||||
- Append a bullet line immediately after acceptance: `- Q: <question> → A: <final answer>`.
|
||||
- Then immediately apply the clarification to the most appropriate section(s):
|
||||
* Functional ambiguity → Update or add a bullet in Functional Requirements.
|
||||
* User interaction / actor distinction → Update User Stories or Actors subsection (if present) with clarified role, constraint, or scenario.
|
||||
* Data shape / entities → Update Data Model (add fields, types, relationships) preserving ordering; note added constraints succinctly.
|
||||
* Non-functional constraint → Add/modify measurable criteria in Non-Functional / Quality Attributes section (convert vague adjective to metric or explicit target).
|
||||
* Edge case / negative flow → Add a new bullet under Edge Cases / Error Handling (or create such subsection if template provides placeholder for it).
|
||||
* Terminology conflict → Normalize term across spec; retain original only if necessary by adding `(formerly referred to as "X")` once.
|
||||
- If the clarification invalidates an earlier ambiguous statement, replace that statement instead of duplicating; leave no obsolete contradictory text.
|
||||
- Save the spec file AFTER each integration to minimize risk of context loss (atomic overwrite).
|
||||
- Preserve formatting: do not reorder unrelated sections; keep heading hierarchy intact.
|
||||
- Keep each inserted clarification minimal and testable (avoid narrative drift).
|
||||
|
||||
6. Validation (performed after EACH write plus final pass):
|
||||
- Clarifications session contains exactly one bullet per accepted answer (no duplicates).
|
||||
- Total asked (accepted) questions ≤ 5.
|
||||
- Updated sections contain no lingering vague placeholders the new answer was meant to resolve.
|
||||
- No contradictory earlier statement remains (scan for now-invalid alternative choices removed).
|
||||
- Markdown structure valid; only allowed new headings: `## Clarifications`, `### Session YYYY-MM-DD`.
|
||||
- Terminology consistency: same canonical term used across all updated sections.
|
||||
|
||||
7. Write the updated spec back to `FEATURE_SPEC`.
|
||||
|
||||
8. Report completion (after questioning loop ends or early termination):
|
||||
- Number of questions asked & answered.
|
||||
- Path to updated spec.
|
||||
- Sections touched (list names).
|
||||
- Coverage summary table listing each taxonomy category with Status: Resolved (was Partial/Missing and addressed), Deferred (exceeds question quota or better suited for planning), Clear (already sufficient), Outstanding (still Partial/Missing but low impact).
|
||||
- If any Outstanding or Deferred remain, recommend whether to proceed to `/plan` or run `/clarify` again later post-plan.
|
||||
- Suggested next command.
|
||||
|
||||
Behavior rules:
|
||||
- If no meaningful ambiguities found (or all potential questions would be low-impact), respond: "No critical ambiguities detected worth formal clarification." and suggest proceeding.
|
||||
- If spec file missing, instruct user to run `/specify` first (do not create a new spec here).
|
||||
- Never exceed 5 total asked questions (clarification retries for a single question do not count as new questions).
|
||||
- Avoid speculative tech stack questions unless the absence blocks functional clarity.
|
||||
- Respect user early termination signals ("stop", "done", "proceed").
|
||||
- If no questions asked due to full coverage, output a compact coverage summary (all categories Clear) then suggest advancing.
|
||||
- If quota reached with unresolved high-impact categories remaining, explicitly flag them under Deferred with rationale.
|
||||
|
||||
Context for prioritization: $ARGUMENTS
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
description: Create or update the project constitution from interactive or provided principle inputs, ensuring all dependent templates stay in sync.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
You are updating the project constitution at `.specify/memory/constitution.md`. This file is a TEMPLATE containing placeholder tokens in square brackets (e.g. `[PROJECT_NAME]`, `[PRINCIPLE_1_NAME]`). Your job is to (a) collect/derive concrete values, (b) fill the template precisely, and (c) propagate any amendments across dependent artifacts.
|
||||
|
||||
Follow this execution flow:
|
||||
|
||||
1. Load the existing constitution template at `.specify/memory/constitution.md`.
|
||||
- Identify every placeholder token of the form `[ALL_CAPS_IDENTIFIER]`.
|
||||
**IMPORTANT**: The user might require less or more principles than the ones used in the template. If a number is specified, respect that - follow the general template. You will update the doc accordingly.
|
||||
|
||||
2. Collect/derive values for placeholders:
|
||||
- If user input (conversation) supplies a value, use it.
|
||||
- Otherwise infer from existing repo context (README, docs, prior constitution versions if embedded).
|
||||
- For governance dates: `RATIFICATION_DATE` is the original adoption date (if unknown ask or mark TODO), `LAST_AMENDED_DATE` is today if changes are made, otherwise keep previous.
|
||||
- `CONSTITUTION_VERSION` must increment according to semantic versioning rules:
|
||||
* MAJOR: Backward incompatible governance/principle removals or redefinitions.
|
||||
* MINOR: New principle/section added or materially expanded guidance.
|
||||
* PATCH: Clarifications, wording, typo fixes, non-semantic refinements.
|
||||
- If version bump type ambiguous, propose reasoning before finalizing.
|
||||
|
||||
3. Draft the updated constitution content:
|
||||
- Replace every placeholder with concrete text (no bracketed tokens left except intentionally retained template slots that the project has chosen not to define yet—explicitly justify any left).
|
||||
- Preserve heading hierarchy and comments can be removed once replaced unless they still add clarifying guidance.
|
||||
- Ensure each Principle section: succinct name line, paragraph (or bullet list) capturing non‑negotiable rules, explicit rationale if not obvious.
|
||||
- Ensure Governance section lists amendment procedure, versioning policy, and compliance review expectations.
|
||||
|
||||
4. Consistency propagation checklist (convert prior checklist into active validations):
|
||||
- Read `.specify/templates/plan-template.md` and ensure any "Constitution Check" or rules align with updated principles.
|
||||
- Read `.specify/templates/spec-template.md` for scope/requirements alignment—update if constitution adds/removes mandatory sections or constraints.
|
||||
- Read `.specify/templates/tasks-template.md` and ensure task categorization reflects new or removed principle-driven task types (e.g., observability, versioning, testing discipline).
|
||||
- Read each command file in `.specify/templates/commands/*.md` (including this one) to verify no outdated references (agent-specific names like CLAUDE only) remain when generic guidance is required.
|
||||
- Read any runtime guidance docs (e.g., `README.md`, `docs/quickstart.md`, or agent-specific guidance files if present). Update references to principles changed.
|
||||
|
||||
5. Produce a Sync Impact Report (prepend as an HTML comment at top of the constitution file after update):
|
||||
- Version change: old → new
|
||||
- List of modified principles (old title → new title if renamed)
|
||||
- Added sections
|
||||
- Removed sections
|
||||
- Templates requiring updates (✅ updated / ⚠ pending) with file paths
|
||||
- Follow-up TODOs if any placeholders intentionally deferred.
|
||||
|
||||
6. Validation before final output:
|
||||
- No remaining unexplained bracket tokens.
|
||||
- Version line matches report.
|
||||
- Dates ISO format YYYY-MM-DD.
|
||||
- Principles are declarative, testable, and free of vague language ("should" → replace with MUST/SHOULD rationale where appropriate).
|
||||
|
||||
7. Write the completed constitution back to `.specify/memory/constitution.md` (overwrite).
|
||||
|
||||
8. Output a final summary to the user with:
|
||||
- New version and bump rationale.
|
||||
- Any files flagged for manual follow-up.
|
||||
- Suggested commit message (e.g., `docs: amend constitution to vX.Y.Z (principle additions + governance update)`).
|
||||
|
||||
Formatting & Style Requirements:
|
||||
- Use Markdown headings exactly as in the template (do not demote/promote levels).
|
||||
- Wrap long rationale lines to keep readability (<100 chars ideally) but do not hard enforce with awkward breaks.
|
||||
- Keep a single blank line between sections.
|
||||
- Avoid trailing whitespace.
|
||||
|
||||
If the user supplies partial updates (e.g., only one principle revision), still perform validation and version decision steps.
|
||||
|
||||
If critical info missing (e.g., ratification date truly unknown), insert `TODO(<FIELD_NAME>): explanation` and include in the Sync Impact Report under deferred items.
|
||||
|
||||
Do not create a new template; always operate on the existing `.specify/memory/constitution.md` file.
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
description: Execute the implementation plan by processing and executing all tasks defined in tasks.md
|
||||
---
|
||||
|
||||
The user input can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute.
|
||||
|
||||
2. Load and analyze the implementation context:
|
||||
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
|
||||
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
|
||||
- **IF EXISTS**: Read data-model.md for entities and relationships
|
||||
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
|
||||
- **IF EXISTS**: Read research.md for technical decisions and constraints
|
||||
- **IF EXISTS**: Read quickstart.md for integration scenarios
|
||||
|
||||
3. Parse tasks.md structure and extract:
|
||||
- **Task phases**: Setup, Tests, Core, Integration, Polish
|
||||
- **Task dependencies**: Sequential vs parallel execution rules
|
||||
- **Task details**: ID, description, file paths, parallel markers [P]
|
||||
- **Execution flow**: Order and dependency requirements
|
||||
|
||||
4. Execute implementation following the task plan:
|
||||
- **Phase-by-phase execution**: Complete each phase before moving to the next
|
||||
- **Respect dependencies**: Run sequential tasks in order, parallel tasks [P] can run together
|
||||
- **Follow TDD approach**: Execute test tasks before their corresponding implementation tasks
|
||||
- **File-based coordination**: Tasks affecting the same files must run sequentially
|
||||
- **Validation checkpoints**: Verify each phase completion before proceeding
|
||||
|
||||
5. Implementation execution rules:
|
||||
- **Setup first**: Initialize project structure, dependencies, configuration
|
||||
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
|
||||
- **Core development**: Implement models, services, CLI commands, endpoints
|
||||
- **Integration work**: Database connections, middleware, logging, external services
|
||||
- **Polish and validation**: Unit tests, performance optimization, documentation
|
||||
|
||||
6. Progress tracking and error handling:
|
||||
- Report progress after each completed task
|
||||
- Halt execution if any non-parallel task fails
|
||||
- For parallel tasks [P], continue with successful tasks, report failed ones
|
||||
- Provide clear error messages with context for debugging
|
||||
- Suggest next steps if implementation cannot proceed
|
||||
- **IMPORTANT** For completed tasks, make sure to mark the task off as [X] in the tasks file.
|
||||
|
||||
7. Completion validation:
|
||||
- Verify all required tasks are completed
|
||||
- Check that implemented features match the original specification
|
||||
- Validate that tests pass and coverage meets requirements
|
||||
- Confirm the implementation follows the technical plan
|
||||
- Report final status with summary of completed work
|
||||
|
||||
Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/tasks` first to regenerate the task list.
|
||||
@@ -1,43 +0,0 @@
|
||||
---
|
||||
description: Execute the implementation planning workflow using the plan template to generate design artifacts.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Given the implementation details provided as an argument, do this:
|
||||
|
||||
1. Run `.specify/scripts/bash/setup-plan.sh --json` from the repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. All future file paths must be absolute.
|
||||
- BEFORE proceeding, inspect FEATURE_SPEC for a `## Clarifications` section with at least one `Session` subheading. If missing or clearly ambiguous areas remain (vague adjectives, unresolved critical choices), PAUSE and instruct the user to run `/clarify` first to reduce rework. Only continue if: (a) Clarifications exist OR (b) an explicit user override is provided (e.g., "proceed without clarification"). Do not attempt to fabricate clarifications yourself.
|
||||
2. Read and analyze the feature specification to understand:
|
||||
- The feature requirements and user stories
|
||||
- Functional and non-functional requirements
|
||||
- Success criteria and acceptance criteria
|
||||
- Any technical constraints or dependencies mentioned
|
||||
|
||||
3. Read the constitution at `.specify/memory/constitution.md` to understand constitutional requirements.
|
||||
|
||||
4. Execute the implementation plan template:
|
||||
- Load `.specify/templates/plan-template.md` (already copied to IMPL_PLAN path)
|
||||
- Set Input path to FEATURE_SPEC
|
||||
- Run the Execution Flow (main) function steps 1-9
|
||||
- The template is self-contained and executable
|
||||
- Follow error handling and gate checks as specified
|
||||
- Let the template guide artifact generation in $SPECS_DIR:
|
||||
* Phase 0 generates research.md
|
||||
* Phase 1 generates data-model.md, contracts/, quickstart.md
|
||||
* Phase 2 generates tasks.md
|
||||
- Incorporate user-provided details from arguments into Technical Context: $ARGUMENTS
|
||||
- Update Progress Tracking as you complete each phase
|
||||
|
||||
5. Verify execution completed:
|
||||
- Check Progress Tracking shows all phases complete
|
||||
- Ensure all required artifacts were generated
|
||||
- Confirm no ERROR states in execution
|
||||
|
||||
6. Report results with branch name, file paths, and generated artifacts.
|
||||
|
||||
Use absolute paths with the repository root for all file operations to avoid path issues.
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
description: Create or update the feature specification from a natural language feature description.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
The text the user typed after `/specify` in the triggering message **is** the feature description. Assume you always have it available in this conversation even if `$ARGUMENTS` appears literally below. Do not ask the user to repeat it unless they provided an empty command.
|
||||
|
||||
Given that feature description, do this:
|
||||
|
||||
1. Run the script `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS"` from repo root and parse its JSON output for BRANCH_NAME and SPEC_FILE. All file paths must be absolute.
|
||||
**IMPORTANT** You must only ever run this script once. The JSON is provided in the terminal as output - always refer to it to get the actual content you're looking for.
|
||||
2. Load `.specify/templates/spec-template.md` to understand required sections.
|
||||
3. Write the specification to SPEC_FILE using the template structure, replacing placeholders with concrete details derived from the feature description (arguments) while preserving section order and headings.
|
||||
4. Report completion with branch name, spec file path, and readiness for the next phase.
|
||||
|
||||
Note: The script creates and checks out the new branch and initializes the spec file before writing.
|
||||
@@ -1,62 +0,0 @@
|
||||
---
|
||||
description: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute.
|
||||
2. Load and analyze available design documents:
|
||||
- Always read plan.md for tech stack and libraries
|
||||
- IF EXISTS: Read data-model.md for entities
|
||||
- IF EXISTS: Read contracts/ for API endpoints
|
||||
- IF EXISTS: Read research.md for technical decisions
|
||||
- IF EXISTS: Read quickstart.md for test scenarios
|
||||
|
||||
Note: Not all projects have all documents. For example:
|
||||
- CLI tools might not have contracts/
|
||||
- Simple libraries might not need data-model.md
|
||||
- Generate tasks based on what's available
|
||||
|
||||
3. Generate tasks following the template:
|
||||
- Use `.specify/templates/tasks-template.md` as the base
|
||||
- Replace example tasks with actual tasks based on:
|
||||
* **Setup tasks**: Project init, dependencies, linting
|
||||
* **Test tasks [P]**: One per contract, one per integration scenario
|
||||
* **Core tasks**: One per entity, service, CLI command, endpoint
|
||||
* **Integration tasks**: DB connections, middleware, logging
|
||||
* **Polish tasks [P]**: Unit tests, performance, docs
|
||||
|
||||
4. Task generation rules:
|
||||
- Each contract file → contract test task marked [P]
|
||||
- Each entity in data-model → model creation task marked [P]
|
||||
- Each endpoint → implementation task (not parallel if shared files)
|
||||
- Each user story → integration test marked [P]
|
||||
- Different files = can be parallel [P]
|
||||
- Same file = sequential (no [P])
|
||||
|
||||
5. Order tasks by dependencies:
|
||||
- Setup before everything
|
||||
- Tests before implementation (TDD)
|
||||
- Models before services
|
||||
- Services before endpoints
|
||||
- Core before integration
|
||||
- Everything before polish
|
||||
|
||||
6. Include parallel execution examples:
|
||||
- Group [P] tasks that can run together
|
||||
- Show actual Task agent commands
|
||||
|
||||
7. Create FEATURE_DIR/tasks.md with:
|
||||
- Correct feature name from implementation plan
|
||||
- Numbered tasks (T001, T002, etc.)
|
||||
- Clear file paths for each task
|
||||
- Dependency notes
|
||||
- Parallel execution guidance
|
||||
|
||||
Context for task generation: $ARGUMENTS
|
||||
|
||||
The tasks.md should be immediately executable - each task must be specific enough that an LLM can complete it without additional context.
|
||||
@@ -1,368 +0,0 @@
|
||||
---
|
||||
name: Anchor Sequence Optimization
|
||||
overview: Replace the current multiprocessing-based anchor sequence algorithm with an efficient single-threaded implementation using hash-based n-gram indexing, greedy longest-first matching, and simplified scoring to achieve sub-30-second performance in both local and cloud environments.
|
||||
todos:
|
||||
- id: create-golden-fixtures
|
||||
content: Create script to generate 10 golden test fixtures from representative song data (varying lengths/complexity), run current algorithm to capture expected anchor outputs
|
||||
status: completed
|
||||
- id: create-golden-test
|
||||
content: Create test_anchor_sequence_golden.py with regression tests that verify anchor finder output matches fixtures, ensure tests pass with current implementation before any changes
|
||||
status: completed
|
||||
- id: remove-multiprocessing
|
||||
content: Remove multiprocessing.Pool from find_anchors() and _remove_overlapping_sequences(), replace with single-threaded loops
|
||||
status: completed
|
||||
- id: build-ngram-index
|
||||
content: Add _build_ngram_index() method to create hash-based lookup table for O(1) n-gram matching
|
||||
status: completed
|
||||
- id: greedy-skip
|
||||
content: Add covered_positions set tracking and skip logic to avoid processing overlapping n-grams
|
||||
status: completed
|
||||
- id: simple-scoring
|
||||
content: Add _simple_score_anchor() with rule-based heuristics, make spaCy scoring optional via use_nlp_scoring flag
|
||||
status: completed
|
||||
- id: optimize-overlap
|
||||
content: Replace O(n^2) overlap removal with O(n log n) sorted single-pass algorithm
|
||||
status: completed
|
||||
- id: add-perf-tests
|
||||
content: Add performance benchmark test asserting <30 second completion for all fixture sizes
|
||||
status: pending
|
||||
---
|
||||
|
||||
# Anchor Sequence Algorithm Optimization
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
The current `AnchorSequenceFinder` has critical performance issues:
|
||||
|
||||
### Root Causes of Hanging
|
||||
|
||||
1. **Multiprocessing Pool with fork()**: Python's `Pool()` uses `fork()` on Linux, which causes deadlocks in Cloud Run containers
|
||||
2. **spaCy model loading in forked processes**: The `PhraseAnalyzer` loads spaCy's `en_core_web_sm` model, which can hang when forked
|
||||
3. **Pool creation blocking**: The hang occurs before job submission, so timeout checks never trigger
|
||||
|
||||
### Algorithmic Inefficiencies
|
||||
|
||||
1. **O(n^2) n-gram matching**: Linear search through references for each n-gram
|
||||
2. **Redundant work**: Processes all n-gram lengths (3 to ~30) even though longer matches are preferred
|
||||
3. **O(n^2) overlap removal**: Each anchor checked against all previous anchors
|
||||
4. **Expensive phrase scoring**: spaCy NLP called twice per anchor (phrase + context)
|
||||
|
||||
## Proposed Solution: Hash-Based Greedy Algorithm
|
||||
|
||||
### Core Algorithm Changes
|
||||
|
||||
**1. Hash-Based N-gram Index (O(1) lookup instead of O(n))**
|
||||
|
||||
- Pre-build a dictionary mapping n-grams to their positions in each reference
|
||||
- Use tuple of cleaned words as key for O(1) lookup
|
||||
- Build index once during preprocessing
|
||||
|
||||
**2. Greedy Longest-First Matching**
|
||||
|
||||
- Process n-gram lengths from longest to shortest (current behavior)
|
||||
- BUT: Skip positions already covered by longer anchors immediately
|
||||
- Use a "covered positions" set to avoid redundant work
|
||||
|
||||
**3. Single-Threaded Processing**
|
||||
|
||||
- Remove all `multiprocessing.Pool` usage
|
||||
- Use efficient algorithms instead of parallelism
|
||||
- Works identically in local and cloud environments
|
||||
|
||||
**4. Simplified Phrase Scoring (Optional)**
|
||||
|
||||
- Make spaCy-based scoring optional via configuration flag
|
||||
- Default to rule-based heuristic scoring:
|
||||
- Prefer sequences starting/ending at word boundaries
|
||||
- Prefer sequences matching more reference sources
|
||||
- Prefer longer sequences
|
||||
- Option to enable full NLP scoring for higher-quality results offline
|
||||
|
||||
**5. O(n log n) Overlap Removal**
|
||||
|
||||
- Sort anchors by position
|
||||
- Single pass with interval tracking using sorted list
|
||||
- Skip anchors that overlap with any already-selected anchor
|
||||
|
||||
### New Data Structures
|
||||
|
||||
```python
|
||||
# N-gram index: maps (word1, word2, ...) -> {source: [positions]}
|
||||
ngram_index: Dict[Tuple[str, ...], Dict[str, List[int]]]
|
||||
|
||||
# Covered positions in transcription (for greedy skip)
|
||||
covered_trans_positions: Set[int]
|
||||
```
|
||||
|
||||
### Performance Targets
|
||||
|
||||
- **Target**: Complete in under 30 seconds for any song
|
||||
- **Typical song**: 200-400 words, 3 reference sources
|
||||
- **Expected performance**: 1-5 seconds for typical inputs
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Execution Order (Critical)
|
||||
|
||||
1. **Phase 0**: Create golden test fixtures and verify tests pass with current code
|
||||
2. **Phase 1-5**: Implement optimizations incrementally, running golden tests after each phase
|
||||
3. **Final**: Add performance benchmarks
|
||||
|
||||
### Files to Create/Modify
|
||||
|
||||
- `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py` (new)
|
||||
- `lyrics_transcriber_temp/tests/fixtures/anchor_golden/*.json` (new)
|
||||
- `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py` (new)
|
||||
- `lyrics_transcriber_temp/lyrics_transcriber/correction/anchor_sequence.py` (modify)
|
||||
|
||||
### Key Changes
|
||||
|
||||
**Phase 0: Golden Test Setup** (MUST complete and pass before any algorithm changes)
|
||||
|
||||
- Create fixture generation script with 10 representative test cases
|
||||
- Run script to generate fixtures using current implementation
|
||||
- Create golden test file that loads fixtures and verifies outputs
|
||||
- Verify all golden tests pass with current code
|
||||
|
||||
**Phase 1: Remove Multiprocessing**
|
||||
|
||||
- Replace `Pool().apply_async()` with direct function calls
|
||||
- Remove all pool creation, cleanup, and timeout handling
|
||||
- Single-threaded loop through n-gram lengths
|
||||
|
||||
**Phase 2: Add Hash-Based Index**
|
||||
|
||||
- New method `_build_ngram_index()` that builds lookup table during preprocessing
|
||||
- Modify `_find_matching_sources()` to use O(1) index lookup
|
||||
- Index built once, shared across all n-gram length iterations
|
||||
|
||||
**Phase 3: Greedy Position Skipping**
|
||||
|
||||
- Track covered transcription positions in a set
|
||||
- Skip n-grams that start in already-covered positions
|
||||
- Reduces redundant work significantly for songs with good matches
|
||||
|
||||
**Phase 4: Simplify Scoring**
|
||||
|
||||
- Add `use_nlp_scoring: bool = False` parameter to constructor
|
||||
- New method `_simple_score_anchor()` using rule-based heuristics
|
||||
- Keep spaCy scoring available for offline/high-quality mode
|
||||
|
||||
**Phase 5: Optimize Overlap Removal**
|
||||
|
||||
- Sort anchors by (start_position, -length)
|
||||
- Single pass: keep anchor if no overlap with last kept anchor
|
||||
- O(n log n) total instead of O(n^2)
|
||||
|
||||
## API Compatibility
|
||||
|
||||
The public interface remains unchanged:
|
||||
|
||||
- `find_anchors(transcribed, references, transcription_result) -> List[ScoredAnchor]`
|
||||
- `find_gaps(transcribed, anchors, references, transcription_result) -> List[GapSequence]`
|
||||
|
||||
New optional constructor parameters:
|
||||
|
||||
- `use_nlp_scoring: bool = False` - Enable spaCy-based phrase scoring (slower but higher quality)
|
||||
- Timeout parameter retained but less critical with efficient algorithm
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
If performance is still insufficient for edge cases:
|
||||
|
||||
1. **Early termination**: Stop after finding sufficient coverage (e.g., 80% of words anchored)
|
||||
2. **Adaptive n-gram range**: Start with medium lengths (8-15), expand only if needed
|
||||
3. **Sampling**: For very long songs, sample n-gram positions instead of exhaustive search
|
||||
|
||||
## Phase 0: Golden Test Setup (BEFORE any algorithm changes)
|
||||
|
||||
### Goal
|
||||
|
||||
Create regression tests using REAL song data from local cache to verify the optimized algorithm produces equivalent results.
|
||||
|
||||
### Cache File Structure (Understanding)
|
||||
|
||||
The local cache at `~/lyrics-transcriber-cache/` contains:
|
||||
|
||||
- **Reference lyrics**: `{provider}_{hash}_converted.json` where provider is spotify/genius/lrclib/musixmatch
|
||||
- Hash is based on (artist, title) lookup - same song shares same hash across providers
|
||||
- `_raw.json` contains metadata (track name, artist)
|
||||
- `_converted.json` contains LyricsData structure with segments/words
|
||||
- **Transcriptions**: `audioshake_{hash}_converted.json`
|
||||
- Hash is based on audio file content (different from reference hash)
|
||||
- Contains TranscriptionData structure
|
||||
- **Anchor results**: `anchors_{hash}.json`
|
||||
- Hash is computed from combined inputs (transcription + all references)
|
||||
|
||||
### Manual Song Selection
|
||||
|
||||
Identify 10 songs from cache with good coverage. Search raw files for track names:
|
||||
|
||||
### Finding Song Mappings
|
||||
|
||||
To link audioshake transcriptions to reference lyrics, search cache files:
|
||||
|
||||
```bash
|
||||
# Find songs by searching raw files for track names
|
||||
grep -h "trackName" ~/lyrics-transcriber-cache/spotify_*_raw.json | head -20
|
||||
|
||||
# Find audioshake files for a known song (e.g., search for filename patterns)
|
||||
ls ~/lyrics-transcriber-cache/audioshake_*_raw.json | head -20
|
||||
|
||||
# Search converted files for specific lyrics text
|
||||
grep -l "Napoleon did surrender" ~/lyrics-transcriber-cache/*_converted.json
|
||||
```
|
||||
|
||||
**Target song list** (reference_hash to be paired with audioshake_hash):
|
||||
|
||||
1. ABBA - Waterloo (refs: 3365f6a1e42388d84bcbfc0ac6f8aa6c) - short, repetitive chorus
|
||||
2. The Format - Time Bomb (refs: 3a3802ec28c4ab6689909b7ef784bb6f) - medium length
|
||||
3. Rancid - Time Bomb (refs: d02d20b854cbc6b890f274bd6c278752) - fast punk lyrics
|
||||
4. Minutemen - Storm in My House (refs: bc625704b26b307d8680ae5235b74994) - short
|
||||
5. + 6 more songs to be identified during fixture generation by exploring cache
|
||||
|
||||
### Fixture Generation Script
|
||||
|
||||
Create `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py`:
|
||||
|
||||
```python
|
||||
# Input: List of (name, audioshake_hash, reference_hash) tuples
|
||||
# For each:
|
||||
# 1. Load audioshake_{hash}_converted.json as transcription
|
||||
# 2. Load {spotify,genius,lrclib}_{hash}_converted.json as references
|
||||
# 3. Run current AnchorSequenceFinder
|
||||
# 4. Save complete fixture with inputs + outputs
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Accept a config file listing the 10 song mappings
|
||||
2. Load real LyricsData from cache files
|
||||
3. Run the CURRENT AnchorSequenceFinder implementation
|
||||
4. Save fixtures that capture the real behavior
|
||||
|
||||
### Fixture Format
|
||||
|
||||
```
|
||||
lyrics_transcriber_temp/tests/fixtures/anchor_golden/
|
||||
- waterloo_abba.json
|
||||
- time_bomb_format.json
|
||||
- time_bomb_rancid.json
|
||||
- ...
|
||||
```
|
||||
|
||||
Each fixture contains REAL data:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "waterloo_abba",
|
||||
"description": "ABBA - Waterloo, short song with repetitive chorus",
|
||||
"source_hashes": {
|
||||
"audioshake": "1654b7c0755fad7f23bc888308f274de",
|
||||
"references": "3365f6a1e42388d84bcbfc0ac6f8aa6c"
|
||||
},
|
||||
"transcription": {
|
||||
"segments": [...], // Full LyricsData from audioshake cache
|
||||
"word_count": 150
|
||||
},
|
||||
"references": {
|
||||
"spotify": { "segments": [...] },
|
||||
"genius": { "segments": [...] },
|
||||
"lrclib": { "segments": [...] }
|
||||
},
|
||||
"expected_anchors": [
|
||||
{
|
||||
"text": "My my at Waterloo Napoleon did surrender",
|
||||
"transcription_position": 0,
|
||||
"length": 7,
|
||||
"reference_positions": {"spotify": 0, "genius": 0, "lrclib": 0},
|
||||
"confidence": 1.0
|
||||
},
|
||||
// ... all anchors from current implementation
|
||||
],
|
||||
"expected_anchor_count": 25,
|
||||
"expected_word_coverage": 142,
|
||||
"expected_coverage_percent": 94.6
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Test
|
||||
|
||||
Create `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
FIXTURES_DIR = Path(__file__).parent.parent.parent / "fixtures" / "anchor_golden"
|
||||
|
||||
@pytest.fixture(params=list(FIXTURES_DIR.glob("*.json")))
|
||||
def golden_fixture(request):
|
||||
with open(request.param) as f:
|
||||
return json.load(f)
|
||||
|
||||
def test_anchor_count_matches(golden_fixture, anchor_finder):
|
||||
"""Anchor count should be within 10% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchor_count"]
|
||||
assert abs(len(anchors) - expected) <= expected * 0.1
|
||||
|
||||
def test_key_anchors_match(golden_fixture, anchor_finder):
|
||||
"""First, last, and longest anchors should match exactly."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchors"]
|
||||
# Compare first anchor
|
||||
assert anchors[0].text == expected[0]["text"]
|
||||
# Compare positions match
|
||||
...
|
||||
|
||||
def test_word_coverage(golden_fixture, anchor_finder):
|
||||
"""Total word coverage should be within 5% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
coverage = calculate_coverage(anchors, golden_fixture)
|
||||
expected = golden_fixture["expected_coverage_percent"]
|
||||
assert abs(coverage - expected) <= 5.0
|
||||
|
||||
def test_no_invalid_positions(golden_fixture, anchor_finder):
|
||||
"""All anchor positions should be valid."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
word_count = golden_fixture["transcription"]["word_count"]
|
||||
for anchor in anchors:
|
||||
assert 0 <= anchor.transcription_position < word_count
|
||||
```
|
||||
|
||||
### Validation Steps
|
||||
|
||||
1. **Generate fixtures from current implementation**:
|
||||
```bash
|
||||
cd lyrics_transcriber_temp
|
||||
python scripts/generate_anchor_fixtures.py
|
||||
```
|
||||
|
||||
2. **Verify fixtures were created**:
|
||||
```bash
|
||||
ls tests/fixtures/anchor_golden/
|
||||
# Should show 10 .json files
|
||||
```
|
||||
|
||||
3. **Run golden tests with CURRENT implementation**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
```
|
||||
|
||||
4. **All tests MUST pass before any algorithm changes**
|
||||
|
||||
5. **After each optimization phase, re-run golden tests**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
# Must still pass!
|
||||
```
|
||||
|
||||
|
||||
## Testing (Post-Optimization)
|
||||
|
||||
- Golden tests verify correctness (same outputs as before)
|
||||
- Existing unit tests verify API compatibility
|
||||
- Performance benchmark test with 30-second timeout assertion
|
||||
- Test with various song lengths via golden fixtures
|
||||
@@ -1,717 +0,0 @@
|
||||
---
|
||||
name: Instrumental Review UI
|
||||
overview: Add an intelligent instrumental selection workflow with audio analysis, waveform visualization, playback controls, and the ability to mute sections of backing vocals to create custom instrumentals. This replaces the current text-only CLI prompt with a web UI that works for both local and remote workflows, with shared core logic for DRY/maintainable code.
|
||||
todos:
|
||||
- id: shared-core-module
|
||||
content: Create karaoke_gen/instrumental_review/ shared module with AudioAnalyzer and AudioEditor classes (used by both local and remote)
|
||||
status: completed
|
||||
- id: shared-core-tests
|
||||
content: Add comprehensive unit tests for shared module with >80% coverage (tests/unit/test_instrumental_review.py)
|
||||
status: completed
|
||||
- id: backend-services
|
||||
content: Create backend service wrappers that use shared core module (audio_analysis_service.py, audio_editing_service.py)
|
||||
status: completed
|
||||
- id: backend-services-tests
|
||||
content: Add unit tests for backend services with mocked GCS interactions (backend/tests/test_audio_services.py)
|
||||
status: completed
|
||||
- id: api-endpoints
|
||||
content: Add new API endpoints for analysis, audio streaming, and custom instrumental creation
|
||||
status: completed
|
||||
- id: api-endpoints-tests
|
||||
content: Add API endpoint tests with >80% coverage (backend/tests/test_instrumental_api.py)
|
||||
status: completed
|
||||
- id: render-worker-integration
|
||||
content: Integrate audio analysis into render_video_worker before transitioning to awaiting_instrumental_selection
|
||||
status: completed
|
||||
- id: render-worker-tests
|
||||
content: Add/update render_video_worker tests to cover new analysis integration
|
||||
status: completed
|
||||
- id: frontend-components
|
||||
content: Create React components (WaveformDisplay, AudioPlayer, RegionSelector, InstrumentalReviewPage)
|
||||
status: completed
|
||||
- id: local-cli-integration
|
||||
content: Update local gen_cli.py to use shared core module with local file paths (feature parity with remote)
|
||||
status: completed
|
||||
- id: local-cli-review-server
|
||||
content: Add lightweight review server to local CLI for browser-based review (similar to lyrics review pattern)
|
||||
status: completed
|
||||
- id: remote-cli-update
|
||||
content: Update remote_cli.py to use new analysis API and open browser for review
|
||||
status: completed
|
||||
- id: cli-tests
|
||||
content: Add CLI tests verifying feature parity between local and remote modes
|
||||
status: completed
|
||||
- id: integration-tests
|
||||
content: Add end-to-end integration tests for the full instrumental review workflow
|
||||
status: completed
|
||||
---
|
||||
|
||||
# Intelligent Instrumental Selection Plan
|
||||
|
||||
## Current State
|
||||
|
||||
The instrumental selection workflow currently:
|
||||
|
||||
- Presents a simple 1/2 choice in the CLI (clean vs. with_backing)
|
||||
- Requires manual inspection in Audacity to evaluate backing vocals quality
|
||||
- Cannot handle the common "Scenario 3" where backing vocals need partial muting
|
||||
- Has no audio preview capability in remote mode
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────────┐
|
||||
│ INSTRUMENTAL REVIEW WORKFLOW │
|
||||
├────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. AUDIO ANALYSIS (automatic after render_video completes) │
|
||||
│ │ │
|
||||
│ ├── Analyze backing vocals stem for amplitude │
|
||||
│ ├── Detect audible segments (> -40dB threshold) │
|
||||
│ ├── Generate waveform visualization image │
|
||||
│ └── Store analysis results in job state_data │
|
||||
│ │
|
||||
│ 2. AUTO-DECISION CHECK │
|
||||
│ │ │
|
||||
│ └── IF no audible segments detected: │
|
||||
│ ├── Set recommended_selection = "clean" │
|
||||
│ └── Skip to step 4 (or show minimal UI with suggestion) │
|
||||
│ │
|
||||
│ 3. HUMAN REVIEW (Web UI) │
|
||||
│ │ │
|
||||
│ ├── Display waveform with time axis │
|
||||
│ ├── Show audible segment markers │
|
||||
│ ├── Audio playback with waveform-click seeking │
|
||||
│ ├── Toggle between: clean / with_backing / preview-custom │
|
||||
│ ├── Select regions to mute (click/drag on waveform) │
|
||||
│ └── Create custom instrumental if regions selected │
|
||||
│ │
|
||||
│ 4. SELECTION SUBMISSION │
|
||||
│ │ │
|
||||
│ └── POST /api/jobs/{job_id}/select-instrumental │
|
||||
│ with selection = "clean" | "with_backing" | "custom" │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Architecture: Shared Core for DRY/SOLID
|
||||
|
||||
To ensure feature parity between local and remote CLIs while keeping code maintainable, we use a **shared core module** pattern:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SHARED CORE ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ karaoke_gen/instrumental_review/ <── SHARED CORE MODULE │
|
||||
│ ├── __init__.py │
|
||||
│ ├── analyzer.py # AudioAnalyzer class (pure Python, no GCS) │
|
||||
│ ├── editor.py # AudioEditor class (pure Python, no GCS) │
|
||||
│ ├── waveform.py # WaveformGenerator class (matplotlib) │
|
||||
│ └── models.py # Pydantic models: AnalysisResult, MuteRegion │
|
||||
│ │
|
||||
│ ┌─────────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ LOCAL CLI │ │ REMOTE BACKEND │ │
|
||||
│ │ (karaoke-gen) │ │ (Cloud Run) │ │
|
||||
│ ├─────────────────────┤ ├──────────────────────┤ │
|
||||
│ │ │ │ │ │
|
||||
│ │ gen_cli.py │ │ audio_analysis_ │ │
|
||||
│ │ └── uses shared ───┼──────────────┼──► service.py │ │
|
||||
│ │ core directly │ │ └── wraps shared│ │
|
||||
│ │ with local │ │ core with │ │
|
||||
│ │ file paths │ │ GCS I/O │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ local_review_ │ │ audio_editing_ │ │
|
||||
│ │ server.py │ │ service.py │ │
|
||||
│ │ └── serves same ───┼──────────────┼──► (same pattern) │ │
|
||||
│ │ React UI │ │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ └─────────────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
│ BENEFITS: │
|
||||
│ ✓ Single source of truth for analysis logic │
|
||||
│ ✓ Bug fixes apply to both local and remote │
|
||||
│ ✓ Easy to test core logic without GCS mocks │
|
||||
│ ✓ Backend services are thin wrappers (Single Responsibility) │
|
||||
│ ✓ Same React UI served by both local server and Cloud Run │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Shared Core Module Design
|
||||
|
||||
```python
|
||||
# karaoke_gen/instrumental_review/analyzer.py
|
||||
class AudioAnalyzer:
|
||||
"""
|
||||
Analyzes audio files for backing vocals content.
|
||||
Pure Python - no cloud dependencies. Works with local file paths.
|
||||
"""
|
||||
def __init__(self, silence_threshold_db: float = -40.0,
|
||||
min_segment_duration_ms: int = 100):
|
||||
self.silence_threshold_db = silence_threshold_db
|
||||
self.min_segment_duration_ms = min_segment_duration_ms
|
||||
|
||||
def analyze(self, audio_path: str) -> AnalysisResult:
|
||||
"""Analyze audio file and return structured result."""
|
||||
...
|
||||
|
||||
def get_amplitude_envelope(self, audio_path: str,
|
||||
window_ms: int = 100) -> List[float]:
|
||||
"""Get amplitude envelope for waveform rendering."""
|
||||
...
|
||||
|
||||
# karaoke_gen/instrumental_review/editor.py
|
||||
class AudioEditor:
|
||||
"""
|
||||
Creates custom instrumentals by muting regions of backing vocals.
|
||||
Pure Python - no cloud dependencies.
|
||||
"""
|
||||
def create_custom_instrumental(
|
||||
self,
|
||||
clean_instrumental_path: str,
|
||||
backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
output_path: str
|
||||
) -> str:
|
||||
"""Create custom instrumental with muted regions."""
|
||||
...
|
||||
|
||||
# karaoke_gen/instrumental_review/waveform.py
|
||||
class WaveformGenerator:
|
||||
"""Generates waveform visualization images."""
|
||||
def generate(self, audio_path: str, output_path: str,
|
||||
segments: List[AudibleSegment] = None,
|
||||
width: int = 1200, height: int = 200) -> str:
|
||||
"""Generate waveform PNG with optional segment highlighting."""
|
||||
...
|
||||
```
|
||||
|
||||
## Implementation Components
|
||||
|
||||
### 1. Shared Core Module (karaoke_gen/instrumental_review/)
|
||||
|
||||
Create the shared module first - this is the foundation:
|
||||
|
||||
```python
|
||||
# karaoke_gen/instrumental_review/models.py
|
||||
from pydantic import BaseModel
|
||||
from typing import List, Optional
|
||||
from enum import Enum
|
||||
|
||||
class RecommendedSelection(str, Enum):
|
||||
CLEAN = "clean"
|
||||
WITH_BACKING = "with_backing"
|
||||
REVIEW_NEEDED = "review_needed"
|
||||
|
||||
class AudibleSegment(BaseModel):
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
duration_seconds: float
|
||||
avg_amplitude_db: float
|
||||
|
||||
class MuteRegion(BaseModel):
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
|
||||
class AnalysisResult(BaseModel):
|
||||
has_audible_content: bool
|
||||
total_duration_seconds: float
|
||||
audible_segments: List[AudibleSegment]
|
||||
recommended_selection: RecommendedSelection
|
||||
waveform_data: Optional[List[float]] = None # For JSON API response
|
||||
```
|
||||
|
||||
### 2. Backend Service Wrappers
|
||||
|
||||
Create `backend/services/audio_analysis_service.py`:
|
||||
|
||||
```python
|
||||
# Key functionality:
|
||||
- analyze_backing_vocals(audio_path) -> AnalysisResult
|
||||
- Uses pydub for audio loading
|
||||
- Calculates RMS amplitude in dB across time windows
|
||||
- Detects audible segments (contiguous regions > threshold)
|
||||
- Returns: {
|
||||
"has_audible_content": bool,
|
||||
"total_duration_seconds": float,
|
||||
"audible_segments": [
|
||||
{"start_seconds": float, "end_seconds": float,
|
||||
"duration_seconds": float, "avg_amplitude_db": float}
|
||||
],
|
||||
"recommended_selection": "clean" | "with_backing" | "review_needed"
|
||||
}
|
||||
|
||||
- generate_waveform_image(audio_path, output_path, segments=None) -> str
|
||||
- Uses matplotlib to render amplitude envelope over time
|
||||
- Highlights audible segments with different colors
|
||||
- Returns path to generated PNG image
|
||||
```
|
||||
|
||||
### 2. Audio Editing Service
|
||||
|
||||
Create `backend/services/audio_editing_service.py`:
|
||||
|
||||
```python
|
||||
# Key functionality:
|
||||
- create_custom_instrumental(
|
||||
clean_instrumental_path: str,
|
||||
backing_vocals_path: str,
|
||||
mute_regions: List[Tuple[float, float]], # [(start_s, end_s), ...]
|
||||
output_path: str
|
||||
) -> str
|
||||
- Loads backing vocals
|
||||
- Applies silence to mute_regions
|
||||
- Mixes with clean instrumental
|
||||
- Saves as new FLAC file
|
||||
```
|
||||
|
||||
### 3. Backend API Endpoints
|
||||
|
||||
Add to `backend/api/routes/jobs.py`:
|
||||
|
||||
```python
|
||||
GET /api/jobs/{job_id}/instrumental-analysis
|
||||
# Returns analysis data + waveform URL
|
||||
|
||||
GET /api/jobs/{job_id}/audio-stream/{stem_type}
|
||||
# Stream audio for playback (clean, backing_vocals, combined)
|
||||
|
||||
POST /api/jobs/{job_id}/create-custom-instrumental
|
||||
# Body: {"mute_regions": [[0.5, 2.3], [45.0, 48.5]]}
|
||||
# Creates custom instrumental, updates job state
|
||||
|
||||
POST /api/jobs/{job_id}/select-instrumental
|
||||
# Extended to accept: "clean" | "with_backing" | "custom"
|
||||
```
|
||||
|
||||
### 4. Frontend Web UI
|
||||
|
||||
Create `frontend/app/jobs/[id]/instrumental-review/page.tsx`:
|
||||
|
||||
```
|
||||
Components needed:
|
||||
- WaveformDisplay: SVG/Canvas waveform with time axis
|
||||
- AudioPlayer: HTML5 audio with controls + waveform sync
|
||||
- RegionSelector: Click/drag to select mute regions
|
||||
- InstrumentalOptions: Radio buttons for clean/backing/custom
|
||||
- SegmentList: Table showing detected audible segments
|
||||
```
|
||||
|
||||
Key interactions:
|
||||
|
||||
- Click on waveform to seek playback position
|
||||
- Drag on waveform to select mute region
|
||||
- Toggle audio source (clean/backing/preview)
|
||||
- Submit selection
|
||||
|
||||
### 5. Backend Worker Updates
|
||||
|
||||
Modify `backend/workers/render_video_worker.py`:
|
||||
|
||||
```python
|
||||
# After video render completes, before transitioning to AWAITING_INSTRUMENTAL_SELECTION:
|
||||
1. Download backing_vocals stem
|
||||
2. Run audio analysis
|
||||
3. Generate waveform image
|
||||
4. Upload waveform to GCS
|
||||
5. Store analysis results in job.state_data
|
||||
6. If no audible content: set recommended_selection = "clean"
|
||||
7. Transition to AWAITING_INSTRUMENTAL_SELECTION
|
||||
```
|
||||
|
||||
### 6. CLI Updates
|
||||
|
||||
Modify `karaoke_gen/utils/remote_cli.py`:
|
||||
|
||||
```python
|
||||
def handle_instrumental_selection(self, job_id: str):
|
||||
# Get analysis data
|
||||
analysis = self.client.get_instrumental_analysis(job_id)
|
||||
|
||||
# If no backing vocals detected, suggest auto-select
|
||||
if analysis['recommended_selection'] == 'clean':
|
||||
if self.config.non_interactive:
|
||||
# Auto-select clean
|
||||
self.client.select_instrumental(job_id, 'clean')
|
||||
return
|
||||
else:
|
||||
# Show suggestion, offer to open browser for verification
|
||||
self.logger.info("No backing vocals detected - recommending clean instrumental")
|
||||
# Open browser anyway so user can verify if desired
|
||||
|
||||
# Open browser to instrumental review UI
|
||||
self.open_instrumental_review_ui(job_id)
|
||||
|
||||
# Poll until selection made
|
||||
while job.status == 'awaiting_instrumental_selection':
|
||||
time.sleep(self.config.poll_interval)
|
||||
```
|
||||
|
||||
### 7. Local CLI Integration (Feature Parity)
|
||||
|
||||
The local CLI (`karaoke-gen`) uses the **same shared core module** as the backend:
|
||||
|
||||
```python
|
||||
# karaoke_gen/utils/gen_cli.py - Updated instrumental selection flow
|
||||
|
||||
def handle_instrumental_selection(self, output_dir: str, base_name: str):
|
||||
"""Handle instrumental selection using shared core module."""
|
||||
from karaoke_gen.instrumental_review import AudioAnalyzer, WaveformGenerator
|
||||
|
||||
# Paths to local files (already exist from audio separation)
|
||||
backing_vocals_path = os.path.join(output_dir, "stems",
|
||||
f"{base_name} (Backing Vocals ...).flac")
|
||||
clean_instrumental_path = os.path.join(output_dir, "stems",
|
||||
f"{base_name} (Instrumental ...).flac")
|
||||
|
||||
# Use SAME analyzer as backend
|
||||
analyzer = AudioAnalyzer(silence_threshold_db=-40.0)
|
||||
analysis = analyzer.analyze(backing_vocals_path)
|
||||
|
||||
# Generate waveform locally
|
||||
waveform_path = os.path.join(output_dir, "backing_vocals_waveform.png")
|
||||
WaveformGenerator().generate(backing_vocals_path, waveform_path,
|
||||
analysis.audible_segments)
|
||||
|
||||
if not analysis.has_audible_content:
|
||||
self.logger.info("No backing vocals detected - using clean instrumental")
|
||||
if not self.non_interactive:
|
||||
# Offer to open browser for verification
|
||||
...
|
||||
return "clean"
|
||||
|
||||
# Start local review server (serves same React UI)
|
||||
from karaoke_gen.instrumental_review.server import InstrumentalReviewServer
|
||||
server = InstrumentalReviewServer(
|
||||
output_dir=output_dir,
|
||||
base_name=base_name,
|
||||
analysis=analysis,
|
||||
waveform_path=waveform_path,
|
||||
)
|
||||
server.start_and_open_browser() # Blocking until user submits selection
|
||||
|
||||
return server.get_selection()
|
||||
```
|
||||
|
||||
### 8. Local Review Server
|
||||
|
||||
Create `karaoke_gen/instrumental_review/server.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
Lightweight local server for instrumental review.
|
||||
Serves the same React frontend used by the cloud backend.
|
||||
Similar pattern to LyricsTranscriber's ReviewServer.
|
||||
"""
|
||||
from http.server import HTTPServer, SimpleHTTPRequestHandler
|
||||
import json
|
||||
import threading
|
||||
|
||||
class InstrumentalReviewServer:
|
||||
"""
|
||||
Local HTTP server for instrumental review UI.
|
||||
|
||||
Endpoints:
|
||||
- GET /api/analysis - Returns analysis data
|
||||
- GET /api/audio/{type} - Streams audio files
|
||||
- GET /api/waveform - Returns waveform image
|
||||
- POST /api/create-custom - Creates custom instrumental
|
||||
- POST /api/select - Submits selection
|
||||
- GET /* - Serves React frontend static files
|
||||
"""
|
||||
def __init__(self, output_dir: str, base_name: str,
|
||||
analysis: AnalysisResult, waveform_path: str):
|
||||
...
|
||||
|
||||
def start_and_open_browser(self, port: int = 8765):
|
||||
"""Start server, open browser, block until selection submitted."""
|
||||
...
|
||||
|
||||
def get_selection(self) -> str:
|
||||
"""Get the user's selection after browser submission."""
|
||||
...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Test Coverage Requirements
|
||||
|
||||
All new code must meet the project's **70% minimum coverage** requirement, with critical paths targeting **>80%**:
|
||||
|
||||
| Module | Target Coverage | Test File |
|
||||
|
||||
|--------|----------------|-----------|
|
||||
|
||||
| `karaoke_gen/instrumental_review/analyzer.py` | >90% | `tests/unit/test_instrumental_analyzer.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/editor.py` | >90% | `tests/unit/test_instrumental_editor.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/waveform.py` | >80% | `tests/unit/test_instrumental_waveform.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/server.py` | >70% | `tests/unit/test_instrumental_server.py` |
|
||||
|
||||
| `backend/services/audio_analysis_service.py` | >80% | `backend/tests/test_audio_analysis_service.py` |
|
||||
|
||||
| `backend/services/audio_editing_service.py` | >80% | `backend/tests/test_audio_editing_service.py` |
|
||||
|
||||
| `backend/api/routes/jobs.py` (new endpoints) | >80% | `backend/tests/test_instrumental_api.py` |
|
||||
|
||||
### Test Categories
|
||||
|
||||
**1. Unit Tests (Shared Core) - tests/unit/test_instrumental_*.py**
|
||||
|
||||
```python
|
||||
# tests/unit/test_instrumental_analyzer.py
|
||||
class TestAudioAnalyzer:
|
||||
def test_analyze_silent_audio_returns_no_audible_content(self):
|
||||
"""Silent audio should return has_audible_content=False."""
|
||||
|
||||
def test_analyze_audio_with_content_detects_segments(self):
|
||||
"""Audio with content should detect audible segments."""
|
||||
|
||||
def test_segment_detection_respects_threshold(self):
|
||||
"""Segments below threshold should not be detected."""
|
||||
|
||||
def test_segment_merging_for_close_segments(self):
|
||||
"""Close segments should be merged into one."""
|
||||
|
||||
def test_recommended_selection_clean_for_silent(self):
|
||||
"""Silent audio should recommend 'clean'."""
|
||||
|
||||
def test_recommended_selection_review_for_content(self):
|
||||
"""Audio with content should recommend 'review_needed'."""
|
||||
|
||||
# tests/unit/test_instrumental_editor.py
|
||||
class TestAudioEditor:
|
||||
def test_mute_single_region(self):
|
||||
"""Single mute region should be applied correctly."""
|
||||
|
||||
def test_mute_multiple_regions(self):
|
||||
"""Multiple mute regions should all be applied."""
|
||||
|
||||
def test_mute_region_boundaries(self):
|
||||
"""Mute region at start/end of audio should work."""
|
||||
|
||||
def test_output_format_is_flac(self):
|
||||
"""Output should be FLAC format."""
|
||||
|
||||
def test_combines_with_clean_instrumental(self):
|
||||
"""Result should be backing vocals mixed with clean instrumental."""
|
||||
```
|
||||
|
||||
**2. Unit Tests (Backend Services) - backend/tests/test_audio_*_service.py**
|
||||
|
||||
```python
|
||||
# backend/tests/test_audio_analysis_service.py
|
||||
class TestAudioAnalysisService:
|
||||
@pytest.fixture
|
||||
def mock_storage(self):
|
||||
"""Mock GCS storage service."""
|
||||
|
||||
def test_analyze_downloads_from_gcs(self, mock_storage):
|
||||
"""Service should download audio from GCS before analysis."""
|
||||
|
||||
def test_analyze_uploads_waveform_to_gcs(self, mock_storage):
|
||||
"""Service should upload waveform image to GCS."""
|
||||
|
||||
def test_analyze_updates_job_state_data(self, mock_storage):
|
||||
"""Service should store analysis in job.state_data."""
|
||||
```
|
||||
|
||||
**3. API Endpoint Tests - backend/tests/test_instrumental_api.py**
|
||||
|
||||
```python
|
||||
# backend/tests/test_instrumental_api.py
|
||||
class TestInstrumentalAnalysisEndpoint:
|
||||
def test_get_analysis_returns_data(self, client, mock_job):
|
||||
"""GET /instrumental-analysis should return analysis data."""
|
||||
|
||||
def test_get_analysis_requires_correct_status(self, client, mock_job):
|
||||
"""Should return 400 if job not in correct status."""
|
||||
|
||||
class TestAudioStreamEndpoint:
|
||||
def test_stream_backing_vocals(self, client, mock_job):
|
||||
"""Should stream backing vocals audio."""
|
||||
|
||||
def test_stream_clean_instrumental(self, client, mock_job):
|
||||
"""Should stream clean instrumental audio."""
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
def test_create_with_mute_regions(self, client, mock_job):
|
||||
"""Should create custom instrumental with mute regions."""
|
||||
|
||||
def test_validates_mute_regions(self, client, mock_job):
|
||||
"""Should validate mute region boundaries."""
|
||||
```
|
||||
|
||||
**4. Integration Tests - tests/integration/test_instrumental_review.py**
|
||||
|
||||
```python
|
||||
# tests/integration/test_instrumental_review_flow.py
|
||||
class TestInstrumentalReviewFlow:
|
||||
"""End-to-end tests for instrumental review workflow."""
|
||||
|
||||
def test_full_flow_clean_selection(self):
|
||||
"""Test complete flow: analysis -> review -> select clean."""
|
||||
|
||||
def test_full_flow_with_backing_selection(self):
|
||||
"""Test complete flow: analysis -> review -> select with_backing."""
|
||||
|
||||
def test_full_flow_custom_creation(self):
|
||||
"""Test complete flow: analysis -> mute regions -> custom."""
|
||||
|
||||
def test_auto_recommend_clean_for_silent_backing(self):
|
||||
"""When no backing vocals, should recommend clean."""
|
||||
```
|
||||
|
||||
**5. CLI Feature Parity Tests - tests/unit/test_cli_feature_parity.py**
|
||||
|
||||
```python
|
||||
# tests/unit/test_cli_feature_parity.py
|
||||
class TestCLIFeatureParity:
|
||||
"""Verify local and remote CLIs have equivalent functionality."""
|
||||
|
||||
def test_both_clis_use_same_analyzer(self):
|
||||
"""Both CLIs should use AudioAnalyzer from shared module."""
|
||||
|
||||
def test_both_clis_support_custom_instrumental(self):
|
||||
"""Both CLIs should support creating custom instrumentals."""
|
||||
|
||||
def test_analysis_results_identical(self):
|
||||
"""Same audio should produce identical analysis from both CLIs."""
|
||||
```
|
||||
|
||||
### Test Data
|
||||
|
||||
Create test fixtures in `tests/data/instrumental_review/`:
|
||||
|
||||
- `silent_backing_vocals.flac` - Silent/near-silent audio (~10s)
|
||||
- `backing_vocals_with_content.flac` - Audio with clear backing vocals (~30s)
|
||||
- `backing_vocals_mixed_quality.flac` - Audio with some good/some bad sections (~30s)
|
||||
|
||||
Use pydub to programmatically generate these in `tests/conftest.py` fixtures.
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
| Category | Files to Create/Modify |
|
||||
|
||||
|----------|------------------------|
|
||||
|
||||
| **Shared Core** | `karaoke_gen/instrumental_review/__init__.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/models.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/analyzer.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/editor.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/waveform.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/server.py` (new) |
|
||||
|
||||
| **Backend Services** | `backend/services/audio_analysis_service.py` (new) |
|
||||
|
||||
| Backend Services | `backend/services/audio_editing_service.py` (new) |
|
||||
|
||||
| **API** | `backend/api/routes/jobs.py` (modify - add endpoints) |
|
||||
|
||||
| API | `backend/api/routes/instrumental.py` (new - if endpoints get large) |
|
||||
|
||||
| **Workers** | `backend/workers/render_video_worker.py` (modify) |
|
||||
|
||||
| **Models** | `backend/models/job.py` (add analysis fields to state_data) |
|
||||
|
||||
| **Frontend** | `frontend/app/jobs/[id]/instrumental-review/page.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/waveform-display.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/audio-player-with-waveform.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/region-selector.tsx` (new) |
|
||||
|
||||
| **Local CLI** | `karaoke_gen/utils/gen_cli.py` (modify) |
|
||||
|
||||
| **Remote CLI** | `karaoke_gen/utils/remote_cli.py` (modify) |
|
||||
|
||||
| **Tests - Shared** | `tests/unit/test_instrumental_analyzer.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_editor.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_waveform.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_server.py` (new) |
|
||||
|
||||
| **Tests - Backend** | `backend/tests/test_audio_analysis_service.py` (new) |
|
||||
|
||||
| Tests - Backend | `backend/tests/test_audio_editing_service.py` (new) |
|
||||
|
||||
| Tests - Backend | `backend/tests/test_instrumental_api.py` (new) |
|
||||
|
||||
| **Tests - Parity** | `tests/unit/test_cli_feature_parity.py` (new) |
|
||||
|
||||
| **Tests - Integration** | `tests/integration/test_instrumental_review.py` (new) |
|
||||
|
||||
| **Test Data** | `tests/conftest.py` (modify - add fixtures) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
Already available in `pyproject.toml`:
|
||||
|
||||
- `pydub>=0.25.1` - Audio loading and manipulation
|
||||
- `matplotlib>=3` - Waveform visualization
|
||||
- `numpy>=2` - Audio array processing
|
||||
|
||||
Frontend:
|
||||
|
||||
- React/Next.js (existing)
|
||||
- HTML5 Audio API (native)
|
||||
- Canvas or SVG for waveform (native)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Execute in this order to enable incremental testing:
|
||||
|
||||
### Phase 1: Shared Core Module (Foundation)
|
||||
|
||||
1. Create `karaoke_gen/instrumental_review/` package with models, analyzer, editor, waveform
|
||||
2. Add comprehensive unit tests for shared core (>90% coverage)
|
||||
3. Create test audio fixtures in `tests/conftest.py`
|
||||
|
||||
### Phase 2: Backend Integration
|
||||
|
||||
4. Create backend service wrappers with GCS integration
|
||||
5. Add backend service unit tests with mocked GCS
|
||||
6. Add new API endpoints to `backend/api/routes/jobs.py`
|
||||
7. Add API endpoint tests
|
||||
8. Integrate analysis into `render_video_worker.py`
|
||||
9. Update render worker tests
|
||||
|
||||
### Phase 3: Frontend
|
||||
|
||||
10. Create React components (WaveformDisplay, AudioPlayer, RegionSelector)
|
||||
11. Create instrumental review page
|
||||
12. Test frontend manually with backend
|
||||
|
||||
### Phase 4: CLI Integration
|
||||
|
||||
13. Update `remote_cli.py` to use new analysis API and open browser
|
||||
14. Create local review server `karaoke_gen/instrumental_review/server.py`
|
||||
15. Update `gen_cli.py` to use shared core and local server
|
||||
16. Add CLI feature parity tests
|
||||
|
||||
### Phase 5: Integration Testing
|
||||
|
||||
17. Add end-to-end integration tests
|
||||
18. Manual testing of full workflow (local and remote)
|
||||
|
||||
### Verification Checklist
|
||||
|
||||
- [ ] All unit tests pass: `pytest tests/unit/ -v`
|
||||
- [ ] All backend tests pass: `pytest backend/tests/ -v`
|
||||
- [ ] Coverage meets 70% minimum: `pytest --cov=karaoke_gen --cov-fail-under=70`
|
||||
- [ ] Local CLI works: `karaoke-gen "Artist" "Title"` (reaches instrumental selection)
|
||||
- [ ] Remote CLI works: `karaoke-gen-remote ./audio.flac "Artist" "Title"`
|
||||
- [ ] Both CLIs produce identical analysis for same audio
|
||||
- [ ] Custom instrumental creation works in both modes
|
||||
- [ ] Version bumped in `pyproject.toml`
|
||||
@@ -1,217 +0,0 @@
|
||||
---
|
||||
name: MidiCo-Style Synchronizer
|
||||
overview: Build a new MidiCo-style Synchronizer component for efficient re-syncing of lyrics timing, particularly for fixing progressive drift in the second half of songs. This involves refactoring the "Edit All" entry point and creating a high-performance timeline-based sync interface.
|
||||
todos:
|
||||
- id: mode-selection-modal
|
||||
content: Create ModeSelectionModal for 'Replace All' vs 'Re-sync Existing' choice
|
||||
status: completed
|
||||
- id: timeline-canvas
|
||||
content: Build canvas-based TimelineCanvas component for high-performance word block rendering
|
||||
status: completed
|
||||
- id: upcoming-words-bar
|
||||
content: Create UpcomingWordsBar component showing next unsynced words in fixed position
|
||||
status: completed
|
||||
- id: sync-controls
|
||||
content: Build SyncControls with Start Sync, Clear Sync, Unsync from Cursor, Delete Selected buttons
|
||||
status: completed
|
||||
- id: lyrics-synchronizer
|
||||
content: Create main LyricsSynchronizer component integrating all sub-components
|
||||
status: completed
|
||||
- id: unsync-from-cursor
|
||||
content: Add 'unsync from cursor' functionality to useManualSync hook
|
||||
status: completed
|
||||
- id: integrate-modal
|
||||
content: Integrate new synchronizer into ReplaceAllLyricsModal with mode selection
|
||||
status: completed
|
||||
- id: word-selection
|
||||
content: Implement single-click and drag-select for word blocks
|
||||
status: completed
|
||||
- id: zoom-slider
|
||||
content: Add smooth zoom slider with 4.5s-24s range
|
||||
status: completed
|
||||
- id: performance-test
|
||||
content: Test performance with long songs and optimize as needed
|
||||
status: completed
|
||||
---
|
||||
|
||||
# MidiCo-Style Synchronizer Implementation
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
The existing implementation in [`ReplaceAllLyricsModal.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/ReplaceAllLyricsModal.tsx) has two phases:
|
||||
|
||||
1. **Paste Phase** - User pastes new lyrics text
|
||||
2. **Sync Phase** - Manual sync using spacebar with timeline view
|
||||
|
||||
Key existing components:
|
||||
|
||||
- [`useManualSync.ts`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/hooks/useManualSync.ts) - Spacebar handling, tap/hold detection
|
||||
- [`EditTimelineSection.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/EditTimelineSection.tsx) - Timeline controls with zoom/scroll
|
||||
- [`TimelineEditor.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/TimelineEditor.tsx) - Timeline rendering
|
||||
|
||||
**Problems with current implementation:**
|
||||
|
||||
1. Forces full lyrics replacement - no option to keep existing sync data
|
||||
2. Performance issues (laggy, slow rendering)
|
||||
3. Poor UX for seeing upcoming words while syncing
|
||||
4. No "unsync from cursor" feature for fixing drift
|
||||
5. No two-level word block layout for overlapping segments
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Entry [Entry Point]
|
||||
EditAll[Edit All Button]
|
||||
EditAll --> ChoiceModal[Mode Selection Modal]
|
||||
ChoiceModal -->|Replace All| PastePhase[Paste New Lyrics]
|
||||
ChoiceModal -->|Re-sync Existing| SyncView[Synchronizer View]
|
||||
PastePhase --> SyncView
|
||||
end
|
||||
|
||||
subgraph Sync [Synchronizer Component]
|
||||
SyncView --> Timeline[Timeline Canvas]
|
||||
SyncView --> Controls[Control Buttons]
|
||||
SyncView --> UpcomingWords[Upcoming Words Display]
|
||||
Timeline --> WordBlocks[Word Blocks Renderer]
|
||||
Timeline --> Playhead[Playhead Marker]
|
||||
Timeline --> TimeBar[Time Bar]
|
||||
end
|
||||
|
||||
subgraph Audio [Audio Integration]
|
||||
AudioPlayer[Audio Player]
|
||||
AudioPlayer <--> SyncView
|
||||
end
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Entry Point Refactor
|
||||
|
||||
Modify the "Edit All" button flow to show a choice modal:
|
||||
|
||||
```typescript
|
||||
// New ModeSelectionModal component
|
||||
interface ModeSelectionModalProps {
|
||||
open: boolean
|
||||
onClose: () => void
|
||||
onSelectReplace: () => void // Goes to paste phase
|
||||
onSelectResync: () => void // Goes directly to sync view
|
||||
}
|
||||
```
|
||||
|
||||
- **Replace All Lyrics**: Current flow (paste box -> sync)
|
||||
- **Re-sync Existing Lyrics**: Load current segments directly into sync view
|
||||
|
||||
### Phase 2: New Synchronizer Component
|
||||
|
||||
Create a new high-performance `LyricsSynchronizer.tsx` component inspired by MidiCo:
|
||||
|
||||
#### Core Features
|
||||
|
||||
1. **Timeline Canvas** (HTML5 Canvas for performance)
|
||||
|
||||
- Time bar at top (MM:SS format with decisecond ticks)
|
||||
- Playhead with arrow indicator + vertical line
|
||||
- Red word blocks with text labels
|
||||
- Two-level word layout for overlapping segments
|
||||
- Zoom slider (4.5s to 24s visible range)
|
||||
- Horizontal scroll with smooth panning
|
||||
|
||||
2. **Upcoming Words Display** (dual location)
|
||||
|
||||
- Fixed position above timeline: next words as red/white blocks
|
||||
- On timeline: words appear to right of playhead during sync
|
||||
|
||||
3. **Control Buttons**
|
||||
|
||||
- **Start Sync** - Begin sync mode from first unsynced word
|
||||
- **Clear Sync** - Reset all word timings to null
|
||||
- **Edit Lyrics** - Modal to edit lyrics text (warns about timing loss)
|
||||
- **Unsync from Cursor** - Critical: reset all words after playhead position
|
||||
- **Split/Edit Word** - Edit single selected word
|
||||
- **Delete Selected** - Remove selected word blocks
|
||||
|
||||
4. **Word Selection**
|
||||
|
||||
- Single click: select one word (white border)
|
||||
- Click-drag on background: selection rectangle for multi-select
|
||||
- Click on time bar: set playhead position
|
||||
|
||||
5. **Spacebar Sync Logic** (reuse/enhance `useManualSync`)
|
||||
|
||||
- Tap: set start time, default 500ms duration
|
||||
- Hold: set both start and end times
|
||||
- Handle previous word end time calculation
|
||||
|
||||
### Phase 3: Performance Optimization
|
||||
|
||||
The current `TimelineEditor` uses DOM elements which causes lag. New approach:
|
||||
|
||||
1. **Canvas-based rendering** for word blocks and timeline
|
||||
|
||||
- Single canvas element, no DOM nodes per word
|
||||
- RequestAnimationFrame for smooth playhead updates
|
||||
- Only re-render changed regions
|
||||
|
||||
2. **Virtual scrolling** for very long songs
|
||||
|
||||
- Only render visible word blocks
|
||||
- Cache off-screen block positions
|
||||
|
||||
3. **Debounced state updates**
|
||||
|
||||
- Batch word timing updates
|
||||
- Throttle zoom/scroll handlers
|
||||
|
||||
### Phase 4: Apply/Cancel Flow
|
||||
|
||||
Changes are held in local state until user clicks Apply:
|
||||
|
||||
```typescript
|
||||
interface SynchronizerState {
|
||||
words: Word[] // Working copy of words
|
||||
segments: LyricsSegment[] // Working copy of segments
|
||||
isDirty: boolean // Has unsaved changes
|
||||
}
|
||||
```
|
||||
|
||||
- **Apply**: Commit changes to parent component
|
||||
- **Cancel**: Discard changes, close modal
|
||||
|
||||
## File Changes
|
||||
|
||||
### New Files
|
||||
|
||||
- `src/components/LyricsSynchronizer.tsx` - Main synchronizer component
|
||||
- `src/components/LyricsSynchronizer/TimelineCanvas.tsx` - Canvas-based timeline
|
||||
- `src/components/LyricsSynchronizer/UpcomingWordsBar.tsx` - Fixed upcoming words display
|
||||
- `src/components/LyricsSynchronizer/SyncControls.tsx` - Control buttons
|
||||
- `src/components/ModeSelectionModal.tsx` - Entry point choice modal
|
||||
|
||||
### Modified Files
|
||||
|
||||
- [`ReplaceAllLyricsModal.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/ReplaceAllLyricsModal.tsx) - Add mode selection, integrate new synchronizer
|
||||
- [`useManualSync.ts`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/hooks/useManualSync.ts) - Add "unsync from cursor" functionality
|
||||
|
||||
## Key UX Details from MidiCo Reference
|
||||
|
||||
Per the documentation at [`MIDICO_SYNCHRONIZER_REFERENCE.md`](lyrics_transcriber_temp/docs/MIDICO_SYNCHRONIZER_REFERENCE.md):
|
||||
|
||||
1. **Zoom range**: 4.5s (max zoom) to 24s (min zoom) visible
|
||||
2. **Timeline click behavior**:
|
||||
|
||||
- Time bar click = set playhead (no playback change)
|
||||
- Background click = nothing (prevent accidental deselection)
|
||||
- Word block click = select
|
||||
|
||||
3. **Two-level word blocks**: Adjacent/overlapping words display on alternating vertical levels
|
||||
4. **Playhead**: Small arrow in time bar + white vertical line spanning timeline height
|
||||
5. **Selection rectangle**: Click-drag on background creates selection box
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. Manual testing with the provided URL: `http://localhost:8000/?baseApiUrl=...`
|
||||
2. Test sync accuracy with songs that have known drift issues
|
||||
3. Performance testing with 400+ word songs
|
||||
4. Test "unsync from cursor" workflow for fixing drift
|
||||
@@ -1,83 +0,0 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
@@ -1,42 +0,0 @@
|
||||
# karaoke-gen backend environment — copy to .env and adjust.
|
||||
#
|
||||
# NOTE: scripts/run-backend-local.sh --with-emulators sets all the required
|
||||
# variables itself; you only need a .env for optional API keys or overrides.
|
||||
# None of the keys below are required to boot the backend locally.
|
||||
|
||||
# --- Core (defaults shown are what run-backend-local.sh uses in emulator mode) ---
|
||||
# ENVIRONMENT=development
|
||||
# GOOGLE_CLOUD_PROJECT=test-project
|
||||
# GCS_BUCKET_NAME=test-bucket
|
||||
# FIRESTORE_COLLECTION=test-jobs
|
||||
# ADMIN_TOKENS=local-dev-token
|
||||
# PORT=8000
|
||||
# FRONTEND_URL=http://localhost:3000
|
||||
|
||||
# --- Emulators (set automatically by run-backend-local.sh --with-emulators) ---
|
||||
# FIRESTORE_EMULATOR_HOST=127.0.0.1:8080
|
||||
# STORAGE_EMULATOR_HOST=http://127.0.0.1:4443
|
||||
|
||||
# --- Audio separation (pick one; without either, jobs stall at separation) ---
|
||||
# Local CPU/GPU separation via the audio-separator package. Models are
|
||||
# downloaded into this directory on first use (~2GB). CPU works but is slow.
|
||||
# MODEL_DIR=./models
|
||||
# ...or point at a remote audio-separator API service:
|
||||
# AUDIO_SEPARATOR_API_URL=
|
||||
|
||||
# --- Lyrics transcription ---
|
||||
# The lyrics worker reads AUDIOSHAKE_API_TOKEN (not AUDIOSHAKE_API_KEY).
|
||||
# Without it, transcription falls back to local Whisper if installed
|
||||
# (poetry install --extras local-whisper).
|
||||
# AUDIOSHAKE_API_TOKEN=
|
||||
|
||||
# --- Lyrics sources (optional; improves reference lyrics) ---
|
||||
# GENIUS_API_KEY=
|
||||
# SPOTIFY_COOKIE_SP_DC=
|
||||
|
||||
# --- Email (optional; without it, emails are logged to the console) ---
|
||||
# POSTMARK_SERVER_TOKEN=
|
||||
|
||||
# --- Payments (optional; only needed to exercise payment endpoints) ---
|
||||
# STRIPE_SECRET_KEY=
|
||||
# STRIPE_WEBHOOK_SECRET=
|
||||
@@ -1,73 +0,0 @@
|
||||
# .gcloudignore - Exclude files from Cloud Build uploads
|
||||
# This dramatically reduces upload time (from 218MB to ~5-10MB)
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend
|
||||
node_modules/
|
||||
frontend-react/node_modules/
|
||||
frontend-react/dist/
|
||||
frontend-react/.vite/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
|
||||
# Documentation (not needed for Docker build)
|
||||
docs/
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Large test files
|
||||
tests/data/*.mp3
|
||||
tests/data/*.wav
|
||||
tests/data/*.mp4
|
||||
tests/data/*.mkv
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# Misc
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
@@ -1,58 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Nomad Karaoke pre-commit hook (karaoke-gen variant)
|
||||
# 1. Gitleaks secret scan (fail fast)
|
||||
# 2. i18n auto-translation when en.json changes
|
||||
# Enable: git config core.hooksPath .githooks
|
||||
set -eu
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Gitleaks secret scan
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
if command -v gitleaks >/dev/null 2>&1; then
|
||||
if ! gitleaks protect --staged --redact --no-banner; then
|
||||
printf '\n\033[31m✗ gitleaks detected a potential secret in staged changes.\033[0m\n' >&2
|
||||
printf ' • If real: remove it, rotate the credential, then re-commit.\n' >&2
|
||||
printf ' • If false positive: add an allowlist entry to .gitleaks.toml.\n' >&2
|
||||
printf ' • Emergency override (not recommended): git commit --no-verify\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
printf '\033[33mwarn:\033[0m gitleaks not installed — skipping secret scan\n' >&2
|
||||
printf ' install: brew install gitleaks\n' >&2
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. i18n auto-translation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
STAGED_EN=$(git diff --cached --name-only --diff-filter=ACM | grep -E '(messages|translations)/en\.json$' || true)
|
||||
|
||||
if [ -z "$STAGED_EN" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "i18n: English translations changed — running translation pipeline..."
|
||||
|
||||
for en_file in $STAGED_EN; do
|
||||
dir=$(dirname "$en_file")
|
||||
|
||||
if [[ "$dir" == frontend/messages ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
elif [[ "$dir" == backend/translations ]]; then
|
||||
script="frontend/scripts/translate.py"
|
||||
else
|
||||
echo " Warning: Unknown messages dir $dir, skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
echo " Translating $dir..."
|
||||
if python "$script" --messages-dir "$dir" --target all --skip-review 2>&1 | tail -5; then
|
||||
git add "$dir"/*.json
|
||||
echo " Done: $dir"
|
||||
else
|
||||
echo " Warning: Translation failed for $dir (GCP auth may be missing). CI will catch this."
|
||||
fi
|
||||
done
|
||||
|
||||
echo "i18n: Translation complete."
|
||||
@@ -1,128 +0,0 @@
|
||||
# GitHub Actions CI/CD
|
||||
|
||||
This directory contains GitHub Actions workflows for automated testing and deployment.
|
||||
|
||||
## Workflows
|
||||
|
||||
### `test.yml` - Automated Testing & Deployment (CI/CD)
|
||||
|
||||
**Triggers:**
|
||||
- Push to `main` or `replace-modal-with-google-cloud` branches
|
||||
- Pull requests to `main`
|
||||
|
||||
**Jobs:**
|
||||
|
||||
#### Testing Phase (runs in parallel)
|
||||
|
||||
1. **Unit Tests** (~30s)
|
||||
- Runs 62 unit tests with mocked dependencies
|
||||
- Fast validation of business logic
|
||||
- Uses Python 3.12 with pip caching
|
||||
|
||||
2. **Emulator Integration Tests** (~2min)
|
||||
- Runs 11 integration tests against local GCP emulators
|
||||
- Tests real Firestore and GCS interactions
|
||||
- Requires:
|
||||
- Java 21 (for Firestore emulator)
|
||||
- Google Cloud SDK (for Firestore emulator)
|
||||
- Docker (for GCS emulator via fake-gcs-server)
|
||||
- Uses the same `scripts/run-emulator-tests.sh` as local development
|
||||
|
||||
3. **Code Quality** (~20s)
|
||||
- Basic Python syntax validation
|
||||
- Import checks
|
||||
- Optional: flake8, black, isort
|
||||
|
||||
#### Deployment Phase (runs after all tests pass)
|
||||
|
||||
4. **Deploy to Cloud Run** (~3-4min)
|
||||
- **Only runs on push to `replace-modal-with-google-cloud` branch** (not on PRs)
|
||||
- Builds Docker image and pushes to Artifact Registry
|
||||
- Deploys to Cloud Run with proper configuration
|
||||
- Verifies deployment health
|
||||
- **Requires GitHub secrets:**
|
||||
- `GCP_SA_KEY` - Service account key JSON
|
||||
- `ADMIN_TOKENS` - Admin authentication tokens
|
||||
|
||||
**Total Runtime:**
|
||||
- **Testing only** (PRs): ~3 minutes
|
||||
- **Testing + Deployment** (push to branch): ~7 minutes
|
||||
|
||||
## Local vs CI
|
||||
|
||||
The CI runs the exact same tests as local development:
|
||||
|
||||
```bash
|
||||
# Local development
|
||||
./scripts/run-tests.sh # Unit tests (same as CI job 1)
|
||||
./scripts/run-emulator-tests.sh # Emulator tests (same as CI job 2)
|
||||
python backend/validate.py # Validation (same as CI job 3)
|
||||
|
||||
# CI automatically runs all three on every push
|
||||
```
|
||||
|
||||
## Viewing Results
|
||||
|
||||
1. Go to the [Actions tab](../../actions) in GitHub
|
||||
2. Click on the latest workflow run
|
||||
3. View job logs and test results
|
||||
4. Download test artifacts (coverage reports) if needed
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
Tests are automatically discovered by pytest. To add new tests:
|
||||
|
||||
1. **Unit tests:** Add to `backend/tests/test_*.py`
|
||||
2. **Emulator tests:** Add to `backend/tests/emulator/test_*.py`
|
||||
3. **Cloud tests:** Add to `backend/tests/test_api_integration.py` (runs manually)
|
||||
|
||||
The CI will automatically pick them up!
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Emulator Tests Fail in CI
|
||||
|
||||
**Common issues:**
|
||||
- Emulator startup timeout → Increase wait time in `scripts/start-emulators.sh`
|
||||
- Docker not available → Check GitHub Actions runner compatibility
|
||||
- Port conflicts → Emulators use ports 8080 (Firestore) and 4443 (GCS)
|
||||
|
||||
### CI is Slow
|
||||
|
||||
**Optimizations:**
|
||||
- ✅ Python pip caching enabled
|
||||
- ✅ Jobs run in parallel
|
||||
- ✅ Docker image pre-pulled
|
||||
- Future: Cache gcloud SDK installation
|
||||
|
||||
### Tests Pass Locally But Fail in CI
|
||||
|
||||
**Debug steps:**
|
||||
1. Check Python version matches (3.12)
|
||||
2. Check environment variables are set correctly
|
||||
3. Review CI logs for missing dependencies
|
||||
4. Run tests in a clean venv locally
|
||||
|
||||
## Setting Up Continuous Deployment
|
||||
|
||||
The CD pipeline is configured but requires GitHub secrets to be set. See the [CD Setup Guide](../docs/03-deployment/CD-SETUP.md) for detailed instructions.
|
||||
|
||||
**Quick setup:**
|
||||
|
||||
1. Create GCP service account with deployment permissions
|
||||
2. Download service account key JSON
|
||||
3. Add GitHub secrets:
|
||||
- `GCP_SA_KEY` - Full service account key JSON content
|
||||
- `ADMIN_TOKENS` - Comma-separated admin tokens
|
||||
|
||||
Once configured, every push to `replace-modal-with-google-cloud` will automatically deploy to Cloud Run after all tests pass.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Add test coverage reporting (coveralls, codecov)
|
||||
- [x] ✅ Add deployment workflow (deploy to Cloud Run after tests pass)
|
||||
- [ ] Add frontend tests (when frontend is built)
|
||||
- [ ] Add performance benchmarking
|
||||
- [ ] Add security scanning (dependabot, snyk)
|
||||
- [ ] Migrate to Workload Identity Federation (more secure than service account keys)
|
||||
|
||||
@@ -1,115 +0,0 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
@@ -1,327 +0,0 @@
|
||||
name: Build GHA Runner Images
|
||||
|
||||
# Produces the custom GCE images used by the ephemeral GitHub Actions runner
|
||||
# dispatcher: gha-runner-general, gha-runner-build, gha-runner-gpu,
|
||||
# gha-runner-gpu-windows.
|
||||
#
|
||||
# Flow per variant:
|
||||
# 1. Create a temporary build VM from the variant's base image with the
|
||||
# variant's provisioning script as its startup script:
|
||||
# - Linux variants: debian-12 + runner-image-provision.sh (startup-script)
|
||||
# - gpu-windows: windows-2022 + runner-image-provision.ps1
|
||||
# (windows-startup-script-ps1)
|
||||
# 2. Poll the serial console for the READY/FAILED marker (survives reboots).
|
||||
# 3. Stop the VM, snapshot its boot disk to a new image in the `gha-runner-<variant>`
|
||||
# image family with name gha-runner-<variant>-<timestamp>.
|
||||
# 4. Delete the build VM (boot disk auto-deletes).
|
||||
# 5. Delete older images in the family, keeping only the newest (GPU/windows images
|
||||
# are 15-21 GiB each; roll back by re-running this workflow at an older ref).
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
variants:
|
||||
description: 'Comma-separated variants to build (general,build,gpu,gpu-windows) or "all" (= all except build)'
|
||||
required: false
|
||||
default: 'all'
|
||||
skip_prune:
|
||||
description: 'Skip pruning (deletion) of old images (debugging)'
|
||||
type: boolean
|
||||
required: false
|
||||
default: false
|
||||
schedule:
|
||||
# Monthly on the 1st at 02:00 UTC
|
||||
- cron: '0 2 1 * *'
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write
|
||||
|
||||
env:
|
||||
GCP_PROJECT: nomadkaraoke
|
||||
BUILD_REGION: us-central1
|
||||
BUILD_ZONE: us-central1-a
|
||||
BASE_IMAGE: projects/debian-cloud/global/images/family/debian-12
|
||||
KEEP_IMAGES_PER_FAMILY: 1
|
||||
|
||||
jobs:
|
||||
resolve-matrix:
|
||||
name: Resolve variant matrix
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
variants: ${{ steps.resolve.outputs.variants }}
|
||||
steps:
|
||||
- id: resolve
|
||||
run: |
|
||||
requested='${{ github.event.inputs.variants || 'all' }}'
|
||||
if [[ "$requested" == "all" || -z "$requested" ]]; then
|
||||
# "build" (docker-build label) is excluded from "all": no job uses it
|
||||
# since deploy-backend moved to ubuntu-latest + Cloud Build
|
||||
# (2026-10-01). Still buildable via an explicit variants=build.
|
||||
echo 'variants=["general","gpu","gpu-windows"]' >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
json="$(echo "$requested" | tr ',' '\n' | jq -R . | jq -sc .)"
|
||||
echo "variants=$json" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
build:
|
||||
name: Build ${{ matrix.variant }} image
|
||||
needs: resolve-matrix
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
variant: ${{ fromJSON(needs.resolve-matrix.outputs.variants) }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to GCP
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
project_id: ${{ env.GCP_PROJECT }}
|
||||
|
||||
- name: Resolve build parameters
|
||||
id: params
|
||||
run: |
|
||||
variant='${{ matrix.variant }}'
|
||||
ts="$(date -u +%Y%m%d-%H%M%S)"
|
||||
name="gha-runner-${variant}-${ts}"
|
||||
family="gha-runner-${variant}"
|
||||
vm="gha-image-build-${variant}-${ts}"
|
||||
|
||||
# Per-variant base image + startup-script metadata key: Windows
|
||||
# VMs only execute scripts under windows-startup-script-ps1.
|
||||
image_flag="--image=${{ env.BASE_IMAGE }}"
|
||||
metadata_file="startup-script=infrastructure/scripts/runner-image-provision.sh"
|
||||
|
||||
case "$variant" in
|
||||
general|build)
|
||||
machine="n2-standard-8"
|
||||
disk_size="50"
|
||||
accel=""
|
||||
;;
|
||||
gpu)
|
||||
# Build with a T4 attached so DKMS module is built and validated
|
||||
# against the kernel that ships in the image.
|
||||
machine="n1-standard-4"
|
||||
disk_size="200"
|
||||
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
|
||||
;;
|
||||
gpu-windows)
|
||||
# T4 attached so the GRID driver install can validate WDDM mode
|
||||
# (required for DirectML). Keep disk in sync with the
|
||||
# gpu-windows FamilySpec in runner_manager/ephemeral.py.
|
||||
machine="n1-standard-4"
|
||||
disk_size="100"
|
||||
accel="--maintenance-policy=TERMINATE --accelerator=type=nvidia-tesla-t4,count=1"
|
||||
image_flag="--image-family=windows-2022 --image-project=windows-cloud"
|
||||
metadata_file="windows-startup-script-ps1=infrastructure/scripts/runner-image-provision.ps1"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown variant: $variant" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
{
|
||||
echo "name=$name"
|
||||
echo "family=$family"
|
||||
echo "vm=$vm"
|
||||
echo "machine=$machine"
|
||||
echo "disk_size=$disk_size"
|
||||
echo "accel=$accel"
|
||||
echo "image_flag=$image_flag"
|
||||
echo "metadata_file=$metadata_file"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create build VM
|
||||
run: |
|
||||
set -euo pipefail
|
||||
gcloud compute instances create '${{ steps.params.outputs.vm }}' \
|
||||
--project='${{ env.GCP_PROJECT }}' \
|
||||
--zone='${{ env.BUILD_ZONE }}' \
|
||||
--machine-type='${{ steps.params.outputs.machine }}' \
|
||||
${{ steps.params.outputs.image_flag }} \
|
||||
--boot-disk-size='${{ steps.params.outputs.disk_size }}GB' \
|
||||
--boot-disk-type=pd-balanced \
|
||||
--boot-disk-auto-delete \
|
||||
--service-account="${{ secrets.GCP_SERVICE_ACCOUNT }}" \
|
||||
--scopes=https://www.googleapis.com/auth/cloud-platform \
|
||||
--metadata=enable-oslogin=TRUE,image-variant='${{ matrix.variant }}' \
|
||||
--metadata-from-file='${{ steps.params.outputs.metadata_file }}' \
|
||||
--labels=purpose=gha-runner-image-builder,variant='${{ matrix.variant }}' \
|
||||
${{ steps.params.outputs.accel }}
|
||||
|
||||
- name: Wait for provisioning to complete
|
||||
env:
|
||||
VM: ${{ steps.params.outputs.vm }}
|
||||
ZONE: ${{ env.BUILD_ZONE }}
|
||||
PROJECT: ${{ env.GCP_PROJECT }}
|
||||
run: |
|
||||
set -uo pipefail
|
||||
# Read the serial console — NOT SSH — to detect completion.
|
||||
#
|
||||
# The provisioner script tees its full output to /dev/ttyS0 (the GCE
|
||||
# serial console). `gcloud compute instances get-serial-port-output`
|
||||
# returns the full console buffer with zero SSH/IAP dependency. This
|
||||
# makes the wait robust against the IAP tunnel issues that previously
|
||||
# caused both false-positive readiness and false-negative timeouts.
|
||||
#
|
||||
# The script writes one of two distinctive marker lines:
|
||||
# "### runner-image: READY: ..." — successful completion
|
||||
# "### runner-image-provision FAILED ..." — fatal error (with line+rc)
|
||||
#
|
||||
# We also stream live progress to the GHA log so the operator can
|
||||
# see what's happening without leaving the run page.
|
||||
deadline=$(( $(date +%s) + 75 * 60 ))
|
||||
last_size=0
|
||||
attempt=0
|
||||
while (( $(date +%s) < deadline )); do
|
||||
attempt=$((attempt+1))
|
||||
console=$(gcloud compute instances get-serial-port-output "$VM" \
|
||||
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null || true)
|
||||
cur_size=${#console}
|
||||
# Stream new provisioner lines (filter syslog noise).
|
||||
if (( cur_size > last_size )); then
|
||||
echo "$console" | tail -c +$((last_size+1)) \
|
||||
| grep -E "runner-image|### |Attempt " || true
|
||||
last_size=$cur_size
|
||||
fi
|
||||
# Success marker
|
||||
if echo "$console" | grep -qE 'runner-image: READY:'; then
|
||||
echo
|
||||
echo "### Image ready (attempt $attempt)"
|
||||
echo "$console" | grep -E 'runner-image: READY:' | tail -1
|
||||
exit 0
|
||||
fi
|
||||
# Failure marker
|
||||
if echo "$console" | grep -qE 'runner-image-provision FAILED'; then
|
||||
echo
|
||||
echo "### Provisioner reported FAILURE — failure block + 40 lines of preceding context:" >&2
|
||||
# -B 40: include 40 lines BEFORE the marker (where the actual
|
||||
# error output of the failing command lives — python tracebacks,
|
||||
# apt errors, curl messages, etc).
|
||||
# -A 5: include the variant/line/cmd metadata lines after the marker.
|
||||
echo "$console" | grep -B 40 -A 5 'runner-image-provision FAILED' \
|
||||
| tail -80 >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 30
|
||||
done
|
||||
echo
|
||||
echo "### Timed out waiting for READY marker after 75 minutes" >&2
|
||||
echo "### Last serial console output (tail):" >&2
|
||||
gcloud compute instances get-serial-port-output "$VM" \
|
||||
--zone="$ZONE" --project="$PROJECT" --port=1 2>/dev/null \
|
||||
| grep -E "runner-image|### " | tail -50 >&2 || true
|
||||
exit 1
|
||||
|
||||
- name: Stop build VM
|
||||
env:
|
||||
VM: ${{ steps.params.outputs.vm }}
|
||||
ZONE: ${{ env.BUILD_ZONE }}
|
||||
PROJECT: ${{ env.GCP_PROJECT }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
gcloud compute instances stop "$VM" --zone="$ZONE" --project="$PROJECT" --quiet
|
||||
# Belt-and-suspenders wait for TERMINATED.
|
||||
for _ in $(seq 1 60); do
|
||||
status=$(gcloud compute instances describe "$VM" --zone="$ZONE" --project="$PROJECT" --format='value(status)')
|
||||
[[ "$status" == "TERMINATED" ]] && break
|
||||
sleep 5
|
||||
done
|
||||
|
||||
- name: Create image
|
||||
env:
|
||||
VM: ${{ steps.params.outputs.vm }}
|
||||
ZONE: ${{ env.BUILD_ZONE }}
|
||||
PROJECT: ${{ env.GCP_PROJECT }}
|
||||
IMAGE: ${{ steps.params.outputs.name }}
|
||||
FAMILY: ${{ steps.params.outputs.family }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# The "Create build VM" step doesn't pass --device-name or
|
||||
# --boot-disk-name, so GCE auto-names the boot disk after the
|
||||
# instance. We rely on that here instead of extracting the name
|
||||
# via a format expression — the previous
|
||||
# `disks[0].source.scope(disks).segment(1)` selector silently
|
||||
# returned empty and gcloud rejected the call with
|
||||
# "You cannot specify [--source-disk-zone] unless you are
|
||||
# specifying [--source-disk]".
|
||||
DISK="$VM"
|
||||
echo "Creating image $IMAGE in family $FAMILY from disk $DISK in $ZONE..."
|
||||
gcloud compute images create "$IMAGE" \
|
||||
--project="$PROJECT" \
|
||||
--source-disk="$DISK" \
|
||||
--source-disk-zone="$ZONE" \
|
||||
--family="$FAMILY" \
|
||||
--description="GHA ephemeral runner image (${{ matrix.variant }}) built ${{ github.run_id }}" \
|
||||
--labels=purpose=gha-ephemeral-runner,variant=${{ matrix.variant }},built-by=gha
|
||||
echo "Created $IMAGE in family $FAMILY"
|
||||
|
||||
- name: Delete build VM
|
||||
if: always()
|
||||
env:
|
||||
VM: ${{ steps.params.outputs.vm }}
|
||||
ZONE: ${{ env.BUILD_ZONE }}
|
||||
PROJECT: ${{ env.GCP_PROJECT }}
|
||||
run: |
|
||||
gcloud compute instances delete "$VM" --zone="$ZONE" --project="$PROJECT" --quiet || true
|
||||
|
||||
- name: Prune old images (keep newest ${{ env.KEEP_IMAGES_PER_FAMILY }})
|
||||
if: ${{ github.event.inputs.skip_prune != 'true' }}
|
||||
env:
|
||||
PROJECT: ${{ env.GCP_PROJECT }}
|
||||
FAMILY: ${{ steps.params.outputs.family }}
|
||||
KEEP: ${{ env.KEEP_IMAGES_PER_FAMILY }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# DELETE (not deprecate — deprecated images still bill for storage)
|
||||
# every image in this family beyond the newest $KEEP. The dispatcher
|
||||
# boots from the family URL, which always resolves to the newest image,
|
||||
# so older ones are only rollback insurance (~$0.05/GB-mo; gpu images
|
||||
# are ~21 GB each). Previously this step used an invalid
|
||||
# `-deprecated.state:*` filter: gcloud errored to stderr, stdout was
|
||||
# empty, and it silently reported "Nothing to deprecate" every month
|
||||
# while images piled up (31 runner images, ~$18/mo, by 2026-09).
|
||||
# Match the family EXACTLY client-side: a gcloud `family=gha-runner-gpu`
|
||||
# filter is a word-boundary regex that ALSO matches gha-runner-gpu-windows.
|
||||
# Rank by ACTIVE (non-deprecated) images: the family URL skips deprecated
|
||||
# images, so the newest $KEEP active ones are the real rollback path.
|
||||
# Deprecated images never count toward $KEEP.
|
||||
rows=$(gcloud compute images list \
|
||||
--project="$PROJECT" \
|
||||
--no-standard-images \
|
||||
--show-deprecated \
|
||||
--sort-by='~creationTimestamp' \
|
||||
--format='value(name,family,deprecated.state)' | awk -v f="$FAMILY" '$2==f')
|
||||
active=$(echo "$rows" | awk 'NF==2 {print $1}')
|
||||
if [[ -z "$active" ]]; then
|
||||
echo "::error::No active images found in family $FAMILY (list failed?) — refusing to prune."
|
||||
exit 1
|
||||
fi
|
||||
keep=$(echo "$active" | head -n "$KEEP")
|
||||
to_delete=$(echo "$rows" | awk '{print $1}' | grep -vxF "$keep" || true)
|
||||
if [[ -z "$to_delete" ]]; then
|
||||
echo "Nothing to prune (<= $KEEP images in $FAMILY)."
|
||||
exit 0
|
||||
fi
|
||||
# Never delete an image some existing disk was created from (e.g. a
|
||||
# long-lived VM still on an older image).
|
||||
in_use=$(gcloud compute disks list --project="$PROJECT" \
|
||||
--format='value(sourceImage.basename())' | sort -u)
|
||||
for img in $to_delete; do
|
||||
if grep -qx "$img" <<<"$in_use"; then
|
||||
echo "Keeping $img (source image of an existing disk)"
|
||||
continue
|
||||
fi
|
||||
echo "Deleting $img"
|
||||
gcloud compute images delete "$img" --project="$PROJECT" --quiet
|
||||
done
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,180 +0,0 @@
|
||||
name: DR backup freshness monitor
|
||||
|
||||
# Off-platform monitor: runs in GitHub (not GCP) so it survives a full GCP
|
||||
# compromise. If the nightly backup pipeline dies silently, this catches it.
|
||||
#
|
||||
# Required secrets:
|
||||
# AWS_BACKUP_READONLY_ACCESS_KEY_ID
|
||||
# AWS_BACKUP_READONLY_SECRET_ACCESS_KEY
|
||||
# DR_MONITOR_DISCORD_WEBHOOK
|
||||
#
|
||||
# Setup: see docs/DISASTER-RECOVERY.md § "Backup freshness monitor".
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 14:00 UTC (~9h after the 1am ET backup window).
|
||||
- cron: "0 14 * * *"
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
check-freshness:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_BACKUP_READONLY_ACCESS_KEY_ID }}
|
||||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_BACKUP_READONLY_SECRET_ACCESS_KEY }}
|
||||
AWS_DEFAULT_REGION: us-east-1
|
||||
BUCKET: nomadkaraoke-backup
|
||||
DISCORD_WEBHOOK: ${{ secrets.DR_MONITOR_DISCORD_WEBHOOK }}
|
||||
|
||||
steps:
|
||||
- name: Verify required secrets are set
|
||||
run: |
|
||||
missing=0
|
||||
for var in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY DISCORD_WEBHOOK; do
|
||||
if [ -z "${!var}" ]; then
|
||||
echo "::error::Required secret $var is empty"
|
||||
missing=1
|
||||
fi
|
||||
done
|
||||
[ "$missing" = "0" ] || exit 1
|
||||
|
||||
- name: Install boto3
|
||||
run: pip install --quiet boto3
|
||||
|
||||
- name: Check freshness of each tracked prefix
|
||||
id: check
|
||||
shell: bash
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import os, sys, datetime, boto3
|
||||
|
||||
BUCKET = os.environ["BUCKET"]
|
||||
# (label, prefix, max_age_hours, require_suffix)
|
||||
# require_suffix: only objects with this key suffix count toward
|
||||
# freshness (None = any object).
|
||||
# NOTE: Firestore and BigQuery export to GCS staging nightly but are
|
||||
# only uploaded to S3 *weekly* (Sundays) to cut cross-cloud egress —
|
||||
# see backup_to_aws/main.py (firestore_to_s3_today). This monitor only
|
||||
# sees the S3 (off-site) copy, so their limits must allow a full week
|
||||
# plus slack (192h ≈ 8 days), NOT the 36h nightly cadence. Secrets DO
|
||||
# upload to S3 nightly, so they stay at 36h. GCS job files are NOT
|
||||
# tracked: their off-site copy was stopped 2026-10-01 (egress cost
|
||||
# cut) — they rely on GCS versioning + soft-delete only.
|
||||
# Git repos are also weekly (Sundays) AND incremental since
|
||||
# 2026-09-26 — only repos pushed that week get a fresh bundle, so a
|
||||
# quiet week legitimately leaves every .bundle older than 8 days.
|
||||
# Freshness is judged on git-repos/manifest.json instead (rewritten
|
||||
# every weekly run), and the manifest's contents are checked below so
|
||||
# a run where every repo failed still reads as stale.
|
||||
TARGETS = [
|
||||
("Firestore", "firestore/", 192, None),
|
||||
("Encrypted secrets", "secrets/", 36, None),
|
||||
("BigQuery weekly", "bigquery/daily-refresh/", 192, None),
|
||||
("Git repos", "git-repos/manifest.json", 192, None),
|
||||
]
|
||||
|
||||
s3 = boto3.client("s3")
|
||||
now = datetime.datetime.now(datetime.timezone.utc)
|
||||
stale, ok = [], []
|
||||
|
||||
# Paginated scan — JMESPath --query on `aws s3api list-objects-v2`
|
||||
# operates per-page, which silently breaks "newest across all pages"
|
||||
# queries on prefixes with >1000 objects (e.g. firestore/ has ~22k).
|
||||
paginator = s3.get_paginator("list_objects_v2")
|
||||
for label, prefix, max_hours, require_suffix in TARGETS:
|
||||
latest = None
|
||||
for page in paginator.paginate(Bucket=BUCKET, Prefix=prefix):
|
||||
for obj in page.get("Contents", []):
|
||||
if require_suffix and not obj["Key"].endswith(require_suffix):
|
||||
continue
|
||||
if latest is None or obj["LastModified"] > latest:
|
||||
latest = obj["LastModified"]
|
||||
|
||||
if latest is None:
|
||||
stale.append(f"{label}: NO OBJECTS in s3://{BUCKET}/{prefix}")
|
||||
continue
|
||||
age = (now - latest).total_seconds()
|
||||
age_h = int(age // 3600)
|
||||
line = f"{label}: {age_h}h old (limit {max_hours}h) — last {latest.isoformat()}"
|
||||
(stale if age > max_hours * 3600 else ok).append(line)
|
||||
|
||||
# Git manifest contents: ANY bundle error alerts, even though the
|
||||
# manifest itself is fresh. The git step is weekly + incremental, so
|
||||
# a repo that failed keeps an ever-older S3 bundle until the next
|
||||
# Sunday — silence here would let it drift past the 7-day RPO.
|
||||
try:
|
||||
import json as _json
|
||||
m = _json.loads(s3.get_object(Bucket=BUCKET, Key="git-repos/manifest.json")["Body"].read())
|
||||
if int(m.get("errors", 0)) > 0:
|
||||
failed = [r.get("full_name") for r in m.get("repos", []) if r.get("status") == "error"]
|
||||
stale.append(f"Git repos: last run had {m.get('errors')} bundle errors: {', '.join(failed[:10])}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
stale.append(f"Git repos: could not read manifest.json ({e})")
|
||||
|
||||
summary = ["## DR backup freshness", ""]
|
||||
if stale:
|
||||
summary.append("### Stale")
|
||||
summary.extend(f"- {s}" for s in stale)
|
||||
summary.append("")
|
||||
summary.append("### Fresh")
|
||||
summary.extend(f"- {o}" for o in ok)
|
||||
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
|
||||
f.write("\n".join(summary) + "\n")
|
||||
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a") as f:
|
||||
f.write(f"ok_count={len(ok)}\n")
|
||||
f.write(f"stale_count={len(stale)}\n")
|
||||
if stale:
|
||||
f.write("STALE_LIST<<EOF\n")
|
||||
f.write("\n".join(stale) + "\n")
|
||||
f.write("EOF\n")
|
||||
|
||||
for o in ok:
|
||||
print("OK:", o)
|
||||
if stale:
|
||||
for s in stale:
|
||||
print("STALE:", s, file=sys.stderr)
|
||||
sys.exit(1)
|
||||
PY
|
||||
|
||||
- name: Alert Discord on stale backup
|
||||
if: failure() && steps.check.outputs.stale_count != ''
|
||||
env:
|
||||
STALE: ${{ steps.check.outputs.STALE_LIST }}
|
||||
run: |
|
||||
payload=$(jq -n \
|
||||
--arg title "🚨 DR backup is stale" \
|
||||
--arg desc "$STALE" \
|
||||
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
|
||||
'{
|
||||
embeds: [{
|
||||
title: $title,
|
||||
description: $desc,
|
||||
color: 15158332,
|
||||
fields: [{name: "Workflow run", value: $url}]
|
||||
}]
|
||||
}')
|
||||
curl -fsS -X POST -H "Content-Type: application/json" \
|
||||
-d "$payload" "$DISCORD_WEBHOOK"
|
||||
|
||||
- name: Alert Discord on workflow failure (catch-all)
|
||||
if: failure() && steps.check.outputs.stale_count == ''
|
||||
run: |
|
||||
payload=$(jq -n \
|
||||
--arg title "⚠️ DR freshness monitor failed to run" \
|
||||
--arg desc "Could not list S3 backup bucket — credentials, network, or bucket policy may be broken." \
|
||||
--arg url "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
|
||||
'{
|
||||
embeds: [{
|
||||
title: $title,
|
||||
description: $desc,
|
||||
color: 15844367,
|
||||
fields: [{name: "Workflow run", value: $url}]
|
||||
}]
|
||||
}')
|
||||
curl -fsS -X POST -H "Content-Type: application/json" \
|
||||
-d "$payload" "$DISCORD_WEBHOOK"
|
||||
@@ -1,646 +0,0 @@
|
||||
name: E2E Daily Test (Payment + Happy Path)
|
||||
|
||||
# Full production E2E: buy credits via real Stripe, then use them for karaoke generation.
|
||||
# Replaces the previous e2e-happy-path.yml with an additional payment stage.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
skip_payment:
|
||||
description: 'Skip Stage 1 (payment) — use admin-granted credits for happy path only'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
use_test_token:
|
||||
description: 'Skip signup entirely — use E2E_TEST_TOKEN (for quick iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
self_heal_force_llm:
|
||||
description: 'Self-healing test: comma-separated Stripe step ids to force through the LLM fallback (e.g. selectCard). Learned recipes still open an auto-merge PR.'
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# =========================================================================
|
||||
# Stage 1: Purchase credits via real Stripe Checkout
|
||||
# =========================================================================
|
||||
e2e-credit-purchase:
|
||||
name: "Stage 1: Credit Purchase"
|
||||
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
|
||||
timeout-minutes: 15
|
||||
if: ${{ github.event.inputs.skip_payment != 'true' && github.event.inputs.use_test_token != 'true' }}
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # keyless GCP auth, only to read the Gemini key for self-healing
|
||||
|
||||
outputs:
|
||||
session_token: ${{ steps.extract_token.outputs.token }}
|
||||
result: ${{ steps.test_result.outputs.result }}
|
||||
failure_reason: ${{ steps.test_result.outputs.failure_reason }}
|
||||
healed: ${{ steps.self_heal.outputs.healed }}
|
||||
env:
|
||||
HAS_GEMINI_SECRET: ${{ secrets.GEMINI_API_KEY != '' }}
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
# ── Self-healing Stripe Checkout: LLM fallback key ───────────────────
|
||||
# The Stripe step runner (frontend/e2e/helpers/self-healing) falls back
|
||||
# to Gemini when every deterministic variant of a step fails. The key is
|
||||
# the AI Studio key (billed outside the nomadkaraoke project). Prefer a
|
||||
# GEMINI_API_KEY repo secret; otherwise read it from Secret Manager via
|
||||
# keyless auth. All of this is best-effort: without a key the test runs
|
||||
# deterministic-only, exactly as before.
|
||||
- name: Authenticate to Google Cloud (Gemini key)
|
||||
id: gcp_auth
|
||||
if: ${{ env.HAS_GEMINI_SECRET != 'true' }}
|
||||
continue-on-error: true
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
if: steps.gcp_auth.outcome == 'success'
|
||||
continue-on-error: true
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
|
||||
- name: Load Gemini key for self-healing fallback
|
||||
continue-on-error: true
|
||||
env:
|
||||
GEMINI_SECRET: ${{ secrets.GEMINI_API_KEY }}
|
||||
GCP_AUTHED: ${{ steps.gcp_auth.outcome == 'success' }}
|
||||
run: |
|
||||
KEY="$GEMINI_SECRET"
|
||||
if [ -z "$KEY" ] && [ "$GCP_AUTHED" = "true" ]; then
|
||||
KEY=$(gcloud secrets versions access latest --secret=gemini-api-key --project=nomadkaraoke 2>/dev/null || true)
|
||||
fi
|
||||
# Drop the deployer credentials before any test code runs.
|
||||
if [ -n "${GOOGLE_GHA_CREDS_PATH:-}" ]; then rm -f "$GOOGLE_GHA_CREDS_PATH"; fi
|
||||
if [ -n "${CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE:-}" ]; then rm -f "$CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE"; fi
|
||||
if [ -n "$KEY" ]; then
|
||||
echo "::add-mask::$KEY"
|
||||
echo "GEMINI_API_KEY=$KEY" >> "$GITHUB_ENV"
|
||||
echo "Self-healing LLM fallback: enabled"
|
||||
else
|
||||
echo "::warning::Self-healing LLM fallback disabled (no Gemini key) — Stripe steps run deterministic-only"
|
||||
fi
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Run Credit Purchase Test
|
||||
id: run_test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
|
||||
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
|
||||
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
|
||||
E2E_STRIPE_CARDHOLDER_NAME: ${{ secrets.E2E_STRIPE_CARDHOLDER_NAME }}
|
||||
E2E_STRIPE_ZIP: ${{ secrets.E2E_STRIPE_ZIP }}
|
||||
E2E_SELF_HEAL_FORCE_LLM: ${{ inputs.self_heal_force_llm || '' }}
|
||||
# No real card available → 'checkout-only' (no charge). Set the repo variable
|
||||
# E2E_REAL_PAYMENT=true to go back to buying a credit with E2E_STRIPE_CARD_*.
|
||||
E2E_PAYMENT_MODE: ${{ vars.E2E_REAL_PAYMENT == 'true' && 'real' || 'checkout-only' }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 1: Credit Purchase via Real Stripe (mode: $E2E_PAYMENT_MODE) ==="
|
||||
echo "Time: $(date)"
|
||||
npx playwright test credit-purchase-real.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=600000 \
|
||||
2>&1 | tee test-output-payment.log
|
||||
# Capture Playwright's real exit code, not tee's (0) — the
|
||||
# authoritative pass/fail signal. See Stage 2 for rationale.
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
run: |
|
||||
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
|
||||
echo "result=success" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "result=failure" >> "$GITHUB_OUTPUT"
|
||||
# A declined card is an issuer/card problem, not a broken checkout —
|
||||
# say so in the alerts (marker comes from e2e/helpers/stripe-checkout.ts).
|
||||
REASON=$(grep -m1 -o 'STRIPE_CARD_DECLINED:.*' frontend/test-output-payment.log 2>/dev/null | sed 's/ — .*//' | tr -d '"\\' || true)
|
||||
if [ -n "$REASON" ]; then
|
||||
echo "failure_reason=Test card declined, not a checkout bug (${REASON#STRIPE_CARD_DECLINED: }). Check the card/issuer for a fraud hold, then re-run." >> "$GITHUB_OUTPUT"
|
||||
echo "::error::$REASON"
|
||||
fi
|
||||
fi
|
||||
|
||||
- name: Detect self-healed Stripe steps
|
||||
id: self_heal
|
||||
if: always()
|
||||
working-directory: frontend
|
||||
run: |
|
||||
REPORT=test-results/self-heal/self-heal-report.json
|
||||
LEARNED=test-results/self-heal/learned-variants.json
|
||||
if [ -f "$REPORT" ]; then
|
||||
{
|
||||
echo "### Stripe Checkout self-healing"
|
||||
echo '```json'
|
||||
jq '{flowVerified, llmCalls, learnedWritten, added, outcomes: [.outcomes[] | {stepId, status, via, variant, model, reason}]}' "$REPORT"
|
||||
echo '```'
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
# Only promote recipes from a run whose purchase was verified server-side.
|
||||
if [ "${{ steps.test_result.outputs.result }}" = "success" ] && [ -f "$LEARNED" ]; then
|
||||
echo "healed=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::warning::Stripe Checkout changed — LLM fallback healed it; an auto-merge PR will record the new variant"
|
||||
else
|
||||
echo "healed=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Upload self-heal learned variants
|
||||
if: steps.self_heal.outputs.healed == 'true'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: self-heal-learned-${{ github.run_number }}
|
||||
path: frontend/test-results/self-heal/
|
||||
retention-days: 30
|
||||
|
||||
- name: Extract session token
|
||||
id: extract_token
|
||||
if: steps.test_result.outputs.result == 'success'
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if [ -f test-results/e2e-session-token.txt ]; then
|
||||
TOKEN=$(cat test-results/e2e-session-token.txt)
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
echo "Session token extracted for Stage 2"
|
||||
else
|
||||
echo "No session token file found"
|
||||
echo "token=" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Scrub secrets from artifacts (public repo)
|
||||
id: scrub
|
||||
if: always()
|
||||
working-directory: frontend
|
||||
env:
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
E2E_STRIPE_CARD_NUMBER: ${{ secrets.E2E_STRIPE_CARD_NUMBER }}
|
||||
E2E_STRIPE_CARD_EXPIRY: ${{ secrets.E2E_STRIPE_CARD_EXPIRY }}
|
||||
E2E_STRIPE_CARD_CVC: ${{ secrets.E2E_STRIPE_CARD_CVC }}
|
||||
E2E_SESSION_TOKEN: ${{ steps.extract_token.outputs.token }}
|
||||
run: node e2e/scripts/scrub-artifacts.mjs --drop-videos --blank-inputs test-results test-output-payment.log
|
||||
|
||||
- name: Upload artifacts
|
||||
if: always() && steps.scrub.outcome == 'success'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-payment-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/test-output-payment.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
env:
|
||||
# Via env + jq, never ${{ }} in the script: the reason contains Stripe's message text.
|
||||
REASON: ${{ steps.test_result.outputs.failure_reason || 'Payment flow is broken. Customers may not be receiving credits.' }}
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
RUN_NUMBER: ${{ github.run_number }}
|
||||
WEBHOOK: ${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}
|
||||
run: |
|
||||
jq -n --arg reason "$REASON" --arg run "$RUN_NUMBER" --arg url "$RUN_URL" \
|
||||
'{embeds: [{title: "🚨 E2E Daily Test FAILED — Stage 1: Credit Purchase", description: "\($reason)\n\n**Run:** #\($run)\n**Details:** \($url)", color: 15158332}]}' |
|
||||
curl -s -H "Content-Type: application/json" -d @- "$WEBHOOK"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Self-heal: record LLM-discovered Stripe variants via an auto-merging PR
|
||||
# =========================================================================
|
||||
# One Stripe layout change → one LLM-healed run → one data-only PR (only
|
||||
# learned-variants.json) that auto-merges, so the next run replays the
|
||||
# recipe deterministically. No human in the loop.
|
||||
#
|
||||
# Merge gates for a bot PR (all verified live with #1117, 2026-10-03):
|
||||
# - its `pull_request` CI run is created "action_required" and, until run,
|
||||
# supersedes other CI Gate results → we approve it via the API;
|
||||
# - the `protect main` ruleset's require_extra_approval_for_unattributed_changes
|
||||
# (GitHub default true since ~Aug 2026) would demand a human review for
|
||||
# bot-authored changes → set to false on ruleset 11423976;
|
||||
# - review threads must be resolved → `@coderabbitai ignore` in the body.
|
||||
self-heal-pr:
|
||||
name: "Self-heal: auto-merge PR"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
needs: e2e-credit-purchase
|
||||
# Only learn from main — branch runs (e.g. testing self_heal_force_llm on a
|
||||
# feature branch) must not open PRs against main.
|
||||
if: always() && needs.e2e-credit-purchase.outputs.healed == 'true' && github.ref == 'refs/heads/main'
|
||||
permissions:
|
||||
contents: write
|
||||
pull-requests: write
|
||||
actions: write
|
||||
outputs:
|
||||
pr_url: ${{ steps.pr.outputs.pr_url }}
|
||||
unchanged: ${{ steps.pr.outputs.unchanged }}
|
||||
steps:
|
||||
- name: Checkout main
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: main
|
||||
|
||||
- name: Download learned variants
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: self-heal-learned-${{ github.run_number }}
|
||||
path: /tmp/self-heal
|
||||
|
||||
- name: Open auto-merge PR
|
||||
id: pr
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
TARGET=frontend/e2e/helpers/self-healing/learned-variants.json
|
||||
|
||||
# A bot-authored PR's `pull_request` CI run is created as
|
||||
# "action_required" (awaiting approval) and, until it runs, supersedes
|
||||
# any other CI Gate result for the ruleset. Approve it; succeed once a
|
||||
# pull_request run exists that is past approval.
|
||||
approve_ci() {
|
||||
local branch=$1 runs pending active
|
||||
for _ in $(seq 1 18); do
|
||||
runs=$(gh run list --workflow ci.yml --branch "$branch" --event pull_request --json databaseId,status,conclusion) || { sleep 10; continue; }
|
||||
pending=$(jq -r '.[] | select(.conclusion == "action_required" or .status == "waiting" or .status == "action_required") | .databaseId' <<<"$runs") || pending=""
|
||||
active=$(jq '[.[] | select(.conclusion != "action_required" and .status != "waiting" and .status != "action_required")] | length' <<<"$runs") || active=0
|
||||
for id in $pending; do
|
||||
if gh api -X POST "repos/${{ github.repository }}/actions/runs/$id/approve" >/dev/null; then
|
||||
echo "Approved CI run $id"
|
||||
else
|
||||
echo "::warning::Could not approve CI run $id with GITHUB_TOKEN"
|
||||
fi
|
||||
done
|
||||
if [ -z "$pending" ] && [ "$active" -gt 0 ]; then
|
||||
echo "CI is running for $branch"
|
||||
return 0
|
||||
fi
|
||||
sleep 10
|
||||
done
|
||||
return 1
|
||||
}
|
||||
REPORT=/tmp/self-heal/self-heal-report.json
|
||||
|
||||
OPEN=$(gh pr list --label e2e-self-heal --state open --json url --jq '.[0].url // ""')
|
||||
if [ -n "$OPEN" ]; then
|
||||
echo "A self-heal PR is already open ($OPEN) — not opening another; making sure its CI can run"
|
||||
echo "pr_url=$OPEN" >> "$GITHUB_OUTPUT"
|
||||
OPEN_BRANCH=$(gh pr view "$OPEN" --json headRefName --jq .headRefName || true)
|
||||
if [ -z "$OPEN_BRANCH" ] || ! approve_ci "$OPEN_BRANCH"; then
|
||||
echo "ci_needs_approval=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
cp /tmp/self-heal/learned-variants.json "$TARGET"
|
||||
if git diff --quiet -- "$TARGET"; then
|
||||
echo "learned-variants.json unchanged vs main — nothing to do"
|
||||
echo "unchanged=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
STEPS=$(jq -r '[.outcomes[] | select(.status == "healed") | .stepId] | unique | join(", ")' "$REPORT")
|
||||
BRANCH="self-heal/stripe-checkout-${{ github.run_id }}"
|
||||
git config user.name "github-actions[bot]"
|
||||
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
|
||||
git checkout -b "$BRANCH"
|
||||
git add "$TARGET"
|
||||
git commit -m "test(e2e): self-heal Stripe Checkout steps: $STEPS" -m "Learned by the LLM fallback in $RUN_URL (purchase verified server-side)."
|
||||
git push origin "$BRANCH"
|
||||
|
||||
gh label create e2e-self-heal --color FBCA04 --description "Auto-generated by the self-healing Stripe E2E" 2>/dev/null || true
|
||||
{
|
||||
echo "## 🩹 Stripe Checkout self-heal"
|
||||
echo
|
||||
echo "Stripe changed its hosted checkout page. In [run #${{ github.run_number }}]($RUN_URL) every deterministic variant of **$STEPS** failed; the Gemini fallback completed the step(s), and the purchase was then **verified server-side** (credit granted)."
|
||||
echo
|
||||
echo "This PR only adds the recipe(s) to \`learned-variants.json\`, so future runs replay them deterministically (no LLM). It auto-merges once CI passes."
|
||||
echo
|
||||
echo "### Learned recipes"
|
||||
echo '```json'
|
||||
jq '[.outcomes[] | select(.status == "healed") | {stepId, model, llmCalls, actions}]' "$REPORT"
|
||||
echo '```'
|
||||
echo
|
||||
echo "@coderabbitai ignore"
|
||||
echo
|
||||
echo "🤖 Generated by the E2E Daily self-healing workflow"
|
||||
} > /tmp/pr-body.md
|
||||
|
||||
PR_URL=$(gh pr create --base main --head "$BRANCH" --label e2e-self-heal \
|
||||
--title "test(e2e): self-heal Stripe Checkout ($STEPS)" --body-file /tmp/pr-body.md)
|
||||
echo "pr_url=$PR_URL" >> "$GITHUB_OUTPUT"
|
||||
gh pr merge "$PR_URL" --auto --squash
|
||||
|
||||
if approve_ci "$BRANCH"; then
|
||||
echo "Opened $PR_URL (auto-merge enabled, CI approved)"
|
||||
else
|
||||
# Fallback: dispatched CI (allowed from GITHUB_TOKEN) puts a CI Gate
|
||||
# check on the head SHA if no pull_request run ever appears.
|
||||
gh workflow run ci.yml --ref "$BRANCH"
|
||||
echo "ci_needs_approval=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::warning::CI for $PR_URL may need a one-click approval (PR → Checks → Approve and run)"
|
||||
fi
|
||||
|
||||
- name: Discord notification (self-heal)
|
||||
# Recipe already on main (e.g. merged between checkout and now) → nothing to report.
|
||||
if: always() && steps.pr.outputs.unchanged != 'true'
|
||||
run: |
|
||||
PR="${{ steps.pr.outputs.pr_url }}"
|
||||
# #alerts = needs a human; #noise = informational (auto-merge handles it).
|
||||
# Noise falls back to the alert webhook if its secret is unset.
|
||||
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
if [ -n "$PR" ] && [ "${{ steps.pr.outputs.ci_needs_approval }}" = "true" ]; then
|
||||
TITLE="🩹 E2E self-healed Stripe — PR needs a CI approval click"
|
||||
DESC="LLM fallback healed a Stripe Checkout change. The auto-merge PR's CI run awaits approval (open the PR → Checks → Approve and run): $PR"
|
||||
COLOR=16776960
|
||||
WEBHOOK="$ALERT_WEBHOOK"
|
||||
elif [ -n "$PR" ]; then
|
||||
TITLE="🩹 E2E self-healed a Stripe Checkout change"
|
||||
DESC="Stripe changed its checkout page; the LLM fallback got through and the purchase was verified. Auto-merge PR: $PR"
|
||||
COLOR=16776960
|
||||
WEBHOOK="$NOISE_WEBHOOK"
|
||||
else
|
||||
TITLE="⚠️ E2E self-heal PR could not be opened"
|
||||
DESC="The LLM fallback healed a Stripe step but opening the PR failed — see the run."
|
||||
COLOR=15105570
|
||||
WEBHOOK="$ALERT_WEBHOOK"
|
||||
fi
|
||||
PAYLOAD="{\"embeds\": [{\"title\": \"$TITLE\", \"description\": \"$DESC\n\n**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": $COLOR}]}"
|
||||
# A failed #noise post retries on #alerts so it is never silently lost.
|
||||
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$WEBHOOK" \
|
||||
|| { [ "$WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|
||||
|| true
|
||||
|
||||
# =========================================================================
|
||||
# Stage 2: Happy Path (uses credit from Stage 1)
|
||||
# =========================================================================
|
||||
e2e-happy-path:
|
||||
name: "Stage 2: Happy Path"
|
||||
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
|
||||
timeout-minutes: 90
|
||||
needs: e2e-credit-purchase
|
||||
if: |
|
||||
always() && (
|
||||
needs.e2e-credit-purchase.result == 'success' ||
|
||||
github.event.inputs.skip_payment == 'true' ||
|
||||
github.event.inputs.use_test_token == 'true'
|
||||
)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps ${{ inputs.browser || 'chromium' }}
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Refresh E2E session token (when use_test_token enabled)
|
||||
id: refresh_token
|
||||
if: github.event.inputs.use_test_token == 'true'
|
||||
run: |
|
||||
RESPONSE=$(curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/users/e2e-test-runner@nomadkaraoke.com/impersonate" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json")
|
||||
TOKEN=$(echo "$RESPONSE" | python3 -c 'import json,sys; print(json.load(sys.stdin)["session_token"])')
|
||||
echo "::add-mask::$TOKEN"
|
||||
echo "token=$TOKEN" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Run Happy Path Test
|
||||
id: run_test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "=== Stage 2: Happy Path ==="
|
||||
echo "Time: $(date)"
|
||||
TOKEN_SOURCE="none"
|
||||
if [ -n "${{ steps.refresh_token.outputs.token }}" ]; then
|
||||
TOKEN_SOURCE="use_test_token (impersonation)"
|
||||
elif [ -n "${{ needs.e2e-credit-purchase.outputs.session_token }}" ]; then
|
||||
TOKEN_SOURCE="Stage 1 (credit purchase)"
|
||||
fi
|
||||
echo "Token source: $TOKEN_SOURCE"
|
||||
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output-happy-path.log
|
||||
# Capture Playwright's real exit code, not tee's (0). This is the
|
||||
# authoritative pass/fail signal — grepping the log for "failed" is
|
||||
# fragile because benign retry warnings (e.g. "goto attempt 1/3
|
||||
# failed") contain that substring and produce false alarms.
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
run: |
|
||||
if [ "${{ steps.run_test.outputs.exit_code }}" = "0" ]; then
|
||||
echo "result=success" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "result=failure" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Scrub secrets from artifacts (public repo)
|
||||
id: scrub
|
||||
if: always()
|
||||
working-directory: frontend
|
||||
env:
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
E2E_TEST_TOKEN: ${{ steps.refresh_token.outputs.token || needs.e2e-credit-purchase.outputs.session_token || '' }}
|
||||
run: node e2e/scripts/scrub-artifacts.mjs test-results playwright-report test-output-happy-path.log
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always() && steps.scrub.outcome == 'success'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output-happy-path.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always() && steps.scrub.outcome == 'success'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
STAGE2="${{ steps.test_result.outputs.result || 'unknown' }}"
|
||||
echo "## E2E Daily Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 1: Credit Purchase | ${STAGE1} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Stage 2: Happy Path | ${STAGE2} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: |
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"🚨 E2E Daily Test FAILED — Stage 2: Happy Path\", \"description\": \"Karaoke generation flow is broken.\n\n**Run:** #${{ github.run_number }}\n**Details:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\", \"color\": 15158332}]}" \
|
||||
"${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
|
||||
# =========================================================================
|
||||
# Final: Send notifications
|
||||
# =========================================================================
|
||||
notify:
|
||||
name: "Notify"
|
||||
runs-on: ubuntu-latest
|
||||
needs: [e2e-credit-purchase, e2e-happy-path, self-heal-pr]
|
||||
if: always()
|
||||
|
||||
steps:
|
||||
- name: Discord notification (success)
|
||||
# Passes are not signal: they go to #noise (fallback: #alerts if the
|
||||
# noise secret is unset, so nothing is ever silently dropped).
|
||||
if: needs.e2e-happy-path.result == 'success'
|
||||
run: |
|
||||
STAGE1="${{ needs.e2e-credit-purchase.result || 'skipped' }}"
|
||||
ALERT_WEBHOOK="${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
NOISE_WEBHOOK="${{ secrets.DISCORD_NOISE_WEBHOOK_URL || secrets.DISCORD_ALERT_WEBHOOK_URL }}"
|
||||
PAYLOAD="{\"embeds\": [{\"title\": \"✅ E2E Daily Test PASSED\", \"description\": \"Payment: ${STAGE1} | Happy Path: success\n\n**Run:** #${{ github.run_number }}\", \"color\": 3066993}]}"
|
||||
curl -fsS -H "Content-Type: application/json" -d "$PAYLOAD" "$NOISE_WEBHOOK" \
|
||||
|| { [ "$NOISE_WEBHOOK" != "$ALERT_WEBHOOK" ] && curl -s -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_WEBHOOK"; } \
|
||||
|| true
|
||||
|
||||
- name: Send email notification
|
||||
if: github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.postmarkapp.com
|
||||
server_port: 587
|
||||
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
|
||||
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
|
||||
subject: "E2E Daily Test ${{ needs.e2e-happy-path.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Daily Test Results
|
||||
======================
|
||||
|
||||
Stage 1 (Credit Purchase): ${{ needs.e2e-credit-purchase.result || 'skipped' }}${{ vars.E2E_REAL_PAYMENT != 'true' && ' [checkout-only: no real payment]' || '' }}${{ needs.e2e-credit-purchase.outputs.failure_reason && format(' — {0}', needs.e2e-credit-purchase.outputs.failure_reason) || '' }}
|
||||
Stage 2 (Happy Path): ${{ needs.e2e-happy-path.result || 'skipped' }}
|
||||
Stripe self-heal: ${{ needs.e2e-credit-purchase.outputs.healed == 'true' && format('Stripe changed its checkout; LLM fallback healed it. Auto-merge PR: {0}', needs.self-heal-pr.outputs.pr_url || (needs.self-heal-pr.outputs.unchanged == 'true' && '(already on main)' || '(failed to open — see run)')) || 'not needed' }}
|
||||
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
@@ -1,141 +0,0 @@
|
||||
name: E2E Daily Test (Tenant Portals)
|
||||
|
||||
# Full production E2E per white-label tenant: submit a job with the tenant's own
|
||||
# mixed + instrumental audio, approve the lyrics review, wait for the render, and assert
|
||||
# the tenant distribution (locked theme, no YouTube, no Google Drive, Dropbox folder +
|
||||
# downloadable outputs). Mirrors the consumer e2e-daily.yml. Each run cleans up its job.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 00:07 UTC = ~8pm US Eastern (8:07pm EDT / 7:07pm EST). Cron is fixed
|
||||
# UTC so the local time shifts 1h across DST. Off the :00 mark to avoid the
|
||||
# top-of-hour scheduler herd.
|
||||
- cron: '7 0 * * *'
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # for Workload Identity Federation
|
||||
|
||||
jobs:
|
||||
e2e-tenant:
|
||||
name: "Tenant E2E: ${{ matrix.tenant }}"
|
||||
runs-on: ubuntu-24.04 # Playwright 1.57 can't install chromium on ubuntu-latest (26.04)
|
||||
timeout-minutes: 60
|
||||
strategy:
|
||||
fail-fast: false
|
||||
# Run tenants sequentially: the GCE encoder is a single shared worker, and two
|
||||
# concurrent full renders overwhelm it ("lost contact with worker"). One at a time.
|
||||
max-parallel: 1
|
||||
matrix:
|
||||
tenant: [vocalstar, singa]
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Authenticate to GCP
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Fetch test audio (private GCS)
|
||||
run: bash scripts/e2e/fetch_test_audio.sh frontend/e2e/fixtures
|
||||
|
||||
- name: Ensure testmail.app domain is allowlisted (rate limit)
|
||||
run: |
|
||||
curl -sf -X POST \
|
||||
"https://api.nomadkaraoke.com/api/admin/rate-limits/blocklists/allowlisted-domains" \
|
||||
-H "X-Admin-Token: ${{ secrets.E2E_ADMIN_TOKEN }}" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"domain": "inbox.testmail.app"}' || echo "Warning: allowlist may already exist"
|
||||
|
||||
- name: Run tenant E2E (frontend journey)
|
||||
id: run_test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
TENANT_ID: ${{ matrix.tenant }}
|
||||
TENANT_PORTAL_URL: https://${{ matrix.tenant }}.nomadkaraoke.com
|
||||
run: |
|
||||
echo "=== Tenant E2E (frontend): ${{ matrix.tenant }} ==="
|
||||
npx playwright test tenant-happy-path.spec.ts \
|
||||
--config=playwright.production.config.ts --project=chromium \
|
||||
--reporter=list --timeout=2400000 2>&1 | tee e2e-output.log
|
||||
echo "exit_code=${PIPESTATUS[0]}" >> "$GITHUB_OUTPUT"
|
||||
continue-on-error: true
|
||||
|
||||
- name: Scrub secrets from artifacts (public repo)
|
||||
id: scrub
|
||||
if: always()
|
||||
working-directory: frontend
|
||||
env:
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
run: node e2e/scripts/scrub-artifacts.mjs test-results playwright-report e2e-output.log
|
||||
|
||||
- name: Upload logs
|
||||
if: always() && steps.scrub.outcome == 'success'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-tenant-${{ matrix.tenant }}-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/e2e-output.log
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
retention-days: 14
|
||||
|
||||
- name: Discord notification (failure)
|
||||
if: ${{ steps.run_test.outputs.exit_code != '0' }}
|
||||
env:
|
||||
DISCORD_WEBHOOK: ${{ secrets.DISCORD_ALERT_WEBHOOK_URL }}
|
||||
run: |
|
||||
if [ -n "$DISCORD_WEBHOOK" ]; then
|
||||
curl -s -H "Content-Type: application/json" \
|
||||
-d "{\"embeds\": [{\"title\": \"❌ Tenant E2E FAILED — ${{ matrix.tenant }}\", \"description\": \"Run: #${{ github.run_number }}\", \"color\": 15158332}]}" \
|
||||
"$DISCORD_WEBHOOK" || true
|
||||
fi
|
||||
|
||||
- name: Fail the job if the test failed
|
||||
if: ${{ steps.run_test.outputs.exit_code != '0' }}
|
||||
run: exit 1
|
||||
|
||||
notify-email:
|
||||
name: "Email summary"
|
||||
runs-on: ubuntu-latest
|
||||
needs: e2e-tenant
|
||||
if: ${{ always() && github.event_name == 'schedule' }}
|
||||
steps:
|
||||
- name: Send email notification
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.postmarkapp.com
|
||||
server_port: 587
|
||||
username: ${{ secrets.POSTMARK_SERVER_TOKEN }}
|
||||
password: ${{ secrets.POSTMARK_SERVER_TOKEN }}
|
||||
subject: "Tenant E2E Daily ${{ needs.e2e-tenant.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
Tenant E2E Daily Test (Vocal Star + Singa)
|
||||
Result: ${{ needs.e2e-tenant.result }}
|
||||
Run: #${{ github.run_number }}
|
||||
Details: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
@@ -0,0 +1,51 @@
|
||||
name: Auto-publish
|
||||
|
||||
on: [push, workflow_dispatch]
|
||||
|
||||
jobs:
|
||||
# Auto-publish when version is increased
|
||||
publish-pypi:
|
||||
# Only publish on `main` branch
|
||||
if: github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
permissions: # Don't forget permissions
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: etils-actions/pypi-auto-publish@v1
|
||||
with:
|
||||
pypi-token: ${{ secrets.PYPI_API_TOKEN }}
|
||||
gh-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
parse-changelog: false
|
||||
|
||||
# build-windows:
|
||||
# runs-on: windows-latest
|
||||
|
||||
# steps:
|
||||
# - uses: actions/checkout@v2
|
||||
|
||||
# - name: Set up Python
|
||||
# uses: actions/setup-python@v2
|
||||
# with:
|
||||
# python-version: '3.9'
|
||||
|
||||
# - name: Install dependencies with Poetry
|
||||
# run: |
|
||||
# pip install poetry
|
||||
# poetry install
|
||||
|
||||
# - name: Download ffmpeg binary
|
||||
# run: python platform/download_ffmpeg.py
|
||||
|
||||
# - name: Build EXE with PyInstaller
|
||||
# run: poetry run pyinstaller karaoke-gen.spec
|
||||
|
||||
# - name: Upload EXE as Artifact
|
||||
# uses: actions/upload-artifact@v2
|
||||
# with:
|
||||
# name: karaoke-gen-windows
|
||||
# path: |
|
||||
# build/
|
||||
# *.exe
|
||||
# */*.exe
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
name: PyPI prune reminder
|
||||
|
||||
# PyPI caps total *project* size at 10 GB. karaoke-gen publishes a ~62 MiB wheel
|
||||
# whenever CLI code changes (scripts/pypi_publish_gate.py), so the project slowly
|
||||
# refills the cap and `poetry publish` then fails with "Project size too large".
|
||||
#
|
||||
# PyPI has NO deletion API (upload tokens are upload-only), so this can't
|
||||
# auto-delete. Instead it runs monthly, computes the retention plan
|
||||
# (scripts/prune_pypi_releases.py), and opens/updates a single tracking issue
|
||||
# containing a paste-into-browser-console snippet you run on your logged-in PyPI
|
||||
# releases page to actually prune. No secrets are stored.
|
||||
#
|
||||
# See docs/PYPI-STORAGE-PRUNE.md.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# 09:00 UTC on the 1st of each month.
|
||||
- cron: "0 9 1 * *"
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
|
||||
jobs:
|
||||
prune-reminder:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
PROJECT: karaoke-gen
|
||||
KEEP_DAYS: "30"
|
||||
# Only nag once the project crosses this watermark — pruning holds it at
|
||||
# ~4-7 GB, and there are always old releases to consolidate, so without a
|
||||
# threshold this would open an issue every month for no real reason. 7 GB
|
||||
# leaves ~1 month of ~3 GB/month growth before the 10 GB cap.
|
||||
NAG_THRESHOLD_GB: "7"
|
||||
ISSUE_TITLE: "🧹 PyPI storage prune due: karaoke-gen"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Set up Python 3.13
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.13"
|
||||
|
||||
- name: Compute prune plan
|
||||
id: plan
|
||||
run: |
|
||||
python scripts/prune_pypi_releases.py \
|
||||
--project "$PROJECT" --keep-days "$KEEP_DAYS" \
|
||||
--format markdown --output plan.md
|
||||
python scripts/prune_pypi_releases.py \
|
||||
--project "$PROJECT" --keep-days "$KEEP_DAYS" \
|
||||
--format json --output plan.json
|
||||
python - <<'PY' >> "$GITHUB_OUTPUT"
|
||||
import json, os
|
||||
d = json.load(open("plan.json"))
|
||||
threshold = float(os.environ["NAG_THRESHOLD_GB"])
|
||||
should_nag = d["delete_count"] > 0 and d["current_gb"] >= threshold
|
||||
print(f"delete_count={d['delete_count']}")
|
||||
print(f"reclaimable_gb={d['reclaimable_gb']:.2f}")
|
||||
print(f"current_gb={d['current_gb']:.2f}")
|
||||
print(f"should_nag={'true' if should_nag else 'false'}")
|
||||
PY
|
||||
|
||||
- name: Job summary
|
||||
run: cat plan.md >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Open or update tracking issue
|
||||
if: steps.plan.outputs.should_nag == 'true'
|
||||
run: |
|
||||
# Search by a stable ASCII substring (emoji in search queries is
|
||||
# flaky); the exact-title jq filter below is the real dedupe guard.
|
||||
existing=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --search "PyPI storage prune due in:title" \
|
||||
--json number,title \
|
||||
--jq "map(select(.title == \"$ISSUE_TITLE\")) | .[0].number // empty")
|
||||
note="🔁 Refreshed $(date -u +%Y-%m-%d): ${{ steps.plan.outputs.delete_count }} releases prunable, ~${{ steps.plan.outputs.reclaimable_gb }} GB reclaimable (project at ${{ steps.plan.outputs.current_gb }} GB)."
|
||||
if [ -n "$existing" ]; then
|
||||
echo "Updating existing issue #$existing"
|
||||
gh issue edit "$existing" --repo "$GITHUB_REPOSITORY" --body-file plan.md
|
||||
gh issue comment "$existing" --repo "$GITHUB_REPOSITORY" --body "$note"
|
||||
else
|
||||
echo "Creating new tracking issue"
|
||||
gh issue create --repo "$GITHUB_REPOSITORY" \
|
||||
--title "$ISSUE_TITLE" --body-file plan.md
|
||||
fi
|
||||
@@ -1,10 +0,0 @@
|
||||
name: Security
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
push:
|
||||
branches: [main]
|
||||
|
||||
jobs:
|
||||
gitleaks:
|
||||
uses: nomadkaraoke/.github/.github/workflows/gitleaks.yml@main
|
||||
+3
-72
@@ -3,72 +3,12 @@
|
||||
.vscode
|
||||
*.csv
|
||||
*.mp3
|
||||
ABBA*
|
||||
|
||||
# Andrew functional adds
|
||||
/audio/
|
||||
/lyrics/
|
||||
/karaoke/
|
||||
audio/
|
||||
lyrics/
|
||||
karaoke/
|
||||
tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Local karaoke-gen CLI test output dirs (created during manual validation)
|
||||
/test-output-*/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
.local-stack
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
frontend/.next/
|
||||
frontend/out/
|
||||
frontend/.swc/
|
||||
frontend/.turbo/
|
||||
frontend/build/
|
||||
frontend/dist/
|
||||
# Generated Next.js static export copied into the Python package at build time (#876).
|
||||
# Bundled into the wheel via pyproject `include`; must not be tracked in git.
|
||||
karaoke_gen/nextjs_frontend/out/
|
||||
|
||||
# Next.js specific
|
||||
frontend/*.tsbuildinfo
|
||||
frontend/next-env.d.ts
|
||||
|
||||
# Environment files (local overrides)
|
||||
frontend/.env*.local
|
||||
|
||||
# Vercel
|
||||
frontend/.vercel/
|
||||
|
||||
# Package manager logs
|
||||
frontend/npm-debug.log*
|
||||
frontend/yarn-debug.log*
|
||||
frontend/yarn-error.log*
|
||||
frontend/pnpm-debug.log*
|
||||
frontend/.pnpm-debug.log*
|
||||
|
||||
# Testing
|
||||
frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
frontend/e2e/screenshots/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
|
||||
# ESLint
|
||||
frontend/.eslintcache
|
||||
|
||||
# Storybook
|
||||
frontend/storybook-static/
|
||||
|
||||
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
|
||||
frontend/**/public/workbox-*.js
|
||||
frontend/**/public/swe-worker-*.js
|
||||
|
||||
|
||||
# Byte-compiled / optimized / DLL files
|
||||
@@ -87,12 +27,8 @@ dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib directories
|
||||
!frontend/lib/
|
||||
!buy-site/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
@@ -235,8 +171,3 @@ cython_debug/
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
.superpowers/
|
||||
@@ -1,63 +0,0 @@
|
||||
# Nomad Karaoke shared gitleaks config.
|
||||
# Extends the default ruleset with one custom rule for Discord webhooks
|
||||
# and an allowlist covering known false-positive paths.
|
||||
|
||||
title = "Nomad Karaoke"
|
||||
|
||||
[extend]
|
||||
useDefault = true
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Custom rules
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
[[rules]]
|
||||
id = "discord-webhook-url"
|
||||
description = "Discord webhook URL (token embedded in URL)"
|
||||
regex = '''https://(?:discord|discordapp)\.com/api/webhooks/\d{17,20}/[A-Za-z0-9_-]{60,}'''
|
||||
tags = ["discord", "webhook", "secret"]
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Global allowlist
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
[allowlist]
|
||||
description = "Paths and patterns that are known false positives"
|
||||
|
||||
paths = [
|
||||
'''(?i)\.(jpg|jpeg|png|gif|bmp|svg|webp|pdf|ico|mp3|mp4|wav|flac|m4a)$''',
|
||||
'''(^|/)(package-lock\.json|yarn\.lock|pnpm-lock\.yaml|poetry\.lock|Podfile\.lock|Cargo\.lock|go\.sum|composer\.lock|Gemfile\.lock|uv\.lock)$''',
|
||||
'''(^|/)node_modules/''',
|
||||
'''(^|/)vendor/''',
|
||||
'''(^|/)wp-content/''',
|
||||
'''(^|/)\.dart_tool/''',
|
||||
'''(^|/)\.venv/''',
|
||||
'''(^|/)\.next/''',
|
||||
'''(^|/)build/''',
|
||||
'''(^|/)dist/''',
|
||||
'''(^|/)\.flutter-plugins''',
|
||||
'''(^|/)test[-_]?fixtures?/''',
|
||||
'''(^|/)tests?/.*conftest\.py$''',
|
||||
'''(^|/)tests?/.*test_[^/]+\.py$''',
|
||||
'''(^|/)tests?/.*_test\.py$''',
|
||||
'''(^|/).*\.(test|spec)\.(js|mjs|cjs|ts|tsx|jsx)$''',
|
||||
'''(^|/).*\.test\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
|
||||
'''(^|/).*\.local\.(js|mjs|cjs|ts|tsx|jsx|py)$''',
|
||||
]
|
||||
|
||||
regexes = [
|
||||
# Explicit placeholder patterns
|
||||
'''YOUR_[A-Z_]{2,}''',
|
||||
'''EXAMPLE_[A-Z_]{2,}''',
|
||||
'''PLACEHOLDER_[A-Z_]{2,}''',
|
||||
'''CHANGE[_-]?ME''',
|
||||
'''<[A-Z_]{2,}>''',
|
||||
'''\$\{[A-Z_]{2,}\}''',
|
||||
|
||||
# Firebase / GCP client API keys — public by design (identify the project,
|
||||
# access is gated by Firebase Security Rules).
|
||||
'''AIzaSy[A-Za-z0-9_-]{33}''',
|
||||
|
||||
# Documented local-development placeholder token (CONTRIBUTING.md, dev scripts)
|
||||
'''local-dev-token''',
|
||||
]
|
||||
@@ -1 +0,0 @@
|
||||
[{"name":"generate-buildid","duration":272,"timestamp":244518298649,"id":4,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"load-custom-routes","duration":314,"timestamp":244518299029,"id":5,"parentId":1,"tags":{},"startTime":1775262331361,"traceId":"b3ef0d179e8acedf"},{"name":"create-dist-dir","duration":333,"timestamp":244518299363,"id":6,"parentId":1,"tags":{},"startTime":1775262331362,"traceId":"b3ef0d179e8acedf"},{"name":"clean","duration":358,"timestamp":244518300558,"id":7,"parentId":1,"tags":{},"startTime":1775262331363,"traceId":"b3ef0d179e8acedf"},{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -1 +0,0 @@
|
||||
[{"name":"next-build","duration":1670851,"timestamp":244516630180,"id":1,"tags":{"buildMode":"default","version":"16.2.2","bundler":"turbopack","failed":true},"startTime":1775262329693,"traceId":"b3ef0d179e8acedf"}]
|
||||
@@ -1,50 +0,0 @@
|
||||
# [PROJECT_NAME] Constitution
|
||||
<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
|
||||
|
||||
## Core Principles
|
||||
|
||||
### [PRINCIPLE_1_NAME]
|
||||
<!-- Example: I. Library-First -->
|
||||
[PRINCIPLE_1_DESCRIPTION]
|
||||
<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
|
||||
|
||||
### [PRINCIPLE_2_NAME]
|
||||
<!-- Example: II. CLI Interface -->
|
||||
[PRINCIPLE_2_DESCRIPTION]
|
||||
<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
|
||||
|
||||
### [PRINCIPLE_3_NAME]
|
||||
<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
|
||||
[PRINCIPLE_3_DESCRIPTION]
|
||||
<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
|
||||
|
||||
### [PRINCIPLE_4_NAME]
|
||||
<!-- Example: IV. Integration Testing -->
|
||||
[PRINCIPLE_4_DESCRIPTION]
|
||||
<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
|
||||
|
||||
### [PRINCIPLE_5_NAME]
|
||||
<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
|
||||
[PRINCIPLE_5_DESCRIPTION]
|
||||
<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
|
||||
|
||||
## [SECTION_2_NAME]
|
||||
<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
|
||||
|
||||
[SECTION_2_CONTENT]
|
||||
<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
|
||||
|
||||
## [SECTION_3_NAME]
|
||||
<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
|
||||
|
||||
[SECTION_3_CONTENT]
|
||||
<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
|
||||
|
||||
## Governance
|
||||
<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
|
||||
|
||||
[GOVERNANCE_RULES]
|
||||
<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
|
||||
|
||||
**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
|
||||
<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->
|
||||
@@ -1,166 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Consolidated prerequisite checking script
|
||||
#
|
||||
# This script provides unified prerequisite checking for Spec-Driven Development workflow.
|
||||
# It replaces the functionality previously spread across multiple scripts.
|
||||
#
|
||||
# Usage: ./check-prerequisites.sh [OPTIONS]
|
||||
#
|
||||
# OPTIONS:
|
||||
# --json Output in JSON format
|
||||
# --require-tasks Require tasks.md to exist (for implementation phase)
|
||||
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
# --paths-only Only output path variables (no validation)
|
||||
# --help, -h Show help message
|
||||
#
|
||||
# OUTPUTS:
|
||||
# JSON mode: {"FEATURE_DIR":"...", "AVAILABLE_DOCS":["..."]}
|
||||
# Text mode: FEATURE_DIR:... \n AVAILABLE_DOCS: \n ✓/✗ file.md
|
||||
# Paths only: REPO_ROOT: ... \n BRANCH: ... \n FEATURE_DIR: ... etc.
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
REQUIRE_TASKS=false
|
||||
INCLUDE_TASKS=false
|
||||
PATHS_ONLY=false
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--require-tasks)
|
||||
REQUIRE_TASKS=true
|
||||
;;
|
||||
--include-tasks)
|
||||
INCLUDE_TASKS=true
|
||||
;;
|
||||
--paths-only)
|
||||
PATHS_ONLY=true
|
||||
;;
|
||||
--help|-h)
|
||||
cat << 'EOF'
|
||||
Usage: check-prerequisites.sh [OPTIONS]
|
||||
|
||||
Consolidated prerequisite checking for Spec-Driven Development workflow.
|
||||
|
||||
OPTIONS:
|
||||
--json Output in JSON format
|
||||
--require-tasks Require tasks.md to exist (for implementation phase)
|
||||
--include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
--paths-only Only output path variables (no prerequisite validation)
|
||||
--help, -h Show this help message
|
||||
|
||||
EXAMPLES:
|
||||
# Check task prerequisites (plan.md required)
|
||||
./check-prerequisites.sh --json
|
||||
|
||||
# Check implementation prerequisites (plan.md + tasks.md required)
|
||||
./check-prerequisites.sh --json --require-tasks --include-tasks
|
||||
|
||||
# Get feature paths only (no validation)
|
||||
./check-prerequisites.sh --paths-only
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown option '$arg'. Use --help for usage information." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Source common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get feature paths and validate branch
|
||||
eval $(get_feature_paths)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# If paths-only mode, output paths and exit (support JSON + paths-only combined)
|
||||
if $PATHS_ONLY; then
|
||||
if $JSON_MODE; then
|
||||
# Minimal JSON paths payload (no validation performed)
|
||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
|
||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
|
||||
else
|
||||
echo "REPO_ROOT: $REPO_ROOT"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "FEATURE_DIR: $FEATURE_DIR"
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "TASKS: $TASKS"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Validate required directories and files
|
||||
if [[ ! -d "$FEATURE_DIR" ]]; then
|
||||
echo "ERROR: Feature directory not found: $FEATURE_DIR" >&2
|
||||
echo "Run /specify first to create the feature structure." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$IMPL_PLAN" ]]; then
|
||||
echo "ERROR: plan.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /plan first to create the implementation plan." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for tasks.md if required
|
||||
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
|
||||
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /tasks first to create the task list." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build list of available documents
|
||||
docs=()
|
||||
|
||||
# Always check these optional docs
|
||||
[[ -f "$RESEARCH" ]] && docs+=("research.md")
|
||||
[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md")
|
||||
|
||||
# Check contracts directory (only if it exists and has files)
|
||||
if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then
|
||||
docs+=("contracts/")
|
||||
fi
|
||||
|
||||
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
|
||||
|
||||
# Include tasks.md if requested and it exists
|
||||
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
|
||||
docs+=("tasks.md")
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
# Build JSON array of documents
|
||||
if [[ ${#docs[@]} -eq 0 ]]; then
|
||||
json_docs="[]"
|
||||
else
|
||||
json_docs=$(printf '"%s",' "${docs[@]}")
|
||||
json_docs="[${json_docs%,}]"
|
||||
fi
|
||||
|
||||
printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s}\n' "$FEATURE_DIR" "$json_docs"
|
||||
else
|
||||
# Text output
|
||||
echo "FEATURE_DIR:$FEATURE_DIR"
|
||||
echo "AVAILABLE_DOCS:"
|
||||
|
||||
# Show status of each potential document
|
||||
check_file "$RESEARCH" "research.md"
|
||||
check_file "$DATA_MODEL" "data-model.md"
|
||||
check_dir "$CONTRACTS_DIR" "contracts/"
|
||||
check_file "$QUICKSTART" "quickstart.md"
|
||||
|
||||
if $INCLUDE_TASKS; then
|
||||
check_file "$TASKS" "tasks.md"
|
||||
fi
|
||||
fi
|
||||
@@ -1,113 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Common functions and variables for all scripts
|
||||
|
||||
# Get repository root, with fallback for non-git repositories
|
||||
get_repo_root() {
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
git rev-parse --show-toplevel
|
||||
else
|
||||
# Fall back to script location for non-git repos
|
||||
local script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
(cd "$script_dir/../../.." && pwd)
|
||||
fi
|
||||
}
|
||||
|
||||
# Get current branch, with fallback for non-git repositories
|
||||
get_current_branch() {
|
||||
# First check if SPECIFY_FEATURE environment variable is set
|
||||
if [[ -n "${SPECIFY_FEATURE:-}" ]]; then
|
||||
echo "$SPECIFY_FEATURE"
|
||||
return
|
||||
fi
|
||||
|
||||
# Then check git if available
|
||||
if git rev-parse --abbrev-ref HEAD >/dev/null 2>&1; then
|
||||
git rev-parse --abbrev-ref HEAD
|
||||
return
|
||||
fi
|
||||
|
||||
# For non-git repos, try to find the latest feature directory
|
||||
local repo_root=$(get_repo_root)
|
||||
local specs_dir="$repo_root/specs"
|
||||
|
||||
if [[ -d "$specs_dir" ]]; then
|
||||
local latest_feature=""
|
||||
local highest=0
|
||||
|
||||
for dir in "$specs_dir"/*; do
|
||||
if [[ -d "$dir" ]]; then
|
||||
local dirname=$(basename "$dir")
|
||||
if [[ "$dirname" =~ ^([0-9]{3})- ]]; then
|
||||
local number=${BASH_REMATCH[1]}
|
||||
number=$((10#$number))
|
||||
if [[ "$number" -gt "$highest" ]]; then
|
||||
highest=$number
|
||||
latest_feature=$dirname
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -n "$latest_feature" ]]; then
|
||||
echo "$latest_feature"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "main" # Final fallback
|
||||
}
|
||||
|
||||
# Check if we have git available
|
||||
has_git() {
|
||||
git rev-parse --show-toplevel >/dev/null 2>&1
|
||||
}
|
||||
|
||||
check_feature_branch() {
|
||||
local branch="$1"
|
||||
local has_git_repo="$2"
|
||||
|
||||
# For non-git repos, we can't enforce branch naming but still provide output
|
||||
if [[ "$has_git_repo" != "true" ]]; then
|
||||
echo "[specify] Warning: Git repository not detected; skipped branch validation" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ ! "$branch" =~ ^[0-9]{3}- ]]; then
|
||||
echo "ERROR: Not on a feature branch. Current branch: $branch" >&2
|
||||
echo "Feature branches should be named like: 001-feature-name" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
get_feature_dir() { echo "$1/specs/$2"; }
|
||||
|
||||
get_feature_paths() {
|
||||
local repo_root=$(get_repo_root)
|
||||
local current_branch=$(get_current_branch)
|
||||
local has_git_repo="false"
|
||||
|
||||
if has_git; then
|
||||
has_git_repo="true"
|
||||
fi
|
||||
|
||||
local feature_dir=$(get_feature_dir "$repo_root" "$current_branch")
|
||||
|
||||
cat <<EOF
|
||||
REPO_ROOT='$repo_root'
|
||||
CURRENT_BRANCH='$current_branch'
|
||||
HAS_GIT='$has_git_repo'
|
||||
FEATURE_DIR='$feature_dir'
|
||||
FEATURE_SPEC='$feature_dir/spec.md'
|
||||
IMPL_PLAN='$feature_dir/plan.md'
|
||||
TASKS='$feature_dir/tasks.md'
|
||||
RESEARCH='$feature_dir/research.md'
|
||||
DATA_MODEL='$feature_dir/data-model.md'
|
||||
QUICKSTART='$feature_dir/quickstart.md'
|
||||
CONTRACTS_DIR='$feature_dir/contracts'
|
||||
EOF
|
||||
}
|
||||
|
||||
check_file() { [[ -f "$1" ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
check_dir() { [[ -d "$1" && -n $(ls -A "$1" 2>/dev/null) ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
JSON_MODE=false
|
||||
ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json) JSON_MODE=true ;;
|
||||
--help|-h) echo "Usage: $0 [--json] <feature_description>"; exit 0 ;;
|
||||
*) ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
FEATURE_DESCRIPTION="${ARGS[*]}"
|
||||
if [ -z "$FEATURE_DESCRIPTION" ]; then
|
||||
echo "Usage: $0 [--json] <feature_description>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to find the repository root by searching for existing project markers
|
||||
find_repo_root() {
|
||||
local dir="$1"
|
||||
while [ "$dir" != "/" ]; do
|
||||
if [ -d "$dir/.git" ] || [ -d "$dir/.specify" ]; then
|
||||
echo "$dir"
|
||||
return 0
|
||||
fi
|
||||
dir="$(dirname "$dir")"
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Resolve repository root. Prefer git information when available, but fall back
|
||||
# to searching for repository markers so the workflow still functions in repositories that
|
||||
# were initialised with --no-git.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
REPO_ROOT=$(git rev-parse --show-toplevel)
|
||||
HAS_GIT=true
|
||||
else
|
||||
REPO_ROOT="$(find_repo_root "$SCRIPT_DIR")"
|
||||
if [ -z "$REPO_ROOT" ]; then
|
||||
echo "Error: Could not determine repository root. Please run this script from within the repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
HAS_GIT=false
|
||||
fi
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
SPECS_DIR="$REPO_ROOT/specs"
|
||||
mkdir -p "$SPECS_DIR"
|
||||
|
||||
HIGHEST=0
|
||||
if [ -d "$SPECS_DIR" ]; then
|
||||
for dir in "$SPECS_DIR"/*; do
|
||||
[ -d "$dir" ] || continue
|
||||
dirname=$(basename "$dir")
|
||||
number=$(echo "$dirname" | grep -o '^[0-9]\+' || echo "0")
|
||||
number=$((10#$number))
|
||||
if [ "$number" -gt "$HIGHEST" ]; then HIGHEST=$number; fi
|
||||
done
|
||||
fi
|
||||
|
||||
NEXT=$((HIGHEST + 1))
|
||||
FEATURE_NUM=$(printf "%03d" "$NEXT")
|
||||
|
||||
BRANCH_NAME=$(echo "$FEATURE_DESCRIPTION" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//')
|
||||
WORDS=$(echo "$BRANCH_NAME" | tr '-' '\n' | grep -v '^$' | head -3 | tr '\n' '-' | sed 's/-$//')
|
||||
BRANCH_NAME="${FEATURE_NUM}-${WORDS}"
|
||||
|
||||
if [ "$HAS_GIT" = true ]; then
|
||||
git checkout -b "$BRANCH_NAME"
|
||||
else
|
||||
>&2 echo "[specify] Warning: Git repository not detected; skipped branch creation for $BRANCH_NAME"
|
||||
fi
|
||||
|
||||
FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME"
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/spec-template.md"
|
||||
SPEC_FILE="$FEATURE_DIR/spec.md"
|
||||
if [ -f "$TEMPLATE" ]; then cp "$TEMPLATE" "$SPEC_FILE"; else touch "$SPEC_FILE"; fi
|
||||
|
||||
# Set the SPECIFY_FEATURE environment variable for the current session
|
||||
export SPECIFY_FEATURE="$BRANCH_NAME"
|
||||
|
||||
if $JSON_MODE; then
|
||||
printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s"}\n' "$BRANCH_NAME" "$SPEC_FILE" "$FEATURE_NUM"
|
||||
else
|
||||
echo "BRANCH_NAME: $BRANCH_NAME"
|
||||
echo "SPEC_FILE: $SPEC_FILE"
|
||||
echo "FEATURE_NUM: $FEATURE_NUM"
|
||||
echo "SPECIFY_FEATURE environment variable set to: $BRANCH_NAME"
|
||||
fi
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [--json]"
|
||||
echo " --json Output results in JSON format"
|
||||
echo " --help Show this help message"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$arg")
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
# Check if we're on a proper feature branch (only for git repos)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# Ensure the feature directory exists
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
# Copy plan template if it exists
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/plan-template.md"
|
||||
if [[ -f "$TEMPLATE" ]]; then
|
||||
cp "$TEMPLATE" "$IMPL_PLAN"
|
||||
echo "Copied plan template to $IMPL_PLAN"
|
||||
else
|
||||
echo "Warning: Plan template not found at $TEMPLATE"
|
||||
# Create a basic plan file if template doesn't exist
|
||||
touch "$IMPL_PLAN"
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
printf '{"FEATURE_SPEC":"%s","IMPL_PLAN":"%s","SPECS_DIR":"%s","BRANCH":"%s","HAS_GIT":"%s"}\n' \
|
||||
"$FEATURE_SPEC" "$IMPL_PLAN" "$FEATURE_DIR" "$CURRENT_BRANCH" "$HAS_GIT"
|
||||
else
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "SPECS_DIR: $FEATURE_DIR"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "HAS_GIT: $HAS_GIT"
|
||||
fi
|
||||
@@ -1,719 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Update agent context files with information from plan.md
|
||||
#
|
||||
# This script maintains AI agent context files by parsing feature specifications
|
||||
# and updating agent-specific configuration files with project information.
|
||||
#
|
||||
# MAIN FUNCTIONS:
|
||||
# 1. Environment Validation
|
||||
# - Verifies git repository structure and branch information
|
||||
# - Checks for required plan.md files and templates
|
||||
# - Validates file permissions and accessibility
|
||||
#
|
||||
# 2. Plan Data Extraction
|
||||
# - Parses plan.md files to extract project metadata
|
||||
# - Identifies language/version, frameworks, databases, and project types
|
||||
# - Handles missing or incomplete specification data gracefully
|
||||
#
|
||||
# 3. Agent File Management
|
||||
# - Creates new agent context files from templates when needed
|
||||
# - Updates existing agent files with new project information
|
||||
# - Preserves manual additions and custom configurations
|
||||
# - Supports multiple AI agent formats and directory structures
|
||||
#
|
||||
# 4. Content Generation
|
||||
# - Generates language-specific build/test commands
|
||||
# - Creates appropriate project directory structures
|
||||
# - Updates technology stacks and recent changes sections
|
||||
# - Maintains consistent formatting and timestamps
|
||||
#
|
||||
# 5. Multi-Agent Support
|
||||
# - Handles agent-specific file paths and naming conventions
|
||||
# - Supports: Claude, Gemini, Copilot, Cursor, Qwen, opencode, Codex, Windsurf
|
||||
# - Can update single agents or all existing agent files
|
||||
# - Creates default Claude file if no agent files exist
|
||||
#
|
||||
# Usage: ./update-agent-context.sh [agent_type]
|
||||
# Agent types: claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf
|
||||
# Leave empty to update all existing agent files
|
||||
|
||||
set -e
|
||||
|
||||
# Enable strict error handling
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
#==============================================================================
|
||||
# Configuration and Global Variables
|
||||
#==============================================================================
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
NEW_PLAN="$IMPL_PLAN" # Alias for compatibility with existing code
|
||||
AGENT_TYPE="${1:-}"
|
||||
|
||||
# Agent-specific file paths
|
||||
CLAUDE_FILE="$REPO_ROOT/CLAUDE.md"
|
||||
GEMINI_FILE="$REPO_ROOT/GEMINI.md"
|
||||
COPILOT_FILE="$REPO_ROOT/.github/copilot-instructions.md"
|
||||
CURSOR_FILE="$REPO_ROOT/.cursor/rules/specify-rules.mdc"
|
||||
QWEN_FILE="$REPO_ROOT/QWEN.md"
|
||||
AGENTS_FILE="$REPO_ROOT/AGENTS.md"
|
||||
WINDSURF_FILE="$REPO_ROOT/.windsurf/rules/specify-rules.md"
|
||||
KILOCODE_FILE="$REPO_ROOT/.kilocode/rules/specify-rules.md"
|
||||
AUGGIE_FILE="$REPO_ROOT/.augment/rules/specify-rules.md"
|
||||
ROO_FILE="$REPO_ROOT/.roo/rules/specify-rules.md"
|
||||
|
||||
# Template file
|
||||
TEMPLATE_FILE="$REPO_ROOT/.specify/templates/agent-file-template.md"
|
||||
|
||||
# Global variables for parsed plan data
|
||||
NEW_LANG=""
|
||||
NEW_FRAMEWORK=""
|
||||
NEW_DB=""
|
||||
NEW_PROJECT_TYPE=""
|
||||
|
||||
#==============================================================================
|
||||
# Utility Functions
|
||||
#==============================================================================
|
||||
|
||||
log_info() {
|
||||
echo "INFO: $1"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo "✓ $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo "ERROR: $1" >&2
|
||||
}
|
||||
|
||||
log_warning() {
|
||||
echo "WARNING: $1" >&2
|
||||
}
|
||||
|
||||
# Cleanup function for temporary files
|
||||
cleanup() {
|
||||
local exit_code=$?
|
||||
rm -f /tmp/agent_update_*_$$
|
||||
rm -f /tmp/manual_additions_$$
|
||||
exit $exit_code
|
||||
}
|
||||
|
||||
# Set up cleanup trap
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
#==============================================================================
|
||||
# Validation Functions
|
||||
#==============================================================================
|
||||
|
||||
validate_environment() {
|
||||
# Check if we have a current branch/feature (git or non-git)
|
||||
if [[ -z "$CURRENT_BRANCH" ]]; then
|
||||
log_error "Unable to determine current feature"
|
||||
if [[ "$HAS_GIT" == "true" ]]; then
|
||||
log_info "Make sure you're on a feature branch"
|
||||
else
|
||||
log_info "Set SPECIFY_FEATURE environment variable or create a feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if plan.md exists
|
||||
if [[ ! -f "$NEW_PLAN" ]]; then
|
||||
log_error "No plan.md found at $NEW_PLAN"
|
||||
log_info "Make sure you're working on a feature with a corresponding spec directory"
|
||||
if [[ "$HAS_GIT" != "true" ]]; then
|
||||
log_info "Use: export SPECIFY_FEATURE=your-feature-name or create a new feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if template exists (needed for new files)
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_warning "Template file not found at $TEMPLATE_FILE"
|
||||
log_warning "Creating new agent files will fail"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Plan Parsing Functions
|
||||
#==============================================================================
|
||||
|
||||
extract_plan_field() {
|
||||
local field_pattern="$1"
|
||||
local plan_file="$2"
|
||||
|
||||
grep "^\*\*${field_pattern}\*\*: " "$plan_file" 2>/dev/null | \
|
||||
head -1 | \
|
||||
sed "s|^\*\*${field_pattern}\*\*: ||" | \
|
||||
sed 's/^[ \t]*//;s/[ \t]*$//' | \
|
||||
grep -v "NEEDS CLARIFICATION" | \
|
||||
grep -v "^N/A$" || echo ""
|
||||
}
|
||||
|
||||
parse_plan_data() {
|
||||
local plan_file="$1"
|
||||
|
||||
if [[ ! -f "$plan_file" ]]; then
|
||||
log_error "Plan file not found: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$plan_file" ]]; then
|
||||
log_error "Plan file is not readable: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Parsing plan data from $plan_file"
|
||||
|
||||
NEW_LANG=$(extract_plan_field "Language/Version" "$plan_file")
|
||||
NEW_FRAMEWORK=$(extract_plan_field "Primary Dependencies" "$plan_file")
|
||||
NEW_DB=$(extract_plan_field "Storage" "$plan_file")
|
||||
NEW_PROJECT_TYPE=$(extract_plan_field "Project Type" "$plan_file")
|
||||
|
||||
# Log what we found
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
log_info "Found language: $NEW_LANG"
|
||||
else
|
||||
log_warning "No language information found in plan"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
log_info "Found framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
log_info "Found database: $NEW_DB"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_PROJECT_TYPE" ]]; then
|
||||
log_info "Found project type: $NEW_PROJECT_TYPE"
|
||||
fi
|
||||
}
|
||||
|
||||
format_technology_stack() {
|
||||
local lang="$1"
|
||||
local framework="$2"
|
||||
local parts=()
|
||||
|
||||
# Add non-empty parts
|
||||
[[ -n "$lang" && "$lang" != "NEEDS CLARIFICATION" ]] && parts+=("$lang")
|
||||
[[ -n "$framework" && "$framework" != "NEEDS CLARIFICATION" && "$framework" != "N/A" ]] && parts+=("$framework")
|
||||
|
||||
# Join with proper formatting
|
||||
if [[ ${#parts[@]} -eq 0 ]]; then
|
||||
echo ""
|
||||
elif [[ ${#parts[@]} -eq 1 ]]; then
|
||||
echo "${parts[0]}"
|
||||
else
|
||||
# Join multiple parts with " + "
|
||||
local result="${parts[0]}"
|
||||
for ((i=1; i<${#parts[@]}; i++)); do
|
||||
result="$result + ${parts[i]}"
|
||||
done
|
||||
echo "$result"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Template and Content Generation Functions
|
||||
#==============================================================================
|
||||
|
||||
get_project_structure() {
|
||||
local project_type="$1"
|
||||
|
||||
if [[ "$project_type" == *"web"* ]]; then
|
||||
echo "backend/\\nfrontend/\\ntests/"
|
||||
else
|
||||
echo "src/\\ntests/"
|
||||
fi
|
||||
}
|
||||
|
||||
get_commands_for_language() {
|
||||
local lang="$1"
|
||||
|
||||
case "$lang" in
|
||||
*"Python"*)
|
||||
echo "cd src && pytest && ruff check ."
|
||||
;;
|
||||
*"Rust"*)
|
||||
echo "cargo test && cargo clippy"
|
||||
;;
|
||||
*"JavaScript"*|*"TypeScript"*)
|
||||
echo "npm test && npm run lint"
|
||||
;;
|
||||
*)
|
||||
echo "# Add commands for $lang"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
get_language_conventions() {
|
||||
local lang="$1"
|
||||
echo "$lang: Follow standard conventions"
|
||||
}
|
||||
|
||||
create_new_agent_file() {
|
||||
local target_file="$1"
|
||||
local temp_file="$2"
|
||||
local project_name="$3"
|
||||
local current_date="$4"
|
||||
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template not found at $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template file is not readable: $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Creating new agent context file from template..."
|
||||
|
||||
if ! cp "$TEMPLATE_FILE" "$temp_file"; then
|
||||
log_error "Failed to copy template file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Replace template placeholders
|
||||
local project_structure
|
||||
project_structure=$(get_project_structure "$NEW_PROJECT_TYPE")
|
||||
|
||||
local commands
|
||||
commands=$(get_commands_for_language "$NEW_LANG")
|
||||
|
||||
local language_conventions
|
||||
language_conventions=$(get_language_conventions "$NEW_LANG")
|
||||
|
||||
# Perform substitutions with error checking using safer approach
|
||||
# Escape special characters for sed by using a different delimiter or escaping
|
||||
local escaped_lang=$(printf '%s\n' "$NEW_LANG" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_framework=$(printf '%s\n' "$NEW_FRAMEWORK" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_branch=$(printf '%s\n' "$CURRENT_BRANCH" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
|
||||
# Build technology stack and recent change strings conditionally
|
||||
local tech_stack
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_lang + $escaped_framework ($escaped_branch)"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
tech_stack="- $escaped_lang ($escaped_branch)"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_framework ($escaped_branch)"
|
||||
else
|
||||
tech_stack="- ($escaped_branch)"
|
||||
fi
|
||||
|
||||
local recent_change
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang + $escaped_framework"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_framework"
|
||||
else
|
||||
recent_change="- $escaped_branch: Added"
|
||||
fi
|
||||
|
||||
local substitutions=(
|
||||
"s|\[PROJECT NAME\]|$project_name|"
|
||||
"s|\[DATE\]|$current_date|"
|
||||
"s|\[EXTRACTED FROM ALL PLAN.MD FILES\]|$tech_stack|"
|
||||
"s|\[ACTUAL STRUCTURE FROM PLANS\]|$project_structure|g"
|
||||
"s|\[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES\]|$commands|"
|
||||
"s|\[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE\]|$language_conventions|"
|
||||
"s|\[LAST 3 FEATURES AND WHAT THEY ADDED\]|$recent_change|"
|
||||
)
|
||||
|
||||
for substitution in "${substitutions[@]}"; do
|
||||
if ! sed -i.bak -e "$substitution" "$temp_file"; then
|
||||
log_error "Failed to perform substitution: $substitution"
|
||||
rm -f "$temp_file" "$temp_file.bak"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Convert \n sequences to actual newlines
|
||||
newline=$(printf '\n')
|
||||
sed -i.bak2 "s/\\\\n/${newline}/g" "$temp_file"
|
||||
|
||||
# Clean up backup files
|
||||
rm -f "$temp_file.bak" "$temp_file.bak2"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
update_existing_agent_file() {
|
||||
local target_file="$1"
|
||||
local current_date="$2"
|
||||
|
||||
log_info "Updating existing agent context file..."
|
||||
|
||||
# Use a single temporary file for atomic update
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Process the file in one pass
|
||||
local tech_stack=$(format_technology_stack "$NEW_LANG" "$NEW_FRAMEWORK")
|
||||
local new_tech_entries=()
|
||||
local new_change_entry=""
|
||||
|
||||
# Prepare new technology entries
|
||||
if [[ -n "$tech_stack" ]] && ! grep -q "$tech_stack" "$target_file"; then
|
||||
new_tech_entries+=("- $tech_stack ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]] && ! grep -q "$NEW_DB" "$target_file"; then
|
||||
new_tech_entries+=("- $NEW_DB ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
# Prepare new change entry
|
||||
if [[ -n "$tech_stack" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $tech_stack"
|
||||
elif [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $NEW_DB"
|
||||
fi
|
||||
|
||||
# Process file line by line
|
||||
local in_tech_section=false
|
||||
local in_changes_section=false
|
||||
local tech_entries_added=false
|
||||
local changes_entries_added=false
|
||||
local existing_changes_count=0
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
# Handle Active Technologies section
|
||||
if [[ "$line" == "## Active Technologies" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=true
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
# Add new tech entries before closing the section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=false
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ -z "$line" ]]; then
|
||||
# Add new tech entries before empty line in tech section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Handle Recent Changes section
|
||||
if [[ "$line" == "## Recent Changes" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
# Add new change entry right after the heading
|
||||
if [[ -n "$new_change_entry" ]]; then
|
||||
echo "$new_change_entry" >> "$temp_file"
|
||||
fi
|
||||
in_changes_section=true
|
||||
changes_entries_added=true
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_changes_section=false
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" == "- "* ]]; then
|
||||
# Keep only first 2 existing changes
|
||||
if [[ $existing_changes_count -lt 2 ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
((existing_changes_count++))
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
|
||||
# Update timestamp
|
||||
if [[ "$line" =~ \*\*Last\ updated\*\*:.*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
|
||||
echo "$line" | sed "s/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/$current_date/" >> "$temp_file"
|
||||
else
|
||||
echo "$line" >> "$temp_file"
|
||||
fi
|
||||
done < "$target_file"
|
||||
|
||||
# Post-loop check: if we're still in the Active Technologies section and haven't added new entries
|
||||
if [[ $in_tech_section == true ]] && [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
fi
|
||||
|
||||
# Move temp file to target atomically
|
||||
if ! mv "$temp_file" "$target_file"; then
|
||||
log_error "Failed to update target file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
#==============================================================================
|
||||
# Main Agent File Update Function
|
||||
#==============================================================================
|
||||
|
||||
update_agent_file() {
|
||||
local target_file="$1"
|
||||
local agent_name="$2"
|
||||
|
||||
if [[ -z "$target_file" ]] || [[ -z "$agent_name" ]]; then
|
||||
log_error "update_agent_file requires target_file and agent_name parameters"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Updating $agent_name context file: $target_file"
|
||||
|
||||
local project_name
|
||||
project_name=$(basename "$REPO_ROOT")
|
||||
local current_date
|
||||
current_date=$(date +%Y-%m-%d)
|
||||
|
||||
# Create directory if it doesn't exist
|
||||
local target_dir
|
||||
target_dir=$(dirname "$target_file")
|
||||
if [[ ! -d "$target_dir" ]]; then
|
||||
if ! mkdir -p "$target_dir"; then
|
||||
log_error "Failed to create directory: $target_dir"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ ! -f "$target_file" ]]; then
|
||||
# Create new file from template
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
if create_new_agent_file "$target_file" "$temp_file" "$project_name" "$current_date"; then
|
||||
if mv "$temp_file" "$target_file"; then
|
||||
log_success "Created new $agent_name context file"
|
||||
else
|
||||
log_error "Failed to move temporary file to $target_file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log_error "Failed to create new agent file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
# Update existing file
|
||||
if [[ ! -r "$target_file" ]]; then
|
||||
log_error "Cannot read existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -w "$target_file" ]]; then
|
||||
log_error "Cannot write to existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if update_existing_agent_file "$target_file" "$current_date"; then
|
||||
log_success "Updated existing $agent_name context file"
|
||||
else
|
||||
log_error "Failed to update existing agent file"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Agent Selection and Processing
|
||||
#==============================================================================
|
||||
|
||||
update_specific_agent() {
|
||||
local agent_type="$1"
|
||||
|
||||
case "$agent_type" in
|
||||
claude)
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
;;
|
||||
gemini)
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
;;
|
||||
copilot)
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
;;
|
||||
cursor)
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
;;
|
||||
qwen)
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
;;
|
||||
opencode)
|
||||
update_agent_file "$AGENTS_FILE" "opencode"
|
||||
;;
|
||||
codex)
|
||||
update_agent_file "$AGENTS_FILE" "Codex CLI"
|
||||
;;
|
||||
windsurf)
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
;;
|
||||
kilocode)
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
;;
|
||||
auggie)
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
;;
|
||||
roo)
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown agent type '$agent_type'"
|
||||
log_error "Expected: claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf|kilocode|auggie|roo"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
update_all_existing_agents() {
|
||||
local found_agent=false
|
||||
|
||||
# Check each possible agent file and update if it exists
|
||||
if [[ -f "$CLAUDE_FILE" ]]; then
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$GEMINI_FILE" ]]; then
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$COPILOT_FILE" ]]; then
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$CURSOR_FILE" ]]; then
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$QWEN_FILE" ]]; then
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AGENTS_FILE" ]]; then
|
||||
update_agent_file "$AGENTS_FILE" "Codex/opencode"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$WINDSURF_FILE" ]]; then
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$KILOCODE_FILE" ]]; then
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AUGGIE_FILE" ]]; then
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$ROO_FILE" ]]; then
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
# If no agent files exist, create a default Claude file
|
||||
if [[ "$found_agent" == false ]]; then
|
||||
log_info "No existing agent files found, creating default Claude file..."
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
fi
|
||||
}
|
||||
print_summary() {
|
||||
echo
|
||||
log_info "Summary of changes:"
|
||||
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
echo " - Added language: $NEW_LANG"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
echo " - Added framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
echo " - Added database: $NEW_DB"
|
||||
fi
|
||||
|
||||
echo
|
||||
log_info "Usage: $0 [claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf|kilocode|auggie|roo]"
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Main Execution
|
||||
#==============================================================================
|
||||
|
||||
main() {
|
||||
# Validate environment before proceeding
|
||||
validate_environment
|
||||
|
||||
log_info "=== Updating agent context files for feature $CURRENT_BRANCH ==="
|
||||
|
||||
# Parse the plan file to extract project information
|
||||
if ! parse_plan_data "$NEW_PLAN"; then
|
||||
log_error "Failed to parse plan data"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Process based on agent type argument
|
||||
local success=true
|
||||
|
||||
if [[ -z "$AGENT_TYPE" ]]; then
|
||||
# No specific agent provided - update all existing agent files
|
||||
log_info "No agent specified, updating all existing agent files..."
|
||||
if ! update_all_existing_agents; then
|
||||
success=false
|
||||
fi
|
||||
else
|
||||
# Specific agent provided - update only that agent
|
||||
log_info "Updating specific agent: $AGENT_TYPE"
|
||||
if ! update_specific_agent "$AGENT_TYPE"; then
|
||||
success=false
|
||||
fi
|
||||
fi
|
||||
|
||||
# Print summary
|
||||
print_summary
|
||||
|
||||
if [[ "$success" == true ]]; then
|
||||
log_success "Agent context update completed successfully"
|
||||
exit 0
|
||||
else
|
||||
log_error "Agent context update completed with errors"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Execute main function if script is run directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -1,23 +0,0 @@
|
||||
# [PROJECT NAME] Development Guidelines
|
||||
|
||||
Auto-generated from all feature plans. Last updated: [DATE]
|
||||
|
||||
## Active Technologies
|
||||
[EXTRACTED FROM ALL PLAN.MD FILES]
|
||||
|
||||
## Project Structure
|
||||
```
|
||||
[ACTUAL STRUCTURE FROM PLANS]
|
||||
```
|
||||
|
||||
## Commands
|
||||
[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES]
|
||||
|
||||
## Code Style
|
||||
[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE]
|
||||
|
||||
## Recent Changes
|
||||
[LAST 3 FEATURES AND WHAT THEY ADDED]
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
<!-- MANUAL ADDITIONS END -->
|
||||
@@ -1,219 +0,0 @@
|
||||
|
||||
# Implementation Plan: [FEATURE]
|
||||
|
||||
**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link]
|
||||
**Input**: Feature specification from `/specs/[###-feature-name]/spec.md`
|
||||
|
||||
## Execution Flow (/plan command scope)
|
||||
```
|
||||
1. Load feature spec from Input path
|
||||
→ If not found: ERROR "No feature spec at {path}"
|
||||
2. Fill Technical Context (scan for NEEDS CLARIFICATION)
|
||||
→ Detect Project Type from file system structure or context (web=frontend+backend, mobile=app+api)
|
||||
→ Set Structure Decision based on project type
|
||||
3. Fill the Constitution Check section based on the content of the constitution document.
|
||||
4. Evaluate Constitution Check section below
|
||||
→ If violations exist: Document in Complexity Tracking
|
||||
→ If no justification possible: ERROR "Simplify approach first"
|
||||
→ Update Progress Tracking: Initial Constitution Check
|
||||
5. Execute Phase 0 → research.md
|
||||
→ If NEEDS CLARIFICATION remain: ERROR "Resolve unknowns"
|
||||
6. Execute Phase 1 → contracts, data-model.md, quickstart.md, agent-specific template file (e.g., `CLAUDE.md` for Claude Code, `.github/copilot-instructions.md` for GitHub Copilot, `GEMINI.md` for Gemini CLI, `QWEN.md` for Qwen Code or `AGENTS.md` for opencode).
|
||||
7. Re-evaluate Constitution Check section
|
||||
→ If new violations: Refactor design, return to Phase 1
|
||||
→ Update Progress Tracking: Post-Design Constitution Check
|
||||
8. Plan Phase 2 → Describe task generation approach (DO NOT create tasks.md)
|
||||
9. STOP - Ready for /tasks command
|
||||
```
|
||||
|
||||
**IMPORTANT**: The /plan command STOPS at step 7. Phases 2-4 are executed by other commands:
|
||||
- Phase 2: /tasks command creates tasks.md
|
||||
- Phase 3-4: Implementation execution (manual or via tools)
|
||||
|
||||
## Summary
|
||||
[Extract from feature spec: primary requirement + technical approach from research]
|
||||
|
||||
## Technical Context
|
||||
**Language/Version**: [e.g., Python 3.11, Swift 5.9, Rust 1.75 or NEEDS CLARIFICATION]
|
||||
**Primary Dependencies**: [e.g., FastAPI, UIKit, LLVM or NEEDS CLARIFICATION]
|
||||
**Storage**: [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
|
||||
**Testing**: [e.g., pytest, XCTest, cargo test or NEEDS CLARIFICATION]
|
||||
**Target Platform**: [e.g., Linux server, iOS 15+, WASM or NEEDS CLARIFICATION]
|
||||
**Project Type**: [single/web/mobile - determines source structure]
|
||||
**Performance Goals**: [domain-specific, e.g., 1000 req/s, 10k lines/sec, 60 fps or NEEDS CLARIFICATION]
|
||||
**Constraints**: [domain-specific, e.g., <200ms p95, <100MB memory, offline-capable or NEEDS CLARIFICATION]
|
||||
**Scale/Scope**: [domain-specific, e.g., 10k users, 1M LOC, 50 screens or NEEDS CLARIFICATION]
|
||||
|
||||
## Constitution Check
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
[Gates determined based on constitution file]
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
```
|
||||
specs/[###-feature]/
|
||||
├── plan.md # This file (/plan command output)
|
||||
├── research.md # Phase 0 output (/plan command)
|
||||
├── data-model.md # Phase 1 output (/plan command)
|
||||
├── quickstart.md # Phase 1 output (/plan command)
|
||||
├── contracts/ # Phase 1 output (/plan command)
|
||||
└── tasks.md # Phase 2 output (/tasks command - NOT created by /plan)
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
<!--
|
||||
ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
|
||||
for this feature. Delete unused options and expand the chosen structure with
|
||||
real paths (e.g., apps/admin, packages/something). The delivered plan must
|
||||
not include Option labels.
|
||||
-->
|
||||
```
|
||||
# [REMOVE IF UNUSED] Option 1: Single project (DEFAULT)
|
||||
src/
|
||||
├── models/
|
||||
├── services/
|
||||
├── cli/
|
||||
└── lib/
|
||||
|
||||
tests/
|
||||
├── contract/
|
||||
├── integration/
|
||||
└── unit/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 2: Web application (when "frontend" + "backend" detected)
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── models/
|
||||
│ ├── services/
|
||||
│ └── api/
|
||||
└── tests/
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ ├── pages/
|
||||
│ └── services/
|
||||
└── tests/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 3: Mobile + API (when "iOS/Android" detected)
|
||||
api/
|
||||
└── [same as backend above]
|
||||
|
||||
ios/ or android/
|
||||
└── [platform-specific structure: feature modules, UI flows, platform tests]
|
||||
```
|
||||
|
||||
**Structure Decision**: [Document the selected structure and reference the real
|
||||
directories captured above]
|
||||
|
||||
## Phase 0: Outline & Research
|
||||
1. **Extract unknowns from Technical Context** above:
|
||||
- For each NEEDS CLARIFICATION → research task
|
||||
- For each dependency → best practices task
|
||||
- For each integration → patterns task
|
||||
|
||||
2. **Generate and dispatch research agents**:
|
||||
```
|
||||
For each unknown in Technical Context:
|
||||
Task: "Research {unknown} for {feature context}"
|
||||
For each technology choice:
|
||||
Task: "Find best practices for {tech} in {domain}"
|
||||
```
|
||||
|
||||
3. **Consolidate findings** in `research.md` using format:
|
||||
- Decision: [what was chosen]
|
||||
- Rationale: [why chosen]
|
||||
- Alternatives considered: [what else evaluated]
|
||||
|
||||
**Output**: research.md with all NEEDS CLARIFICATION resolved
|
||||
|
||||
## Phase 1: Design & Contracts
|
||||
*Prerequisites: research.md complete*
|
||||
|
||||
1. **Extract entities from feature spec** → `data-model.md`:
|
||||
- Entity name, fields, relationships
|
||||
- Validation rules from requirements
|
||||
- State transitions if applicable
|
||||
|
||||
2. **Generate API contracts** from functional requirements:
|
||||
- For each user action → endpoint
|
||||
- Use standard REST/GraphQL patterns
|
||||
- Output OpenAPI/GraphQL schema to `/contracts/`
|
||||
|
||||
3. **Generate contract tests** from contracts:
|
||||
- One test file per endpoint
|
||||
- Assert request/response schemas
|
||||
- Tests must fail (no implementation yet)
|
||||
|
||||
4. **Extract test scenarios** from user stories:
|
||||
- Each story → integration test scenario
|
||||
- Quickstart test = story validation steps
|
||||
|
||||
5. **Update agent file incrementally** (O(1) operation):
|
||||
- Run `.specify/scripts/bash/update-agent-context.sh cursor`
|
||||
**IMPORTANT**: Execute it exactly as specified above. Do not add or remove any arguments.
|
||||
- If exists: Add only NEW tech from current plan
|
||||
- Preserve manual additions between markers
|
||||
- Update recent changes (keep last 3)
|
||||
- Keep under 150 lines for token efficiency
|
||||
- Output to repository root
|
||||
|
||||
**Output**: data-model.md, /contracts/*, failing tests, quickstart.md, agent-specific file
|
||||
|
||||
## Phase 2: Task Planning Approach
|
||||
*This section describes what the /tasks command will do - DO NOT execute during /plan*
|
||||
|
||||
**Task Generation Strategy**:
|
||||
- Load `.specify/templates/tasks-template.md` as base
|
||||
- Generate tasks from Phase 1 design docs (contracts, data model, quickstart)
|
||||
- Each contract → contract test task [P]
|
||||
- Each entity → model creation task [P]
|
||||
- Each user story → integration test task
|
||||
- Implementation tasks to make tests pass
|
||||
|
||||
**Ordering Strategy**:
|
||||
- TDD order: Tests before implementation
|
||||
- Dependency order: Models before services before UI
|
||||
- Mark [P] for parallel execution (independent files)
|
||||
|
||||
**Estimated Output**: 25-30 numbered, ordered tasks in tasks.md
|
||||
|
||||
**IMPORTANT**: This phase is executed by the /tasks command, NOT by /plan
|
||||
|
||||
## Phase 3+: Future Implementation
|
||||
*These phases are beyond the scope of the /plan command*
|
||||
|
||||
**Phase 3**: Task execution (/tasks command creates tasks.md)
|
||||
**Phase 4**: Implementation (execute tasks.md following constitutional principles)
|
||||
**Phase 5**: Validation (run tests, execute quickstart.md, performance validation)
|
||||
|
||||
## Complexity Tracking
|
||||
*Fill ONLY if Constitution Check has violations that must be justified*
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
|
||||
| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |
|
||||
|
||||
|
||||
## Progress Tracking
|
||||
*This checklist is updated during execution flow*
|
||||
|
||||
**Phase Status**:
|
||||
- [ ] Phase 0: Research complete (/plan command)
|
||||
- [ ] Phase 1: Design complete (/plan command)
|
||||
- [ ] Phase 2: Task planning complete (/plan command - describe approach only)
|
||||
- [ ] Phase 3: Tasks generated (/tasks command)
|
||||
- [ ] Phase 4: Implementation complete
|
||||
- [ ] Phase 5: Validation passed
|
||||
|
||||
**Gate Status**:
|
||||
- [ ] Initial Constitution Check: PASS
|
||||
- [ ] Post-Design Constitution Check: PASS
|
||||
- [ ] All NEEDS CLARIFICATION resolved
|
||||
- [ ] Complexity deviations documented
|
||||
|
||||
---
|
||||
*Based on Constitution v2.1.1 - See `/memory/constitution.md`*
|
||||
@@ -1,116 +0,0 @@
|
||||
# Feature Specification: [FEATURE NAME]
|
||||
|
||||
**Feature Branch**: `[###-feature-name]`
|
||||
**Created**: [DATE]
|
||||
**Status**: Draft
|
||||
**Input**: User description: "$ARGUMENTS"
|
||||
|
||||
## Execution Flow (main)
|
||||
```
|
||||
1. Parse user description from Input
|
||||
→ If empty: ERROR "No feature description provided"
|
||||
2. Extract key concepts from description
|
||||
→ Identify: actors, actions, data, constraints
|
||||
3. For each unclear aspect:
|
||||
→ Mark with [NEEDS CLARIFICATION: specific question]
|
||||
4. Fill User Scenarios & Testing section
|
||||
→ If no clear user flow: ERROR "Cannot determine user scenarios"
|
||||
5. Generate Functional Requirements
|
||||
→ Each requirement must be testable
|
||||
→ Mark ambiguous requirements
|
||||
6. Identify Key Entities (if data involved)
|
||||
7. Run Review Checklist
|
||||
→ If any [NEEDS CLARIFICATION]: WARN "Spec has uncertainties"
|
||||
→ If implementation details found: ERROR "Remove tech details"
|
||||
8. Return: SUCCESS (spec ready for planning)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Guidelines
|
||||
- ✅ Focus on WHAT users need and WHY
|
||||
- ❌ Avoid HOW to implement (no tech stack, APIs, code structure)
|
||||
- 👥 Written for business stakeholders, not developers
|
||||
|
||||
### Section Requirements
|
||||
- **Mandatory sections**: Must be completed for every feature
|
||||
- **Optional sections**: Include only when relevant to the feature
|
||||
- When a section doesn't apply, remove it entirely (don't leave as "N/A")
|
||||
|
||||
### For AI Generation
|
||||
When creating this spec from a user prompt:
|
||||
1. **Mark all ambiguities**: Use [NEEDS CLARIFICATION: specific question] for any assumption you'd need to make
|
||||
2. **Don't guess**: If the prompt doesn't specify something (e.g., "login system" without auth method), mark it
|
||||
3. **Think like a tester**: Every vague requirement should fail the "testable and unambiguous" checklist item
|
||||
4. **Common underspecified areas**:
|
||||
- User types and permissions
|
||||
- Data retention/deletion policies
|
||||
- Performance targets and scale
|
||||
- Error handling behaviors
|
||||
- Integration requirements
|
||||
- Security/compliance needs
|
||||
|
||||
---
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### Primary User Story
|
||||
[Describe the main user journey in plain language]
|
||||
|
||||
### Acceptance Scenarios
|
||||
1. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
2. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
|
||||
### Edge Cases
|
||||
- What happens when [boundary condition]?
|
||||
- How does system handle [error scenario]?
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
- **FR-001**: System MUST [specific capability, e.g., "allow users to create accounts"]
|
||||
- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"]
|
||||
- **FR-003**: Users MUST be able to [key interaction, e.g., "reset their password"]
|
||||
- **FR-004**: System MUST [data requirement, e.g., "persist user preferences"]
|
||||
- **FR-005**: System MUST [behavior, e.g., "log all security events"]
|
||||
|
||||
*Example of marking unclear requirements:*
|
||||
- **FR-006**: System MUST authenticate users via [NEEDS CLARIFICATION: auth method not specified - email/password, SSO, OAuth?]
|
||||
- **FR-007**: System MUST retain user data for [NEEDS CLARIFICATION: retention period not specified]
|
||||
|
||||
### Key Entities *(include if feature involves data)*
|
||||
- **[Entity 1]**: [What it represents, key attributes without implementation]
|
||||
- **[Entity 2]**: [What it represents, relationships to other entities]
|
||||
|
||||
---
|
||||
|
||||
## Review & Acceptance Checklist
|
||||
*GATE: Automated checks run during main() execution*
|
||||
|
||||
### Content Quality
|
||||
- [ ] No implementation details (languages, frameworks, APIs)
|
||||
- [ ] Focused on user value and business needs
|
||||
- [ ] Written for non-technical stakeholders
|
||||
- [ ] All mandatory sections completed
|
||||
|
||||
### Requirement Completeness
|
||||
- [ ] No [NEEDS CLARIFICATION] markers remain
|
||||
- [ ] Requirements are testable and unambiguous
|
||||
- [ ] Success criteria are measurable
|
||||
- [ ] Scope is clearly bounded
|
||||
- [ ] Dependencies and assumptions identified
|
||||
|
||||
---
|
||||
|
||||
## Execution Status
|
||||
*Updated by main() during processing*
|
||||
|
||||
- [ ] User description parsed
|
||||
- [ ] Key concepts extracted
|
||||
- [ ] Ambiguities marked
|
||||
- [ ] User scenarios defined
|
||||
- [ ] Requirements generated
|
||||
- [ ] Entities identified
|
||||
- [ ] Review checklist passed
|
||||
|
||||
---
|
||||
@@ -1,127 +0,0 @@
|
||||
# Tasks: [FEATURE NAME]
|
||||
|
||||
**Input**: Design documents from `/specs/[###-feature-name]/`
|
||||
**Prerequisites**: plan.md (required), research.md, data-model.md, contracts/
|
||||
|
||||
## Execution Flow (main)
|
||||
```
|
||||
1. Load plan.md from feature directory
|
||||
→ If not found: ERROR "No implementation plan found"
|
||||
→ Extract: tech stack, libraries, structure
|
||||
2. Load optional design documents:
|
||||
→ data-model.md: Extract entities → model tasks
|
||||
→ contracts/: Each file → contract test task
|
||||
→ research.md: Extract decisions → setup tasks
|
||||
3. Generate tasks by category:
|
||||
→ Setup: project init, dependencies, linting
|
||||
→ Tests: contract tests, integration tests
|
||||
→ Core: models, services, CLI commands
|
||||
→ Integration: DB, middleware, logging
|
||||
→ Polish: unit tests, performance, docs
|
||||
4. Apply task rules:
|
||||
→ Different files = mark [P] for parallel
|
||||
→ Same file = sequential (no [P])
|
||||
→ Tests before implementation (TDD)
|
||||
5. Number tasks sequentially (T001, T002...)
|
||||
6. Generate dependency graph
|
||||
7. Create parallel execution examples
|
||||
8. Validate task completeness:
|
||||
→ All contracts have tests?
|
||||
→ All entities have models?
|
||||
→ All endpoints implemented?
|
||||
9. Return: SUCCESS (tasks ready for execution)
|
||||
```
|
||||
|
||||
## Format: `[ID] [P?] Description`
|
||||
- **[P]**: Can run in parallel (different files, no dependencies)
|
||||
- Include exact file paths in descriptions
|
||||
|
||||
## Path Conventions
|
||||
- **Single project**: `src/`, `tests/` at repository root
|
||||
- **Web app**: `backend/src/`, `frontend/src/`
|
||||
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
|
||||
- Paths shown below assume single project - adjust based on plan.md structure
|
||||
|
||||
## Phase 3.1: Setup
|
||||
- [ ] T001 Create project structure per implementation plan
|
||||
- [ ] T002 Initialize [language] project with [framework] dependencies
|
||||
- [ ] T003 [P] Configure linting and formatting tools
|
||||
|
||||
## Phase 3.2: Tests First (TDD) ⚠️ MUST COMPLETE BEFORE 3.3
|
||||
**CRITICAL: These tests MUST be written and MUST FAIL before ANY implementation**
|
||||
- [ ] T004 [P] Contract test POST /api/users in tests/contract/test_users_post.py
|
||||
- [ ] T005 [P] Contract test GET /api/users/{id} in tests/contract/test_users_get.py
|
||||
- [ ] T006 [P] Integration test user registration in tests/integration/test_registration.py
|
||||
- [ ] T007 [P] Integration test auth flow in tests/integration/test_auth.py
|
||||
|
||||
## Phase 3.3: Core Implementation (ONLY after tests are failing)
|
||||
- [ ] T008 [P] User model in src/models/user.py
|
||||
- [ ] T009 [P] UserService CRUD in src/services/user_service.py
|
||||
- [ ] T010 [P] CLI --create-user in src/cli/user_commands.py
|
||||
- [ ] T011 POST /api/users endpoint
|
||||
- [ ] T012 GET /api/users/{id} endpoint
|
||||
- [ ] T013 Input validation
|
||||
- [ ] T014 Error handling and logging
|
||||
|
||||
## Phase 3.4: Integration
|
||||
- [ ] T015 Connect UserService to DB
|
||||
- [ ] T016 Auth middleware
|
||||
- [ ] T017 Request/response logging
|
||||
- [ ] T018 CORS and security headers
|
||||
|
||||
## Phase 3.5: Polish
|
||||
- [ ] T019 [P] Unit tests for validation in tests/unit/test_validation.py
|
||||
- [ ] T020 Performance tests (<200ms)
|
||||
- [ ] T021 [P] Update docs/api.md
|
||||
- [ ] T022 Remove duplication
|
||||
- [ ] T023 Run manual-testing.md
|
||||
|
||||
## Dependencies
|
||||
- Tests (T004-T007) before implementation (T008-T014)
|
||||
- T008 blocks T009, T015
|
||||
- T016 blocks T018
|
||||
- Implementation before polish (T019-T023)
|
||||
|
||||
## Parallel Example
|
||||
```
|
||||
# Launch T004-T007 together:
|
||||
Task: "Contract test POST /api/users in tests/contract/test_users_post.py"
|
||||
Task: "Contract test GET /api/users/{id} in tests/contract/test_users_get.py"
|
||||
Task: "Integration test registration in tests/integration/test_registration.py"
|
||||
Task: "Integration test auth in tests/integration/test_auth.py"
|
||||
```
|
||||
|
||||
## Notes
|
||||
- [P] tasks = different files, no dependencies
|
||||
- Verify tests fail before implementing
|
||||
- Commit after each task
|
||||
- Avoid: vague tasks, same file conflicts
|
||||
|
||||
## Task Generation Rules
|
||||
*Applied during main() execution*
|
||||
|
||||
1. **From Contracts**:
|
||||
- Each contract file → contract test task [P]
|
||||
- Each endpoint → implementation task
|
||||
|
||||
2. **From Data Model**:
|
||||
- Each entity → model creation task [P]
|
||||
- Relationships → service layer tasks
|
||||
|
||||
3. **From User Stories**:
|
||||
- Each story → integration test [P]
|
||||
- Quickstart scenarios → validation tasks
|
||||
|
||||
4. **Ordering**:
|
||||
- Setup → Tests → Models → Services → Endpoints → Polish
|
||||
- Dependencies block parallel execution
|
||||
|
||||
## Validation Checklist
|
||||
*GATE: Checked by main() before returning*
|
||||
|
||||
- [ ] All contracts have corresponding tests
|
||||
- [ ] All entities have model tasks
|
||||
- [ ] All tests come before implementation
|
||||
- [ ] Parallel tasks truly independent
|
||||
- [ ] Each task specifies exact file path
|
||||
- [ ] No task modifies same file as another [P] task
|
||||
@@ -1,229 +0,0 @@
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Product vision & goals | `docs/PRODUCT-VISION.md` |
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Operational runbooks** | `docs/TROUBLESHOOTING.md` |
|
||||
| GDrive validator & gaps | `docs/GDRIVE-VALIDATOR.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
| **Making medleys** (multi-song karaoke) | `docs/MEDLEYS.md` + `scripts/medley/make_medley.py` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Cloud Run GPU (L4 audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required)
|
||||
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
|
||||
- Which test types are needed (unit, integration, E2E)
|
||||
- Where tests should be placed
|
||||
- What mocking approach to use
|
||||
- Coverage expectations
|
||||
- **Production E2E tests for user-facing features** - see below
|
||||
|
||||
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
|
||||
|
||||
**Before committing**, run all tests:
|
||||
```bash
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
This single command:
|
||||
- Installs dependencies automatically (backend + frontend)
|
||||
- Runs all backend tests (unit, integration, emulator)
|
||||
- Runs all frontend tests (Jest unit + Playwright E2E)
|
||||
- Takes ~5-10 minutes
|
||||
|
||||
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
|
||||
|
||||
### Testing in Production (for Agents)
|
||||
|
||||
When asked to "test it yourself in prod" or verify a fix in production:
|
||||
|
||||
```bash
|
||||
# Get admin token. claude-readonly@ can read all secrets (Pulumi:
|
||||
# infrastructure/modules/iam/claude_readonly_sa.py), so no gcloud login is needed.
|
||||
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com | cut -d',' -f1)
|
||||
|
||||
# Option 1: Use the debug template directly
|
||||
cd frontend && node e2e/helpers/debug-prod-template.mjs
|
||||
|
||||
# Option 2: Copy template for customization (gitignored)
|
||||
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
|
||||
# Edit the script, then run it
|
||||
node test-my-issue.local.mjs
|
||||
|
||||
# Option 3: Run existing production E2E tests
|
||||
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
|
||||
```
|
||||
|
||||
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
|
||||
|
||||
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- **All GCP changes via Pulumi PRs** - but CI does **not** run Pulumi (see the note in `.github/workflows/ci.yml`): a merged `infrastructure/` change is not live until someone runs `pulumi up` locally. Claude sessions have read-only ADC, so they need Andrew's `/gcp-admin` grant to apply. Ask for it early, and vet with a targeted `pulumi preview --diff` first
|
||||
- **`infrastructure/` is the shared platform IaC home**: the GCP project (`infrastructure/`) and the Nomad Karaoke Cloudflare account (`infrastructure/cloudflare/`, own Pulumi project + stack, see its README). Never hand-edit Cloudflare config that stack manages; app repos keep only app-specific stacks
|
||||
- **Run `pulumi up` locally BEFORE merging infra PRs** - CI only validates; it never applies. If you merge first, main and the live state drift until someone applies
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Stop and notify user on auth issues
|
||||
- **GCS ops on macOS: never use `gsutil -m`** — its fork-based multiprocessing segfaults
|
||||
on macOS (crash-report popups + the parent process hangs forever). Use `gcloud storage`
|
||||
instead (`gcloud storage cp/rm/rsync …`) — no fork, and faster. Plain `gsutil` (no `-m`)
|
||||
is safe but slower. `~/.boto` pins `parallel_process_count=1` as a safety net; don't remove it.
|
||||
|
||||
### Internationalization (i18n)
|
||||
|
||||
- All user-facing strings live in `frontend/messages/{locale}.json` (33 locales), NOT in components
|
||||
- Backend strings live in `backend/translations/{locale}.json` (currently 3 locales: en, es, de)
|
||||
- Components use `useTranslations('namespace')` from next-intl
|
||||
- Pages are under `frontend/app/[locale]/` — locale-aware routing (customer-facing: `/`, `/app/*`, `/order/*`, `/auth/*`, `/payment/*`, `/r/*`)
|
||||
- Non-locale routes at those same paths (e.g. `frontend/app/app/jobs/[[...slug]]/page.tsx`) are `<LocaleRedirect />` shims that redirect into `[locale]` — they are NOT English-only surfaces
|
||||
- The `/app/jobs/#/{id}/review`, `/instrumental`, and `/audio-edit` review UIs are fully multilingual (components under `components/lyrics-review/`, `components/instrumental-review/`, `components/audio-editor/` use `useTranslations` heavily)
|
||||
- Only `/admin/*` is English-only — it has no `[locale]/admin` counterpart and inherits `DefaultIntlProvider` from the root layout
|
||||
- Internal links use `Link` from `@/i18n/routing` (locale-aware)
|
||||
- After adding/changing English strings: `python frontend/scripts/translate.py --messages-dir frontend/messages --target all`
|
||||
- Translation uses Gemini 3.8 Flash via the **Gemini Developer API** (key: `GEMINI_API_KEY`, else Secret Manager `gemini-api-key` via your gcloud login — `frontend/scripts/gemini_client.py`) with GCS cache for efficiency. **Vertex AI is disabled in the GCP project — never use `vertexai=True`; backend code uses `backend/services/gemini_client.get_genai_client()`.**
|
||||
- CI validates all locale files have matching keys — **PR will fail if translations are missing**
|
||||
- Pre-commit hook auto-translates when en.json changes (enable: `git config core.hooksPath .githooks`)
|
||||
- Don't hardcode user-facing strings — add to `messages/en.json` and use `t('key')`
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke --account=claude-readonly@nomadkaraoke.iam.gserviceaccount.com)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
Since 2026-11 the only GCE encoding VM is the stopped on-demand fallback
|
||||
`encoding-worker-fallback-c2df` (c2d-highcpu-16, us-central1-f); the c4d Spot pair
|
||||
(`encoding-worker-a`/`-b`) was retired and deleted, and `primary_vm`/`secondary_vm` in
|
||||
`config/encoding-worker` are empty. The serving worker is `active_override_vm` (normally the
|
||||
AWS box below). Idle VMs stop after 5 min. CI's "blue-green" deploy now only runs when worker
|
||||
code changed (`infrastructure/encoding-worker/worker_code_paths.txt`, diffed against
|
||||
`validated_version`): it boots c2df, runs a test encode, records `validated_version` and stops
|
||||
it again, leaving the AWS override serving. Force it via Actions → CI → Run workflow →
|
||||
"Force a blue-green redeploy of the GCE encoding worker".
|
||||
```bash
|
||||
# Check health (includes wheel_version) on the GCE fallback.
|
||||
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
|
||||
# Restart the service (pick up new worker-side code after a deploy)
|
||||
gcloud compute ssh encoding-worker-fallback-c2df --zone=us-central1-f --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
```
|
||||
|
||||
For job stuck at `encoding` status, see [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md).
|
||||
|
||||
**AWS encoding worker** (`aws-encoding-worker-1`, EC2 c7a.4xlarge on-demand, us-east-2) is a pool
|
||||
candidate too (`provider: aws` in `encoding-worker-fallback-vms`): last resort by default,
|
||||
preferred when Firestore `config/encoding-worker.speed_rank_overrides = {"c7a": 5}`. Not
|
||||
blue-green deployed, idle-stops itself, shell via SSM, logs are on the box (journald). See
|
||||
`infrastructure/aws/README.md`.
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-PR Checklist
|
||||
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
-109
@@ -1,109 +0,0 @@
|
||||
# Contributing to karaoke-gen
|
||||
|
||||
Thanks for your interest in contributing! This guide focuses on getting a
|
||||
working **local development environment with no Google Cloud credentials or
|
||||
paid API keys** — everything runs against local emulators.
|
||||
|
||||
## What runs locally (no credentials needed)
|
||||
|
||||
| Component | Local story |
|
||||
|-----------|-------------|
|
||||
| FastAPI backend | Runs directly via uvicorn |
|
||||
| Firestore | Google's Firestore emulator |
|
||||
| Cloud Storage | [fake-gcs-server](https://github.com/fsouza/fake-gcs-server) (Docker) |
|
||||
| Background workers (audio, lyrics, screens, video…) | Run as local subprocesses / in-process tasks |
|
||||
| Audio separation | Local CPU/GPU via `audio-separator` (set `MODEL_DIR`; models download on first use) |
|
||||
| Lyrics transcription | Local Whisper (`poetry install --extras local-whisper`); AudioShake used only if you set a token |
|
||||
| Login (magic links) | Printed to the backend console instead of emailed |
|
||||
| Signed URLs (review audio, downloads, uploads) | Rewritten to plain emulator URLs |
|
||||
| Frontend (Next.js) | `npm run dev`, pointed at the local backend |
|
||||
|
||||
Things that do NOT run locally: payments (Stripe), production email
|
||||
(Postmark), YouTube/Drive/Dropbox distribution, and the `flacfetch`
|
||||
URL-download service — **jobs from URLs won't process locally; use file
|
||||
upload instead.**
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.12/3.13 + [Poetry](https://python-poetry.org/)
|
||||
- Node.js 20+ (for the frontend)
|
||||
- Docker (for the GCS emulator)
|
||||
- Java 21+ and the Google Cloud SDK with the Firestore emulator:
|
||||
`gcloud components install cloud-firestore-emulator`
|
||||
(no login/account needed — the emulator runs entirely offline)
|
||||
- ffmpeg (`brew install ffmpeg` / `apt install ffmpeg`)
|
||||
|
||||
## Backend quickstart
|
||||
|
||||
```bash
|
||||
poetry install # heavy: torch etc, ~10 min first time
|
||||
./scripts/run-backend-local.sh --with-emulators
|
||||
```
|
||||
|
||||
This starts the Firestore emulator (port 8080), fake-gcs-server (port 4443),
|
||||
creates the bucket, seeds a default theme (`scripts/seed-local-data.py`), and
|
||||
runs the backend at <http://localhost:8000> (API docs at `/docs`).
|
||||
|
||||
Optional API keys go in `.env` (see `.env.example`) — none are required to boot.
|
||||
|
||||
### Try it
|
||||
|
||||
```bash
|
||||
# Health
|
||||
curl http://localhost:8000/api/health
|
||||
|
||||
# Create a job from a local audio file (admin token is "local-dev-token")
|
||||
curl -X POST http://localhost:8000/api/jobs/upload \
|
||||
-H 'Authorization: Bearer local-dev-token' \
|
||||
-F "file=@/path/to/song.flac" -F "artist=Some Artist" -F "title=Some Song"
|
||||
|
||||
# Watch it process; when status hits awaiting_review, open the review UI
|
||||
curl http://localhost:8000/api/jobs/<job_id> -H 'Authorization: Bearer local-dev-token'
|
||||
```
|
||||
|
||||
For audio separation to run, set `MODEL_DIR` (e.g. `MODEL_DIR=./models`) —
|
||||
separation models (~2 GB) are downloaded on first use. Without it the job
|
||||
fails at the separation step with a clear error.
|
||||
|
||||
## Frontend quickstart
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
NEXT_PUBLIC_API_URL=http://localhost:8000 npx next dev
|
||||
```
|
||||
|
||||
Then open <http://localhost:3000>. To log in, enter any email address — the
|
||||
magic link is printed in the **backend** console (look for `EMAIL TO:`); open
|
||||
that link in your browser.
|
||||
|
||||
### Review UI without running the full pipeline
|
||||
|
||||
If you're working on the lyrics review UI specifically, there's a fixtures
|
||||
harness that serves real correction data through the review API shape without
|
||||
processing any audio:
|
||||
|
||||
```bash
|
||||
python scripts/review_test_fixtures.py # serves on :8765
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
make test # everything (backend + frontend)
|
||||
make test-backend # backend only (unit + emulator tests)
|
||||
```
|
||||
|
||||
Emulator-backed integration tests live in `backend/tests/emulator/` and use
|
||||
the same two emulators as the local dev setup.
|
||||
|
||||
## Pull requests
|
||||
|
||||
- Keep PRs focused; include tests for behavior changes.
|
||||
- Run `make test` before pushing — CI runs the same suite.
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes.
|
||||
- CI must pass before merge; a maintainer will review from there.
|
||||
|
||||
See `docs/DEVELOPMENT.md` for deeper docs (architecture, deployment,
|
||||
observability) — most of that file concerns the production GCP deployment,
|
||||
which contributors don't need.
|
||||
@@ -1,154 +0,0 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help install install-backend install-frontend build-frontend dev-install test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop local-stack-up local-stack-down test-local-stack eval-custom-lyrics
|
||||
|
||||
# Per-stage timeout in seconds (default: 10 minutes)
|
||||
# Prevents hangs from stale dev servers, broken emulators, etc.
|
||||
# Override with: make test TEST_TIMEOUT=300
|
||||
TEST_TIMEOUT ?= 600
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend unit tests (Jest)"
|
||||
@echo " make test-frontend-e2e - Run frontend E2E tests (Playwright, requires dev server)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make build-frontend - Build frontend and copy to Python package"
|
||||
@echo " make dev-install - Build frontend + pip install -e . (for local CLI testing)"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Install dependencies (only if needed)
|
||||
install-backend:
|
||||
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
|
||||
echo "=== Installing backend dependencies ==="; \
|
||||
poetry install; \
|
||||
fi
|
||||
|
||||
install-frontend:
|
||||
@if [ ! -d "frontend/node_modules" ]; then \
|
||||
echo "=== Installing frontend dependencies ==="; \
|
||||
cd frontend && npm install; \
|
||||
fi
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Build frontend and copy to Python package
|
||||
build-frontend: install-frontend
|
||||
@echo "=== Building frontend ==="
|
||||
cd frontend && npm run build
|
||||
@echo "=== Copying build to Python package ==="
|
||||
rm -rf karaoke_gen/nextjs_frontend/out
|
||||
mkdir -p karaoke_gen/nextjs_frontend/out
|
||||
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
|
||||
@# Drop Next.js RSC payloads (*.txt, ~66 MB across 33 locales): the CLI review
|
||||
@# server never serves them (only HTML entrypoints, /_next and /static), and they
|
||||
@# pushed the wheel past PyPI's 100 MiB per-file limit (2026-10-27..29).
|
||||
find karaoke_gen/nextjs_frontend/out -name '*.txt' -type f -delete
|
||||
@echo "✅ Frontend built and copied to karaoke_gen/nextjs_frontend/out/"
|
||||
|
||||
# Build frontend + install package in editable mode for local CLI testing
|
||||
# After running this, `karaoke-gen` CLI will use the freshly built frontend.
|
||||
dev-install: build-frontend
|
||||
@echo "=== Installing package in editable mode ==="
|
||||
pip install -e .
|
||||
@echo ""
|
||||
@echo "✅ Ready! You can now run karaoke-gen with the updated frontend."
|
||||
@echo " Example: karaoke-gen input.flac \"Artist\" \"Title\""
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit: install-backend
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
@echo "=== Running backend unit tests ==="
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e: install-backend
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
|
||||
export GOOGLE_CLOUD_PROJECT=test-project && \
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
timeout $(TEST_TIMEOUT) poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all backend tests (unit + emulator)
|
||||
test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend unit tests (Jest only - E2E requires manual setup)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend unit tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:unit
|
||||
|
||||
# Run frontend E2E tests (requires dev server - run manually, not in make test)
|
||||
test-frontend-e2e: install-frontend
|
||||
@echo "=== Running frontend E2E tests ==="
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
# Alias for test
|
||||
test-all: test
|
||||
|
||||
# Lint checks
|
||||
lint:
|
||||
@echo "=== Running lint checks ==="
|
||||
poetry run ruff check karaoke_gen/ backend/
|
||||
poetry run ruff format --check karaoke_gen/ backend/
|
||||
|
||||
# Start emulators for local development
|
||||
emulators-start:
|
||||
@echo "=== Starting GCP emulators ==="
|
||||
./scripts/start-emulators.sh
|
||||
|
||||
# Stop emulators
|
||||
emulators-stop:
|
||||
@echo "=== Stopping GCP emulators ==="
|
||||
./scripts/stop-emulators.sh
|
||||
|
||||
# Local E2E stack: real backend on emulators + fake workers (see docs/TESTING.md)
|
||||
local-stack-up:
|
||||
./scripts/local-stack.sh up --frontend
|
||||
|
||||
local-stack-down:
|
||||
./scripts/local-stack.sh down
|
||||
|
||||
# Full-journey Playwright suite against the local stack (starts it if needed)
|
||||
test-local-stack: install-frontend
|
||||
cd frontend && timeout $(TEST_TIMEOUT) npm run test:e2e:local-stack
|
||||
|
||||
# Clean up temporary files
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
|
||||
@echo "Cleaned up temporary files"
|
||||
|
||||
.PHONY: eval-custom-lyrics
|
||||
eval-custom-lyrics:
|
||||
cd backend && poetry run python -m backend.eval.custom_lyrics.run
|
||||
|
||||
@@ -1,680 +1,95 @@
|
||||
# Karaoke Generator 🎶 🎥 🚀
|
||||
# Karaoke Gen
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens.
|
||||
|
||||
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a **local CLI** (`karaoke-gen`) or **cloud-based CLI** (`karaoke-gen-remote`) that offloads processing to Google Cloud.
|
||||
## Overview
|
||||
|
||||
## ✨ Two Ways to Generate Karaoke
|
||||
Karaoke Gen is a comprehensive tool for creating high-quality karaoke videos. It automates the entire workflow:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
|
||||
1. **Download** audio and lyrics for a specified song
|
||||
2. **Separate** audio stems (vocals, instrumental)
|
||||
3. **Synchronize** lyrics with the audio
|
||||
4. **Generate** title and end screens
|
||||
5. **Combine** everything into a polished final video
|
||||
6. **Organize** and **share** the output files
|
||||
|
||||
```bash
|
||||
karaoke-gen "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Features
|
||||
|
||||
### Core Pipeline
|
||||
- **Audio Separation**: AI-powered vocal/instrumental separation using MDX and Demucs models
|
||||
- **Lyrics Transcription**: Word-level timestamps via AudioShake API
|
||||
- **Lyrics Correction**: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
|
||||
- **Human Review**: Interactive UI for correcting lyrics before final render
|
||||
- **Video Rendering**: High-quality 4K karaoke videos with customizable styles
|
||||
- **Multiple Outputs**: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
|
||||
|
||||
### Distribution Features
|
||||
- **YouTube Upload**: Automatic upload to your YouTube channel
|
||||
- **Dropbox Integration**: Organize output in brand-coded folders
|
||||
- **Google Drive**: Upload to public share folders
|
||||
- **Discord Notifications**: Webhook notifications on completion
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install karaoke-gen
|
||||
```
|
||||
|
||||
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
|
||||
|
||||
### Requirements
|
||||
- Python 3.10-3.13
|
||||
- FFmpeg
|
||||
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
|
||||
|
||||
### Transcription Provider Setup
|
||||
|
||||
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
|
||||
|
||||
#### Option 1: AudioShake (Recommended)
|
||||
Commercial service with high-quality transcription. Best for production use.
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
```
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
|
||||
```
|
||||
|
||||
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
|
||||
|
||||
#### Without Transcription (Instrumental Only)
|
||||
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
|
||||
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
This creates an instrumental-only karaoke video without lyrics overlay.
|
||||
|
||||
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Local CLI (`karaoke-gen`)
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Generate from local audio file
|
||||
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
|
||||
|
||||
# Search and download audio automatically
|
||||
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Process from YouTube URL
|
||||
# Generate a karaoke video from a YouTube URL
|
||||
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Or let it search YouTube for you
|
||||
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Remote Audio Separation (Optional)
|
||||
## Workflow Options
|
||||
|
||||
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
|
||||
Karaoke Gen supports different workflow options to fit your needs:
|
||||
|
||||
```bash
|
||||
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
|
||||
karaoke-gen "Artist" "Title"
|
||||
# Run only the preparation phase (download, separate stems, create title screens)
|
||||
karaoke-gen --prep-only "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Run only the finalisation phase (must be run in a directory prepared by the prep phase)
|
||||
karaoke-gen --finalise-only
|
||||
|
||||
# Skip automatic lyrics transcription/synchronization (for manual syncing)
|
||||
karaoke-gen --skip-transcription "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Skip audio separation (if you already have instrumental)
|
||||
karaoke-gen --skip-separation --existing-instrumental="path/to/instrumental.mp3" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Key Options
|
||||
## Advanced Features
|
||||
|
||||
### Audio Processing
|
||||
|
||||
```bash
|
||||
# Custom styling
|
||||
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# Skip video encoding (CDG/TXT only, faster)
|
||||
karaoke-gen --no-video --enable_cdg "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
# Full production run
|
||||
karaoke-gen \
|
||||
--style_params_json="./branding.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix="BRAND" \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./description.txt" \
|
||||
"Artist" "Title"
|
||||
# Specify custom audio separation models
|
||||
karaoke-gen --clean_instrumental_model="model_name.ckpt" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Full Options Reference
|
||||
### Lyrics Handling
|
||||
|
||||
```bash
|
||||
# Use a local lyrics file instead of fetching from online
|
||||
karaoke-gen --lyrics_file="path/to/lyrics.txt" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Adjust subtitle timing
|
||||
karaoke-gen --subtitle_offset_ms=500 "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Finalisation Options
|
||||
|
||||
```bash
|
||||
# Enable CDG ZIP generation
|
||||
karaoke-gen --enable_cdg --style_params_json="path/to/style.json" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Enable TXT ZIP generation
|
||||
karaoke-gen --enable_txt "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Upload to YouTube
|
||||
karaoke-gen --youtube_client_secrets_file="path/to/client_secret.json" --youtube_description_file="path/to/description.txt" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Organize files with brand code
|
||||
karaoke-gen --brand_prefix="BRAND" --organised_dir="path/to/Tracks-Organized" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
## Full Command Reference
|
||||
|
||||
For a complete list of options:
|
||||
|
||||
```bash
|
||||
karaoke-gen --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Remote CLI (`karaoke-gen-remote`)
|
||||
|
||||
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Set the backend URL:**
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend
|
||||
```
|
||||
|
||||
2. **Authenticate with Google Cloud:**
|
||||
```bash
|
||||
gcloud auth login
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Submit a job
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
|
||||
# The CLI will:
|
||||
# 1. Upload your audio file
|
||||
# 2. Monitor processing progress
|
||||
# 3. Open lyrics review UI when ready
|
||||
# 4. Prompt for instrumental selection
|
||||
# 5. Download all outputs when complete
|
||||
```
|
||||
|
||||
### Job Management
|
||||
|
||||
```bash
|
||||
# List all jobs
|
||||
karaoke-gen-remote --list
|
||||
|
||||
# Resume monitoring an existing job
|
||||
karaoke-gen-remote --resume abc12345
|
||||
|
||||
# Cancel a running job
|
||||
karaoke-gen-remote --cancel abc12345
|
||||
|
||||
# Delete a job and its files
|
||||
karaoke-gen-remote --delete abc12345
|
||||
```
|
||||
|
||||
### Full Production Run
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote \
|
||||
--style_params_json="./karaoke-styles.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix=NOMAD \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./youtube-description.txt" \
|
||||
./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `KARAOKE_GEN_URL` | Backend service URL | Required |
|
||||
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
|
||||
| `POLL_INTERVAL` | Seconds between status polls | `5` |
|
||||
|
||||
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
|
||||
|
||||
**For admin access:**
|
||||
```bash
|
||||
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
|
||||
```
|
||||
|
||||
The token must match one of the tokens configured in the backend's `ADMIN_TOKENS` environment variable.
|
||||
|
||||
### Non-Interactive Mode
|
||||
|
||||
For automated/CI usage:
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote -y ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
The `-y` flag auto-accepts default corrections and selects clean instrumental.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Style Configuration
|
||||
|
||||
Create a `styles.json` file to customize the karaoke video appearance:
|
||||
|
||||
```json
|
||||
{
|
||||
"intro": {
|
||||
"video_duration": 5,
|
||||
"background_image": "/path/to/title-background.png",
|
||||
"font": "/path/to/Font.ttf",
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/path/to/karaoke-background.png",
|
||||
"font_path": "/path/to/Font.ttf"
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/path/to/end-background.png"
|
||||
},
|
||||
"cdg": {
|
||||
"font_path": "/path/to/Font.ttf",
|
||||
"instrumental_background": "/path/to/cdg-background.png"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When using `karaoke-gen-remote`, all referenced files are automatically uploaded with your job.
|
||||
|
||||
---
|
||||
|
||||
## 📤 Output Files
|
||||
|
||||
A completed job produces:
|
||||
|
||||
```
|
||||
BRAND-1234 - Artist - Title/
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
|
||||
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
|
||||
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
|
||||
├── Artist - Title (Karaoke).cdg # Individual CDG
|
||||
├── Artist - Title (Karaoke).mp3 # Karaoke audio
|
||||
├── Artist - Title (Karaoke).lrc # LRC lyrics
|
||||
├── Artist - Title (Karaoke).ass # ASS subtitles
|
||||
├── Artist - Title (Title).mov # Title screen video
|
||||
├── Artist - Title (End).mov # End screen video
|
||||
├── Artist - Title (Instrumental...).flac # Clean instrumental
|
||||
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
|
||||
└── stems/ # All audio stems
|
||||
├── ...Vocals....flac
|
||||
├── ...Bass....flac
|
||||
├── ...Drums....flac
|
||||
└── ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Deploy Your Own Backend
|
||||
|
||||
The cloud backend runs on Google Cloud Platform using:
|
||||
- **Cloud Run**: Serverless API hosting
|
||||
- **Firestore**: Job state management
|
||||
- **Cloud Storage**: File uploads and outputs
|
||||
- **Modal.com**: GPU-accelerated audio separation
|
||||
- **AudioShake**: Lyrics transcription API
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Google Cloud account with billing enabled
|
||||
- [Pulumi CLI](https://www.pulumi.com/docs/install/)
|
||||
- Modal.com account (for audio separation)
|
||||
- AudioShake API key
|
||||
|
||||
### Infrastructure Setup
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Login to Pulumi
|
||||
pulumi login
|
||||
|
||||
# Create a stack
|
||||
pulumi stack init prod
|
||||
|
||||
# Configure GCP project
|
||||
pulumi config set gcp:project your-project-id
|
||||
pulumi config set gcp:region us-central1
|
||||
|
||||
# Deploy infrastructure
|
||||
pulumi up
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Firestore database
|
||||
- Cloud Storage bucket
|
||||
- Artifact Registry
|
||||
- Service account with IAM roles
|
||||
- Secret Manager secrets (you add values)
|
||||
|
||||
### Add Secret Values
|
||||
|
||||
```bash
|
||||
# AudioShake API key
|
||||
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
|
||||
# Genius API key
|
||||
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
|
||||
|
||||
# Modal API URL
|
||||
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
|
||||
|
||||
# YouTube OAuth credentials (JSON)
|
||||
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
|
||||
|
||||
# Dropbox OAuth credentials (JSON)
|
||||
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
|
||||
|
||||
# Google Drive service account (JSON)
|
||||
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
```
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
|
||||
See `.github/workflows/ci.yml` for the full deployment workflow.
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://your-backend.run.app"
|
||||
karaoke-gen-remote ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Backend API Reference
|
||||
|
||||
The backend exposes a REST API for job management.
|
||||
|
||||
### Job Submission
|
||||
|
||||
**POST** `/api/jobs/upload`
|
||||
|
||||
Submit a new karaoke generation job with audio file and options.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/upload" \
|
||||
-F "file=@song.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo" \
|
||||
-F "enable_cdg=true" \
|
||||
-F "enable_txt=true" \
|
||||
-F "brand_prefix=NOMAD" \
|
||||
-F "style_params=@styles.json" \
|
||||
-F "style_karaoke_background=@background.png"
|
||||
```
|
||||
|
||||
### Job Status
|
||||
|
||||
**GET** `/api/jobs/{job_id}`
|
||||
|
||||
Get job status and details.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs/abc12345"
|
||||
```
|
||||
|
||||
### List Jobs
|
||||
|
||||
**GET** `/api/jobs`
|
||||
|
||||
List all jobs with optional status filter.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs?status=complete&limit=10"
|
||||
```
|
||||
|
||||
### Cancel Job
|
||||
|
||||
**POST** `/api/jobs/{job_id}/cancel`
|
||||
|
||||
Cancel a running job.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "User cancelled"}'
|
||||
```
|
||||
|
||||
### Delete Job
|
||||
|
||||
**DELETE** `/api/jobs/{job_id}`
|
||||
|
||||
Delete a job and its files.
|
||||
|
||||
```bash
|
||||
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
|
||||
```
|
||||
|
||||
### Lyrics Review
|
||||
|
||||
**GET** `/api/review/{job_id}/correction-data`
|
||||
|
||||
Get correction data for lyrics review.
|
||||
|
||||
**POST** `/api/review/{job_id}/complete`
|
||||
|
||||
Submit corrected lyrics and trigger video rendering.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
**GET** `/api/jobs/{job_id}/instrumental-options`
|
||||
|
||||
Get available instrumental options.
|
||||
|
||||
**POST** `/api/jobs/{job_id}/select-instrumental`
|
||||
|
||||
Submit instrumental selection (clean or with_backing).
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"selection": "clean"}'
|
||||
```
|
||||
|
||||
### Download Files
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download-urls`
|
||||
|
||||
Get download URLs for all output files.
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download/{category}/{file_key}`
|
||||
|
||||
Stream download a specific file.
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET** `/api/health`
|
||||
|
||||
Check backend health status.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "No suitable files found for processing"
|
||||
|
||||
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
|
||||
|
||||
**Most common cause:** No transcription provider configured.
|
||||
|
||||
**Quick fix:**
|
||||
1. Check if transcription providers are configured:
|
||||
```bash
|
||||
echo $AUDIOSHAKE_API_TOKEN
|
||||
echo $RUNPOD_API_KEY
|
||||
```
|
||||
|
||||
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
|
||||
|
||||
3. Or use `--skip-lyrics` for instrumental-only karaoke:
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
**Other causes:**
|
||||
- Invalid API credentials - verify your tokens are correct and active
|
||||
- API service unavailable - check service status pages
|
||||
- Network connectivity issues - ensure you can reach the API endpoints
|
||||
- Transcription timeout - try again or use a different provider
|
||||
|
||||
### Transcription Fails Silently
|
||||
|
||||
If karaoke-gen runs without errors but produces no synchronized lyrics:
|
||||
|
||||
1. **Check logs** - Run with `--log_level debug` for detailed output:
|
||||
```bash
|
||||
karaoke-gen --log_level debug "Artist" "Title"
|
||||
```
|
||||
|
||||
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
|
||||
```bash
|
||||
# Check if set
|
||||
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
|
||||
|
||||
# Set in current session
|
||||
export AUDIOSHAKE_API_TOKEN="your_token"
|
||||
```
|
||||
|
||||
3. **Test API connectivity** - Verify you can reach the transcription service
|
||||
|
||||
### "No lyrics found from any source"
|
||||
|
||||
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
|
||||
|
||||
**To fix:**
|
||||
- Set `GENIUS_API_TOKEN` for Genius lyrics
|
||||
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
|
||||
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
|
||||
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
If the output video has quality problems:
|
||||
- Ensure FFmpeg is properly installed: `ffmpeg -version`
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest tests/ backend/tests/ -v
|
||||
|
||||
# Run only unit tests
|
||||
pytest tests/unit/ -v
|
||||
|
||||
# Run with coverage
|
||||
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── karaoke_gen/ # Core CLI package
|
||||
│ ├── utils/
|
||||
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
|
||||
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
|
||||
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
|
||||
│ └── style_loader.py # Unified style configuration
|
||||
├── backend/ # Cloud backend (FastAPI)
|
||||
│ ├── api/routes/ # API endpoints
|
||||
│ ├── workers/ # Background processing workers
|
||||
│ └── services/ # Business logic services
|
||||
├── infrastructure/ # Pulumi IaC for GCP
|
||||
├── docs/ # Documentation
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) — the whole
|
||||
backend runs locally against emulators with no Google Cloud credentials or
|
||||
paid API keys required.
|
||||
@@ -1,75 +0,0 @@
|
||||
[
|
||||
{
|
||||
"test_name": "Sound-alike error (wurld -> world)",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "ReplaceWord",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w1"
|
||||
],
|
||||
"replacement_text": "world",
|
||||
"confidence": 0.75,
|
||||
"reason": "Sound-alike error. Reference suggests: 'world'. The transcription 'wurld' is a phonetic misspelling or sound-alike of the word 'world' which appears in both reference sources (Genius and LRCLIB).",
|
||||
"gap_category": "GapCategory.SOUND_ALIKE",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Punctuation difference",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w2"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The gap text 'Hello' followed by the context 'is it me you're looking for' matches the Genius reference lyrics exactly, excluding punctuation. According to the guidelines, if any reference source matches the transcription text (ignoring punctuation), it should be classified as NO_ERROR.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Extra filler word",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "DeleteWord",
|
||||
"word_id": "w3",
|
||||
"word_ids": null,
|
||||
"replacement_text": null,
|
||||
"confidence": 0.8,
|
||||
"reason": "Extra filler word 'And' at sentence start not in reference. The transcription includes the filler word 'And' at the beginning of the phrase, which is not present in the reference lyrics. This matches the definition of EXTRA_WORDS.",
|
||||
"gap_category": "GapCategory.EXTRA_WORDS",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Complex multi-word error",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w7",
|
||||
"w8",
|
||||
"w9",
|
||||
"w10"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The transcription text 'gonna give you up' matches the Genius reference lyrics exactly when ignoring punctuation and capitalization. In context, the full phrase 'never gonna give you up' is correctly transcribed and aligns perfectly with the known lyrics and title of the song.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,20 +0,0 @@
|
||||
[run]
|
||||
source = backend
|
||||
omit =
|
||||
backend/tests/*
|
||||
backend/validate.py
|
||||
backend/tests/emulator/*
|
||||
# Workers that require external APIs are tested via integration tests
|
||||
backend/workers/screens_worker.py
|
||||
backend/workers/video_worker.py
|
||||
|
||||
[report]
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
def __repr__
|
||||
raise NotImplementedError
|
||||
if __name__ == .__main__.:
|
||||
pass
|
||||
|
||||
show_missing = True
|
||||
|
||||
@@ -1,39 +0,0 @@
|
||||
# Ignore Python cache and virtual environments
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Environment files
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Temporary files
|
||||
*.log
|
||||
temp/
|
||||
tmp/
|
||||
|
||||
# Google Cloud
|
||||
.gcloud/
|
||||
*.json
|
||||
# Allow i18n translation files
|
||||
!translations/*.json
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
# Fast application image - uses pre-built base with all dependencies
|
||||
#
|
||||
# The base image (karaoke-backend-base) contains:
|
||||
# - Python 3.11, ffmpeg, sox, system deps
|
||||
# - All pip dependencies from requirements-frozen.txt (NO code)
|
||||
#
|
||||
# This Dockerfile adds the actual application code on top.
|
||||
# Build time: ~1-2 minutes (deps already cached in base)
|
||||
#
|
||||
# If the base image doesn't exist yet, build it first:
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Use pre-built base image with all dependencies
|
||||
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
|
||||
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
# Set working directory (should already exist from base)
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend code (this is the only thing that changes frequently)
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
# GPU images (MODEL_DIR set in base): use --no-deps to preserve pinned torch cu126
|
||||
# and audio-separator[gpu]. Without this, pip upgrades torch to cu130 (breaks CUDA
|
||||
# on Cloud Run L4 drivers) and reinstalls audio-separator[cpu] (breaks GPU ONNX).
|
||||
# CPU images: normal install resolves all deps from pyproject.toml.
|
||||
RUN if [ -n "$MODEL_DIR" ]; then \
|
||||
pip install --no-cache-dir --no-deps -e .; \
|
||||
else \
|
||||
pip install --no-cache-dir -e .; \
|
||||
fi
|
||||
|
||||
# Precompile bytecode so lazy imports never write .pyc at runtime: a cold
|
||||
# import that compiles + writes on the event loop has stalled requests 10-15s.
|
||||
RUN python -m compileall -q /app/backend /app/karaoke_gen
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PORT=8080
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Run uvicorn directly
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
|
||||
@@ -1,106 +0,0 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock or this Dockerfile changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 3. pip install (the slow part - cached unless pyproject.toml changes)
|
||||
# 4. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 5. spaCy model download (cached unless pip install layer changes)
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install system dependencies required by karaoke-gen
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
chromium \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds, so partial
|
||||
# dependency updates only download changed packages instead of everything
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Pre-build matplotlib's font cache at BUILD time. Without it, the first
|
||||
# `import matplotlib.pyplot` on every new Cloud Run instance scanned all the
|
||||
# Noto fonts above (20-40s of GIL-holding CPU) and froze the API's event loop
|
||||
# — prod logged "Matplotlib is building the font cache" in 17 of 28
|
||||
# "servers unavailable" banner episodes (2026-10-03 analysis). The cache lives
|
||||
# in MPLCONFIGDIR so it ships in the image instead of the in-memory $HOME.
|
||||
ENV MPLCONFIGDIR=/opt/mplconfig
|
||||
RUN mkdir -p /opt/mplconfig && \
|
||||
python -c "import matplotlib; matplotlib.use('Agg'); import matplotlib.font_manager as fm; print(len(fm.fontManager.ttflist), 'fonts cached')" && \
|
||||
ls /opt/mplconfig/fontlist-*.json && \
|
||||
chmod -R a+rwX /opt/mplconfig
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-base"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -1,144 +0,0 @@
|
||||
# GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models
|
||||
# This image is rebuilt only when deps or models change (infrequently).
|
||||
#
|
||||
# Build with: gcloud builds submit --config=backend/cloudbuild-gpu-base.yaml --project=nomadkaraoke --timeout=3600s
|
||||
# Or manually: docker build -f backend/Dockerfile.gpu-base -t karaoke-backend-gpu-base .
|
||||
#
|
||||
# Layer caching strategy:
|
||||
# 1. System deps + fonts + ffmpeg (rarely change, always cached)
|
||||
# 2. CUDA library path configuration
|
||||
# 3. pyproject.toml + package stubs (changes with dep updates)
|
||||
# 4. pip install karaoke-gen deps + audio-separator[gpu] (slow, cached unless pyproject.toml changes)
|
||||
# 5. Source code copy (changes every rebuild, but AFTER deps are installed)
|
||||
# 6. spaCy model download (cached unless pip install layer changes)
|
||||
# 7. Pre-downloaded ensemble preset models (~1-1.5 GB, cached unless pip layer changes)
|
||||
|
||||
FROM nvidia/cuda:12.6.3-runtime-ubuntu22.04
|
||||
|
||||
# Prevent interactive prompts during package installation
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python 3.11 from deadsnakes PPA and system dependencies
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
software-properties-common \
|
||||
&& add-apt-repository -y ppa:deadsnakes/ppa \
|
||||
&& apt-get update && apt-get install -y --no-install-recommends \
|
||||
# Python 3.11
|
||||
python3.11 \
|
||||
python3.11-dev \
|
||||
python3.11-venv \
|
||||
# Audio libraries
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
# Build tools
|
||||
build-essential \
|
||||
# Utilities
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& python3.11 --version
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install FFmpeg from Ubuntu package
|
||||
# The audio worker only uses FFmpeg for format conversion and padding, not
|
||||
# encoding-intensive tasks, so the Debian package is sufficient. The static
|
||||
# build from johnvansickle.com is used for the encoding worker instead.
|
||||
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Set Python 3.11 as default and install pip
|
||||
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 \
|
||||
&& update-alternatives --install /usr/bin/python python /usr/bin/python3.11 1 \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python3.11 \
|
||||
&& python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
|
||||
|
||||
# Set up CUDA library paths so GPU-accelerated packages can find CUDA libs at runtime
|
||||
RUN echo '/usr/local/cuda/lib64' >> /etc/ld.so.conf.d/cuda.conf && ldconfig
|
||||
|
||||
# Environment configuration for CUDA and Python
|
||||
ENV LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
|
||||
PATH=/usr/local/cuda/bin:$PATH \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
MODEL_DIR=/models
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy ONLY dependency specification files first (changes less often than source code)
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
|
||||
# Create minimal package stubs so pip can resolve the editable install
|
||||
# pip install -e . needs package directories to exist but doesn't validate content
|
||||
RUN mkdir -p /app/karaoke_gen /app/lyrics_transcriber_temp /app/backend && \
|
||||
touch /app/karaoke_gen/__init__.py /app/lyrics_transcriber_temp/__init__.py /app/backend/__init__.py
|
||||
|
||||
# Install all karaoke-gen Python dependencies (the slow part)
|
||||
# BuildKit cache mount preserves downloaded wheels across builds
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install --upgrade pip && \
|
||||
pip install -e .
|
||||
|
||||
# Install audio-separator with GPU support (PyTorch + CUDA + onnxruntime-gpu)
|
||||
# This is separate from pip install -e . to keep layer caching granular
|
||||
#
|
||||
# IMPORTANT: Install PyTorch cu126 FIRST, then audio-separator[gpu].
|
||||
# The default audio-separator[gpu] pulls PyTorch cu130, but Cloud Run L4 GPUs
|
||||
# have NVIDIA driver 570 which only supports up to CUDA 12.8. PyTorch cu130
|
||||
# fails with "NVIDIA driver is too old". PyTorch cu126 works with 12.8 drivers.
|
||||
# Installing torch first means audio-separator[gpu] sees it as already satisfied.
|
||||
#
|
||||
# PINNED to an exact version: an unpinned install silently picked up 0.47.0 on the
|
||||
# 2026-10-17 rebuild and separation went from ~5 to ~10-16 min. Cause: 0.47 (#299) made
|
||||
# RoFormer `overlap` a window count defaulting to each model YAML's num_overlap (4-8 for
|
||||
# the karaoke models = 4-6x the forward passes of 0.44's fixed 8 s step). karaoke_gen's
|
||||
# audio_processor now passes overlap=2 on >= 0.47 (SEPARATOR_MDXC_PARAMS). Time a real
|
||||
# job before bumping this, and read the audio-separator release notes for compute changes.
|
||||
ARG AUDIO_SEPARATOR_VERSION=0.47.1
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
pip install torch==2.6.0+cu126 torchvision==0.21.0+cu126 --index-url https://download.pytorch.org/whl/cu126 && \
|
||||
pip install "audio-separator[gpu]==${AUDIO_SEPARATOR_VERSION}"
|
||||
|
||||
# Now copy actual source code (AFTER deps installed, so code changes don't
|
||||
# invalidate the expensive pip install layer in registry cache)
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# Must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Create models directory and pre-download ensemble preset models
|
||||
# Baking models into the image eliminates cold-start model download latency.
|
||||
# Total model size: ~1-1.5 GB for instrumental_clean + karaoke presets.
|
||||
RUN mkdir -p /models
|
||||
COPY backend/scripts/download_models.py /tmp/download_models.py
|
||||
RUN python /tmp/download_models.py /models && \
|
||||
rm /tmp/download_models.py && \
|
||||
ls -lh /models/
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-gpu-base"
|
||||
LABEL org.opencontainers.image.description="GPU base image with CUDA, PyTorch, audio-separator, and pre-downloaded models for karaoke-backend audio worker"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -1,242 +0,0 @@
|
||||
# Backend Development Setup
|
||||
|
||||
## One-Time Setup
|
||||
|
||||
### 1. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
### 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
|
||||
|
||||
```bash
|
||||
# Backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
|
||||
# Install lyrics_transcriber submodule
|
||||
pip install -e lyrics_transcriber_local/
|
||||
|
||||
# Install Google API client (for YouTube upload)
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
```
|
||||
|
||||
### 4. Add Shell Alias (Optional)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then you can just run:
|
||||
```bash
|
||||
backend
|
||||
# Now you're in the project directory with venv activated!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Daily Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Or if you set up the alias:
|
||||
backend
|
||||
```
|
||||
|
||||
### After Making Changes
|
||||
|
||||
```bash
|
||||
# 1. Run full validation (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 2. If validation passes, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Options
|
||||
|
||||
### Full Validation (Recommended - Requires venv)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ **Import errors** (like the processing_service issue!)
|
||||
- ✅ Configuration issues
|
||||
- ✅ FastAPI app creation problems
|
||||
- ✅ Missing dependencies
|
||||
|
||||
**This would have caught the ModuleNotFoundError!**
|
||||
|
||||
### Quick Check (No venv needed)
|
||||
|
||||
```bash
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ Missing files
|
||||
- ✅ Some obvious import issues
|
||||
|
||||
**Good for quick checks, but won't catch all import errors**
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Session
|
||||
|
||||
```bash
|
||||
# 1. Activate environment
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
# ✅ All validations passed!
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
|
||||
# 5. Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What About karaoke_gen Imports?
|
||||
|
||||
The backend imports from `karaoke_gen` (the CLI package):
|
||||
|
||||
```python
|
||||
from karaoke_gen.karaoke_gen import KaraokePrep
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
### Option 1: Install in development mode (Recommended)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
This makes `karaoke_gen` available in the venv without copying files.
|
||||
|
||||
### Option 2: Use system Python
|
||||
|
||||
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
|
||||
|
||||
---
|
||||
|
||||
## Running the Backend Locally
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Set required environment variables
|
||||
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
|
||||
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Optional: Point to local credentials
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
|
||||
# Run the server
|
||||
cd backend
|
||||
uvicorn main:app --reload --port 8080
|
||||
```
|
||||
|
||||
Visit: http://localhost:8080/api/health
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Validation passes locally but fails in Cloud Run
|
||||
|
||||
**Check:**
|
||||
1. Is `requirements.txt` up to date?
|
||||
2. Did you add a new dependency without adding it to `requirements.txt`?
|
||||
3. Run: `pip freeze | grep <package-name>`
|
||||
|
||||
---
|
||||
|
||||
## Updating Dependencies
|
||||
|
||||
When you add new packages:
|
||||
|
||||
```bash
|
||||
# 1. Install in venv
|
||||
source backend/venv/bin/activate
|
||||
pip install new-package
|
||||
|
||||
# 2. Update requirements.txt
|
||||
pip freeze > backend/requirements.txt
|
||||
|
||||
# 3. Or manually add just the new package:
|
||||
echo "new-package==1.2.3" >> backend/requirements.txt
|
||||
|
||||
# 4. Validate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 5. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Best practice workflow:**
|
||||
|
||||
1. **One-time setup:** Create venv and install dependencies
|
||||
2. **Daily:** Activate venv when working on backend
|
||||
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
|
||||
4. **Deploy:** Use `./scripts/deploy.sh`
|
||||
|
||||
**The import error would have been caught if you ran validation in a venv!** ✅
|
||||
Whitespace-only changes.
Whitespace-only changes.
@@ -1,360 +0,0 @@
|
||||
"""
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable, Union
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def generate_review_token() -> str:
|
||||
"""Generate a cryptographically secure review token."""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def get_review_token_expiry(hours: int = 24) -> datetime:
|
||||
"""Get expiry time for a review token."""
|
||||
return datetime.now(UTC) + timedelta(hours=hours)
|
||||
|
||||
|
||||
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
||||
security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
async def get_token_from_request(
|
||||
request: Request,
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Extract access token from request.
|
||||
|
||||
Supports three methods (in priority order):
|
||||
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
||||
2. Authorization header: Authorization: Bearer <token>
|
||||
3. Query parameter: ?token=<token> (for download links)
|
||||
|
||||
Returns:
|
||||
Token string or None
|
||||
"""
|
||||
# Check X-Admin-Token header first (used by Cloud Tasks)
|
||||
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
||||
admin_token = request.headers.get("X-Admin-Token")
|
||||
if admin_token:
|
||||
return admin_token
|
||||
|
||||
# Try Authorization header
|
||||
if credentials:
|
||||
return credentials.credentials
|
||||
|
||||
# Try query parameter
|
||||
if token:
|
||||
return token
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def require_auth(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication context including:
|
||||
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
||||
- user_email: Email of authenticated user (if session/API key auth)
|
||||
- is_admin: Whether user has admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token using the full method
|
||||
# Sync Firestore reads: run off the event loop (every authed request passes here).
|
||||
auth_result = await asyncio.to_thread(auth_service.validate_token_full, token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
|
||||
if not auth_result.is_valid:
|
||||
# Log auth failure with request_id for correlation
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
"auth_failed",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"audit_type": "auth_event",
|
||||
"auth_message": auth_result.message,
|
||||
"token_provided": bool(token_str),
|
||||
"token_length": len(token_str) if token_str else 0,
|
||||
"auth_header_present": bool(auth_header),
|
||||
}
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# If token has a tenant_id, override the hostname-based tenant detection.
|
||||
# This allows tenant-scoped tokens to enforce tenant context regardless of domain.
|
||||
# We must also load tenant_config so downstream handlers (e.g. job creation)
|
||||
# that rely on tenant_config for setting tenant_id on jobs work correctly.
|
||||
if auth_result.tenant_id:
|
||||
request.state.tenant_id = auth_result.tenant_id
|
||||
if not getattr(request.state, "tenant_config", None):
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = await asyncio.to_thread(tenant_service.get_tenant_config, auth_result.tenant_id)
|
||||
if tenant_config and tenant_config.is_active:
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"user_email": auth_result.user_email,
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"tenant_id": auth_result.tenant_id,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
async def require_auth_legacy(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Legacy authentication dependency that returns tuple.
|
||||
|
||||
DEPRECATED: Use require_auth which returns AuthResult instead.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
"""
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = await asyncio.to_thread(auth_service.validate_token, token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
Admin access is granted if:
|
||||
- Using an admin token from ADMIN_TOKENS env var
|
||||
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
||||
- User role is set to ADMIN in database
|
||||
|
||||
Returns:
|
||||
AuthResult with admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
if not auth_result.is_admin:
|
||||
logger.warning(
|
||||
f"Admin access denied for {auth_result.user_type} user "
|
||||
f"(email: {auth_result.user_email or 'unknown'})"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
def optional_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Optional[Tuple[str, UserType, int]]:
|
||||
"""
|
||||
Optional authentication - doesn't fail if no token provided.
|
||||
|
||||
Useful for endpoints that have different behavior for authenticated users.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses) if authenticated, None otherwise
|
||||
"""
|
||||
# Get token
|
||||
token_str = None
|
||||
if credentials:
|
||||
token_str = credentials.credentials
|
||||
elif token:
|
||||
token_str = token
|
||||
|
||||
if not token_str:
|
||||
return None
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
logger.debug(f"Optional auth failed: {message}")
|
||||
return None
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
async def require_review_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = await asyncio.to_thread(auth_service.validate_token_full, full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Admin, owner, or a member of the job's tenant portal
|
||||
# (backend/services/job_access.py).
|
||||
from backend.services.job_access import can_access_job
|
||||
|
||||
if not await asyncio.to_thread(
|
||||
can_access_job, job, auth_result.user_email, bool(auth_result.is_admin)
|
||||
):
|
||||
logger.warning(
|
||||
f"Review access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
# Who is reviewing — handlers record it on first open
|
||||
# (state_data.review_started_by, see JobManager.record_review_started).
|
||||
request.state.review_auth_email = auth_result.user_email
|
||||
request.state.review_auth_is_admin = bool(auth_result.is_admin)
|
||||
return job_id, "full"
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job = await asyncio.to_thread(job_manager.get_job, job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
if job.review_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
# Default instance for most review endpoints
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
Whitespace-only changes.
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -1,354 +0,0 @@
|
||||
"""
|
||||
OAuth credential management API routes.
|
||||
|
||||
Provides endpoints for:
|
||||
1. Checking credential status
|
||||
2. Device authorization flow for re-authentication
|
||||
3. Credential validation before job submission
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
get_credential_manager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Request/Response Models
|
||||
# ===========================================================================
|
||||
|
||||
class CredentialStatusResponse(BaseModel):
|
||||
"""Response model for credential status."""
|
||||
service: str
|
||||
status: str
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
|
||||
class AllCredentialsStatusResponse(BaseModel):
|
||||
"""Response model for all credentials status."""
|
||||
youtube: CredentialStatusResponse
|
||||
gdrive: CredentialStatusResponse
|
||||
dropbox: CredentialStatusResponse
|
||||
all_valid: bool
|
||||
services_needing_auth: list[str]
|
||||
|
||||
|
||||
class DeviceAuthStartRequest(BaseModel):
|
||||
"""Request to start device authorization flow."""
|
||||
client_id: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
client_secret: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
|
||||
|
||||
class DeviceAuthStartResponse(BaseModel):
|
||||
"""Response with device authorization info."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
instructions: str
|
||||
|
||||
|
||||
class DeviceAuthPollResponse(BaseModel):
|
||||
"""Response from polling device authorization."""
|
||||
status: str # "pending", "complete", "expired", "error"
|
||||
message: Optional[str] = None
|
||||
credentials_saved: bool = False
|
||||
|
||||
|
||||
class CredentialValidationRequest(BaseModel):
|
||||
"""Request to validate credentials for specific services."""
|
||||
youtube: bool = False
|
||||
gdrive: bool = False
|
||||
dropbox: bool = False
|
||||
|
||||
|
||||
class CredentialValidationResponse(BaseModel):
|
||||
"""Response from credential validation."""
|
||||
valid: bool
|
||||
invalid_services: list[str]
|
||||
message: str
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.get("/status", response_model=AllCredentialsStatusResponse)
|
||||
def get_credentials_status():
|
||||
"""
|
||||
Get the status of all OAuth credentials.
|
||||
|
||||
Returns the validation status for YouTube, Google Drive, and Dropbox
|
||||
credentials, including whether they are valid, expired, or need
|
||||
re-authorization.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
def to_response(result: CredentialCheckResult) -> CredentialStatusResponse:
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
services_needing_auth = [
|
||||
name for name, result in results.items()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED, CredentialStatus.NOT_CONFIGURED)
|
||||
]
|
||||
|
||||
return AllCredentialsStatusResponse(
|
||||
youtube=to_response(results["youtube"]),
|
||||
gdrive=to_response(results["gdrive"]),
|
||||
dropbox=to_response(results["dropbox"]),
|
||||
all_valid=len(services_needing_auth) == 0,
|
||||
services_needing_auth=services_needing_auth
|
||||
)
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
def get_service_credential_status(service: str, request: Request):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
manager = get_credential_manager()
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
result = manager.check_gdrive_credentials()
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=t(locale, "auth.unknownService", service=service))
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
def validate_credentials(request: CredentialValidationRequest, http_request: Request):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=t(locale, "auth.invalidCredentials", services=", ".join(invalid_services))
|
||||
)
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message=t(locale, "auth.validCredentials")
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Device Authorization Flow Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=t(locale, "auth.youtubeAuthInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.youtubeAuthStartFailed", error=str(e)))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
def poll_youtube_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for YouTube device authorization completion.
|
||||
|
||||
Call this endpoint at the interval specified in the device auth
|
||||
start response. Keep polling while status is "pending".
|
||||
|
||||
Status values:
|
||||
- pending: User has not yet authorized
|
||||
- complete: Authorization successful, credentials saved
|
||||
- expired: Device code expired, start new flow
|
||||
- error: An error occurred
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("youtube", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None, http_request: Request = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request) if http_request else "en"
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=t(locale, "auth.gdriveLinkInstructions", verification_url=device_info.verification_url, user_code=device_info.user_code)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=t(locale, "auth.gdriveAuthStartFailed", error=str(e)))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
def poll_gdrive_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for Google Drive device authorization completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("gdrive", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Alert Testing
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
def test_credential_alert(background_tasks: BackgroundTasks, request: Request):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=t(locale, "auth.discordAlertNotConfigured")
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
# Create a fake invalid result for testing
|
||||
from backend.services.credential_manager import CredentialCheckResult
|
||||
test_results = [
|
||||
CredentialCheckResult(
|
||||
service="test",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="This is a test alert",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
background_tasks.add_task(
|
||||
manager.send_credential_alert,
|
||||
test_results,
|
||||
discord_url
|
||||
)
|
||||
|
||||
return {"message": "Test alert queued"}
|
||||
@@ -1,528 +0,0 @@
|
||||
"""Bulk Mode API routes.
|
||||
|
||||
Lets a user submit up to 100 karaoke jobs at once, by text (artist/title rows)
|
||||
or by album (MusicBrainz release lookup). Album tracklists and text rows are
|
||||
enriched with KaraokeNerds community-version availability so the user can skip
|
||||
tracks that already exist.
|
||||
|
||||
Submission (POST /bulk/submit) and progress (GET /bulk/{batch_id}) live here too
|
||||
(added in later phases).
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
from typing import Literal, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.i18n import get_full_locale_from_request
|
||||
from backend.services.auth_service import AuthResult
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Strong refs to shielded tasks so they aren't garbage-collected if the request is cancelled.
|
||||
_BACKGROUND_TASKS: set = set()
|
||||
|
||||
router = APIRouter(prefix="/bulk", tags=["bulk"])
|
||||
|
||||
MAX_BULK_SONGS = 100
|
||||
|
||||
|
||||
# --- Models -----------------------------------------------------------------
|
||||
|
||||
|
||||
class BulkArtist(BaseModel):
|
||||
mbid: Optional[str] = None
|
||||
name: str
|
||||
disambiguation: str = ""
|
||||
type: Optional[str] = None
|
||||
country: Optional[str] = None
|
||||
# Set when only a MusicBrainz alias matched the query (e.g. "Hikaru Utada").
|
||||
alias: Optional[str] = None
|
||||
|
||||
|
||||
class BulkAlbum(BaseModel):
|
||||
release_group_mbid: str
|
||||
title: str
|
||||
primary_type: Optional[str] = None
|
||||
secondary_types: list[str] = []
|
||||
first_release_date: str = ""
|
||||
is_studio: bool = True
|
||||
|
||||
|
||||
class BulkEdition(BaseModel):
|
||||
release_mbid: str
|
||||
title: Optional[str] = None
|
||||
status: Optional[str] = None
|
||||
date: str = ""
|
||||
country: Optional[str] = None
|
||||
track_count: int = 0
|
||||
|
||||
|
||||
class BulkEditionVariant(BaseModel):
|
||||
"""A distinct tracklist variant (country pressings collapsed by track count)."""
|
||||
|
||||
representative_release_mbid: str
|
||||
label: Literal["Original", "Reissue"] = "Original"
|
||||
track_count: int = 0
|
||||
year: str = ""
|
||||
pressing_count: int = 0
|
||||
delta_vs_original: int = 0
|
||||
|
||||
|
||||
class CommunityVersion(BaseModel):
|
||||
"""An existing community karaoke version the user can preview on YouTube."""
|
||||
|
||||
brand: str
|
||||
url: str
|
||||
|
||||
|
||||
class BulkTrack(BaseModel):
|
||||
position: Optional[int] = None
|
||||
title: str
|
||||
recording_mbid: Optional[str] = None
|
||||
length_ms: Optional[int] = None
|
||||
is_extra: bool = False
|
||||
extra_reason: str = ""
|
||||
# Availability (filled from KaraokeNerds); available=True => already exists
|
||||
available: bool = False
|
||||
brands: list[str] = []
|
||||
versions: list[CommunityVersion] = []
|
||||
|
||||
|
||||
class BulkTracklistResponse(BaseModel):
|
||||
release_mbid: Optional[str] = None
|
||||
canonical_release_mbid: Optional[str] = None
|
||||
selected_variant_mbid: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
date: str = ""
|
||||
tracks: list[BulkTrack] = []
|
||||
editions: list[BulkEdition] = []
|
||||
variants: list[BulkEditionVariant] = []
|
||||
|
||||
|
||||
class AvailabilityItem(BaseModel):
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
|
||||
|
||||
class AvailabilityRequest(BaseModel):
|
||||
tracks: list[AvailabilityItem]
|
||||
|
||||
|
||||
class AvailabilityResult(BaseModel):
|
||||
artist: str
|
||||
title: str
|
||||
available: bool
|
||||
brands: list[str] = []
|
||||
brand_count: int = 0
|
||||
versions: list[CommunityVersion] = []
|
||||
|
||||
|
||||
class AvailabilityResponse(BaseModel):
|
||||
results: list[AvailabilityResult]
|
||||
|
||||
|
||||
# --- Album lookup routes ----------------------------------------------------
|
||||
|
||||
|
||||
@router.get("/album/artists", response_model=list[BulkArtist])
|
||||
async def album_artists(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Artist name search"),
|
||||
limit: int = Query(8, ge=1, le=25),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Search MusicBrainz for artists (album-mode step 1)."""
|
||||
from backend.services.musicbrainz_service import get_musicbrainz_service
|
||||
|
||||
try:
|
||||
results = await get_musicbrainz_service().search_artists(q, limit=limit)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("MusicBrainz artist search failed: %r", e, exc_info=True)
|
||||
raise HTTPException(status_code=502, detail="Artist lookup is temporarily unavailable")
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/album/albums", response_model=list[BulkAlbum])
|
||||
async def album_albums(
|
||||
request: Request,
|
||||
artist_mbid: str = Query(..., min_length=1),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""List an artist's albums (release-groups) for the album picker."""
|
||||
from backend.services.musicbrainz_service import get_musicbrainz_service
|
||||
|
||||
try:
|
||||
return await get_musicbrainz_service().get_albums(artist_mbid)
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("MusicBrainz album list failed: %r", e, exc_info=True)
|
||||
raise HTTPException(status_code=502, detail="Album lookup is temporarily unavailable")
|
||||
|
||||
|
||||
@router.get("/album/tracklist", response_model=BulkTracklistResponse)
|
||||
async def album_tracklist(
|
||||
request: Request,
|
||||
artist: str = Query(..., min_length=1, description="Artist name for availability checks"),
|
||||
release_group_mbid: Optional[str] = Query(None),
|
||||
release_mbid: Optional[str] = Query(None, description="Specific edition to load"),
|
||||
locale: Optional[str] = Query(None, description="UI locale → biases representative-pressing choice"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Resolve an album's canonical tracklist (or a specific edition) and enrich
|
||||
each track with KaraokeNerds community-version availability.
|
||||
|
||||
Response shape note: ``variants`` is only populated on the ``release_group_mbid``
|
||||
path (the initial album load). When loading a specific edition via
|
||||
``release_mbid`` it is intentionally ``[]`` — the client already holds the
|
||||
variant list from the initial load and only needs the chosen edition's tracks.
|
||||
"""
|
||||
from backend.services.musicbrainz_service import (
|
||||
get_musicbrainz_service,
|
||||
country_prefs_for_locale,
|
||||
)
|
||||
from backend.services.karaokenerds_service import check_community_versions_batch
|
||||
|
||||
if not release_group_mbid and not release_mbid:
|
||||
raise HTTPException(status_code=422, detail="release_group_mbid or release_mbid required")
|
||||
|
||||
mb = get_musicbrainz_service()
|
||||
try:
|
||||
if release_mbid:
|
||||
# Loading a specific variant — variants already known to the client.
|
||||
tracklist = await mb.get_release_tracklist(release_mbid, release_group_mbid)
|
||||
tracklist.setdefault("editions", [])
|
||||
tracklist.setdefault("canonical_release_mbid", release_mbid)
|
||||
tracklist.setdefault("variants", [])
|
||||
tracklist.setdefault("selected_variant_mbid", release_mbid)
|
||||
else:
|
||||
tracklist = await mb.get_album_tracklist(
|
||||
release_group_mbid, preferred_countries=country_prefs_for_locale(locale)
|
||||
)
|
||||
except LookupError:
|
||||
# Not in our mirror (or not part of that album) — retrying won't help.
|
||||
raise HTTPException(status_code=404, detail="Release not found")
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("MusicBrainz tracklist failed: %r", e, exc_info=True)
|
||||
raise HTTPException(status_code=502, detail="Tracklist lookup is temporarily unavailable")
|
||||
|
||||
tracks = tracklist.get("tracks", [])
|
||||
# Availability enrichment — never fatal.
|
||||
try:
|
||||
avail = await check_community_versions_batch(
|
||||
[{"artist": artist, "title": t["title"]} for t in tracks]
|
||||
)
|
||||
for t, a in zip(tracks, avail):
|
||||
t["available"] = a["available"]
|
||||
t["brands"] = a["brands"]
|
||||
t["versions"] = a.get("versions", [])
|
||||
except Exception as e: # noqa: BLE001
|
||||
logger.warning("Availability enrichment failed (non-fatal): %s", e)
|
||||
|
||||
return BulkTracklistResponse(
|
||||
release_mbid=tracklist.get("release_mbid"),
|
||||
canonical_release_mbid=tracklist.get("canonical_release_mbid"),
|
||||
selected_variant_mbid=tracklist.get("selected_variant_mbid"),
|
||||
title=tracklist.get("title"),
|
||||
date=tracklist.get("date", ""),
|
||||
tracks=[BulkTrack(**t) for t in tracks],
|
||||
editions=[BulkEdition(**e) for e in tracklist.get("editions", [])],
|
||||
variants=[BulkEditionVariant(**v) for v in tracklist.get("variants", [])],
|
||||
)
|
||||
|
||||
|
||||
@router.post("/availability", response_model=AvailabilityResponse)
|
||||
async def bulk_availability(
|
||||
request: Request,
|
||||
body: AvailabilityRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Batch KaraokeNerds availability check for text-mode rows."""
|
||||
from backend.services.karaokenerds_service import check_community_versions_batch
|
||||
|
||||
if len(body.tracks) > MAX_BULK_SONGS:
|
||||
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} tracks per request")
|
||||
|
||||
results = await check_community_versions_batch(
|
||||
[{"artist": t.artist, "title": t.title} for t in body.tracks]
|
||||
)
|
||||
return AvailabilityResponse(results=[AvailabilityResult(**r) for r in results])
|
||||
|
||||
|
||||
# --- Submission -------------------------------------------------------------
|
||||
|
||||
|
||||
class BulkSong(BaseModel):
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
display_artist: Optional[str] = None
|
||||
display_title: Optional[str] = None
|
||||
# Album mode: the track's number on its disc (album-search tie-break).
|
||||
position: Optional[int] = None
|
||||
|
||||
|
||||
class BulkAlbumRef(BaseModel):
|
||||
"""The album an album-mode batch was picked from. Lets the search worker find
|
||||
the album's torrent once instead of searching every track separately."""
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
release_group_mbid: Optional[str] = None
|
||||
release_mbid: Optional[str] = None
|
||||
|
||||
|
||||
class BulkSettings(BaseModel):
|
||||
auto_select_if_lossless: bool = True
|
||||
is_private: bool = False
|
||||
skip_audio_edit: bool = True
|
||||
skip_customization: bool = True
|
||||
# Full-auto review (workstream C): batch default autonomy + backing preference.
|
||||
review_mode: str = "auto"
|
||||
backing_preference: str = "auto"
|
||||
translation_language: Optional[str] = None
|
||||
|
||||
|
||||
class BulkSubmitRequest(BaseModel):
|
||||
songs: list[BulkSong]
|
||||
settings: BulkSettings = Field(default_factory=BulkSettings)
|
||||
album: Optional[BulkAlbumRef] = None
|
||||
|
||||
|
||||
class BulkSubmitResponse(BaseModel):
|
||||
batch_id: str
|
||||
job_ids: list[str]
|
||||
total: int
|
||||
|
||||
|
||||
@router.post("/submit", response_model=BulkSubmitResponse)
|
||||
async def bulk_submit(
|
||||
request: Request,
|
||||
body: BulkSubmitRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Create one job per song, gate on credits upfront, and dispatch the batch
|
||||
search worker. Jobs start parked (auto_download=False) and the worker either
|
||||
auto-selects a confident lossless match or leaves them awaiting selection."""
|
||||
from backend.config import get_settings
|
||||
from backend.exceptions import InsufficientCreditsError
|
||||
from backend.models.job import JobCreate
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.worker_service import get_worker_service
|
||||
|
||||
n = len(body.songs)
|
||||
if n == 0:
|
||||
raise HTTPException(status_code=422, detail="At least one song is required")
|
||||
if n > MAX_BULK_SONGS:
|
||||
raise HTTPException(status_code=422, detail=f"At most {MAX_BULK_SONGS} songs per batch")
|
||||
|
||||
user_email = auth_result.user_email
|
||||
is_admin = bool(auth_result.is_admin)
|
||||
|
||||
locale = get_full_locale_from_request(request)
|
||||
|
||||
# Everything up to dispatch is sync Firestore/GCS work (credit check, default
|
||||
# theme, one create_job + update_job per song). Run it off the event loop: a
|
||||
# 2026-10-22 album batch stalled the loop 26s and 9 other requests with it.
|
||||
def _create_batch_jobs() -> tuple[str, list[str]]:
|
||||
# Upfront credit gate (1 credit/song minimum). Admins bypass.
|
||||
if not is_admin and user_email:
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
available = get_user_service().check_credits(user_email)
|
||||
if available < n:
|
||||
raise HTTPException(
|
||||
status_code=402,
|
||||
detail={
|
||||
"error": "insufficient_credits",
|
||||
"message": "You don't have enough credits for this batch.",
|
||||
"credits_available": available,
|
||||
"credits_required": n,
|
||||
},
|
||||
)
|
||||
|
||||
settings = get_settings()
|
||||
theme_service = get_theme_service()
|
||||
theme_id = theme_service.get_default_theme_id()
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(theme_id, None, None)
|
||||
|
||||
batch_id = uuid.uuid4().hex[:12]
|
||||
bulk_settings = body.settings.model_dump()
|
||||
bulk_album = body.album.model_dump() if body.album else None
|
||||
job_manager = JobManager()
|
||||
job_ids: list[str] = []
|
||||
|
||||
for song in body.songs:
|
||||
display_artist = (song.display_artist or song.artist).strip()
|
||||
display_title = (song.display_title or song.title).strip()
|
||||
request_metadata = {
|
||||
"created_from": "bulk",
|
||||
"batch_id": batch_id,
|
||||
"user_email": user_email,
|
||||
}
|
||||
job_create = JobCreate(
|
||||
artist=display_artist,
|
||||
title=display_title,
|
||||
theme_id=theme_id,
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
user_email=user_email,
|
||||
audio_search_artist=song.artist.strip(),
|
||||
audio_search_title=song.title.strip(),
|
||||
auto_download=False,
|
||||
is_private=body.settings.is_private,
|
||||
review_mode=body.settings.review_mode,
|
||||
backing_preference=body.settings.backing_preference,
|
||||
translation_language=body.settings.translation_language,
|
||||
brand_prefix=settings.default_brand_prefix,
|
||||
enable_youtube_upload=settings.default_enable_youtube_upload,
|
||||
youtube_description=settings.default_youtube_description,
|
||||
discord_webhook_url=settings.default_discord_webhook_url,
|
||||
dropbox_path=settings.default_dropbox_path,
|
||||
gdrive_folder_id=settings.default_gdrive_folder_id,
|
||||
request_metadata=request_metadata,
|
||||
locale=locale,
|
||||
)
|
||||
try:
|
||||
job = job_manager.create_job(job_create, is_admin=is_admin)
|
||||
except InsufficientCreditsError:
|
||||
# Ran out mid-batch (e.g. concurrent spend after the gate). Stop here;
|
||||
# already-created jobs are valid and will proceed.
|
||||
logger.warning("[batch:%s] ran out of credits after %d/%d jobs", batch_id, len(job_ids), n)
|
||||
break
|
||||
|
||||
bulk_state = {
|
||||
"state_data.batch_id": batch_id,
|
||||
"state_data.bulk_settings": bulk_settings,
|
||||
"state_data.bulk_pending_search": True,
|
||||
}
|
||||
if bulk_album is not None:
|
||||
bulk_state["state_data.bulk_album"] = bulk_album
|
||||
bulk_state["state_data.bulk_track_position"] = song.position
|
||||
job_manager.update_job(job.job_id, bulk_state)
|
||||
job_ids.append(job.job_id)
|
||||
return batch_id, job_ids
|
||||
|
||||
# Dispatch the batch search worker. Retry once on transient failure. If it
|
||||
# still fails the jobs exist (and are charged) but won't process, so surface
|
||||
# an explicit error with the batch_id — and tell the user NOT to resubmit
|
||||
# (that would create+charge a second batch); the batch can be re-dispatched.
|
||||
worker_service = get_worker_service()
|
||||
|
||||
async def _create_and_dispatch() -> tuple[str, list[str], bool]:
|
||||
batch_id, job_ids = await asyncio.to_thread(_create_batch_jobs)
|
||||
if not job_ids:
|
||||
return batch_id, job_ids, False
|
||||
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
|
||||
if not triggered:
|
||||
triggered = await worker_service.trigger_bulk_search_worker(batch_id)
|
||||
return batch_id, job_ids, triggered
|
||||
|
||||
# Shielded: if the request is cancelled (client disconnect) mid-way, the jobs
|
||||
# being created and charged in the thread still get dispatched.
|
||||
task = asyncio.create_task(_create_and_dispatch())
|
||||
_BACKGROUND_TASKS.add(task)
|
||||
task.add_done_callback(_BACKGROUND_TASKS.discard)
|
||||
batch_id, job_ids, triggered = await asyncio.shield(task)
|
||||
|
||||
if not job_ids:
|
||||
raise HTTPException(status_code=402, detail="No jobs could be created")
|
||||
if not triggered:
|
||||
logger.critical(
|
||||
"[batch:%s] created %d jobs but FAILED to dispatch bulk-search worker",
|
||||
batch_id, len(job_ids),
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail={
|
||||
"error": "dispatch_failed",
|
||||
"message": "Your batch was created but processing couldn't start. "
|
||||
"Please don't resubmit — contact support with this batch ID and we'll start it.",
|
||||
"batch_id": batch_id,
|
||||
"job_ids": job_ids,
|
||||
},
|
||||
)
|
||||
logger.info("[batch:%s] created %d jobs, dispatched bulk-search worker", batch_id, len(job_ids))
|
||||
|
||||
return BulkSubmitResponse(batch_id=batch_id, job_ids=job_ids, total=len(job_ids))
|
||||
|
||||
|
||||
# --- Progress ---------------------------------------------------------------
|
||||
|
||||
|
||||
class BulkJobStatus(BaseModel):
|
||||
job_id: str
|
||||
artist: str
|
||||
title: str
|
||||
status: str
|
||||
auto_selected: bool = False
|
||||
|
||||
|
||||
class BulkBatchResponse(BaseModel):
|
||||
batch_id: str
|
||||
total: int
|
||||
counts: dict
|
||||
jobs: list[BulkJobStatus]
|
||||
|
||||
|
||||
# Status -> progress bucket for the batch summary.
|
||||
_AWAITING = "awaiting_audio_selection"
|
||||
_TERMINAL_OK = "complete"
|
||||
_TERMINAL_FAIL = "failed"
|
||||
_SEARCHING = {"pending", "searching_audio"}
|
||||
|
||||
|
||||
def _bucket(status: str, pending_search: bool) -> str:
|
||||
if pending_search or status in _SEARCHING:
|
||||
return "searching"
|
||||
if status == _AWAITING:
|
||||
return "awaiting_selection"
|
||||
if status == _TERMINAL_OK:
|
||||
return "complete"
|
||||
if status == _TERMINAL_FAIL:
|
||||
return "failed"
|
||||
return "processing"
|
||||
|
||||
|
||||
@router.get("/{batch_id}", response_model=BulkBatchResponse)
|
||||
def bulk_batch(
|
||||
request: Request,
|
||||
batch_id: str,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Progress summary for a batch: counts by bucket + per-job status."""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
jobs = job_manager.list_jobs_by_batch_id(batch_id)
|
||||
|
||||
if not bool(auth_result.is_admin):
|
||||
email = (auth_result.user_email or "").lower()
|
||||
jobs = [j for j in jobs if (j.user_email or "").lower() == email]
|
||||
|
||||
if not jobs:
|
||||
raise HTTPException(status_code=404, detail="Batch not found")
|
||||
|
||||
counts = {"searching": 0, "awaiting_selection": 0, "processing": 0, "complete": 0, "failed": 0}
|
||||
job_statuses: list[BulkJobStatus] = []
|
||||
for j in jobs:
|
||||
status = j.status.value if hasattr(j.status, "value") else str(j.status)
|
||||
sd = j.state_data or {}
|
||||
bucket = _bucket(status, bool(sd.get("bulk_pending_search")))
|
||||
counts[bucket] += 1
|
||||
job_statuses.append(
|
||||
BulkJobStatus(
|
||||
job_id=j.job_id,
|
||||
artist=j.audio_search_artist or j.artist or "",
|
||||
title=j.audio_search_title or j.title or "",
|
||||
status=status,
|
||||
auto_selected=bool(sd.get("bulk_auto_selected")),
|
||||
)
|
||||
)
|
||||
|
||||
return BulkBatchResponse(
|
||||
batch_id=batch_id, total=len(jobs), counts=counts, jobs=job_statuses
|
||||
)
|
||||
@@ -1,229 +0,0 @@
|
||||
"""
|
||||
Catalog API routes for song/artist lookup and community version detection.
|
||||
|
||||
Provides autocomplete data from karaoke-decide's MusicBrainz + Spotify catalog,
|
||||
and checks karaokenerds.com for existing community karaoke versions.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Literal, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/catalog", tags=["catalog"])
|
||||
|
||||
# Simple per-user rate limiting: 20 requests/minute
|
||||
# { user_email: [timestamp, timestamp, ...] }
|
||||
_rate_limit_windows: dict[str, list[float]] = {}
|
||||
RATE_LIMIT_MAX_REQUESTS = 20
|
||||
RATE_LIMIT_WINDOW_SECONDS = 60
|
||||
|
||||
|
||||
def _check_rate_limit(user_email: str, locale: str = "en") -> None:
|
||||
"""Check per-user rate limit. Raises 429 if exceeded."""
|
||||
now = time.monotonic()
|
||||
window_start = now - RATE_LIMIT_WINDOW_SECONDS
|
||||
|
||||
# Get or create window for this user
|
||||
if user_email not in _rate_limit_windows:
|
||||
_rate_limit_windows[user_email] = []
|
||||
|
||||
# Remove expired entries
|
||||
timestamps = _rate_limit_windows[user_email]
|
||||
_rate_limit_windows[user_email] = [ts for ts in timestamps if ts > window_start]
|
||||
|
||||
if len(_rate_limit_windows[user_email]) >= RATE_LIMIT_MAX_REQUESTS:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail=t(locale, "catalog.rateLimitExceeded", max_requests=RATE_LIMIT_MAX_REQUESTS),
|
||||
)
|
||||
|
||||
_rate_limit_windows[user_email].append(now)
|
||||
|
||||
|
||||
# --- Response Models ---
|
||||
|
||||
|
||||
class ArtistResult(BaseModel):
|
||||
"""Artist search result from MusicBrainz/Spotify catalog."""
|
||||
name: str
|
||||
mbid: Optional[str] = None
|
||||
disambiguation: Optional[str] = None
|
||||
artist_type: Optional[str] = None
|
||||
spotify_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
genres: Optional[list[str]] = None
|
||||
tags: Optional[list[str]] = None
|
||||
|
||||
|
||||
class TrackResult(BaseModel):
|
||||
"""Track search result from Spotify catalog."""
|
||||
track_name: str
|
||||
artist_name: str
|
||||
track_id: Optional[str] = None
|
||||
artist_id: Optional[str] = None
|
||||
popularity: Optional[int] = None
|
||||
duration_ms: Optional[int] = None
|
||||
explicit: Optional[bool] = None
|
||||
|
||||
|
||||
class CommunityTrack(BaseModel):
|
||||
"""A community-approved karaoke track from karaokenerds."""
|
||||
brand_name: str
|
||||
brand_code: str
|
||||
youtube_url: str
|
||||
is_community: bool = True
|
||||
|
||||
|
||||
class CommunityCheckSong(BaseModel):
|
||||
"""A song with community karaoke tracks."""
|
||||
title: str
|
||||
artist: str
|
||||
community_tracks: list[CommunityTrack]
|
||||
|
||||
|
||||
class CommunityCheckRequest(BaseModel):
|
||||
"""Request body for community version check."""
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
|
||||
|
||||
class CommunityCheckResponse(BaseModel):
|
||||
"""Response for community version check."""
|
||||
has_community: bool
|
||||
songs: list[CommunityCheckSong]
|
||||
best_youtube_url: Optional[str] = None
|
||||
|
||||
|
||||
class MatchJudgeRequest(BaseModel):
|
||||
"""Request body for the artist/title match judge."""
|
||||
artist: str = Field(..., min_length=1)
|
||||
title: str = Field(..., min_length=1)
|
||||
# The audio search's confidence tier (1=strong..3=weak), so the judge can
|
||||
# tell whether weak results hint at a typo. Optional.
|
||||
audio_confidence_tier: Optional[int] = Field(None, ge=1, le=3)
|
||||
# "fast" = catalog-only pass (no AI, no tier needed) the client fires in
|
||||
# parallel with the audio search; "full" = complete pipeline once the tier is
|
||||
# known. Defaults to "full" so a single legacy call still works.
|
||||
stage: Literal["fast", "full"] = "full"
|
||||
|
||||
|
||||
class MatchJudgeAlternative(BaseModel):
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
|
||||
class MatchJudgeResponse(BaseModel):
|
||||
"""Verdict on a typed artist/title — formatting + match acceptance."""
|
||||
kind: str # "cosmetic" | "content" | "ambiguous" | "none"
|
||||
confident: bool
|
||||
canonical_artist: str
|
||||
canonical_title: str
|
||||
alternatives: list[MatchJudgeAlternative] = []
|
||||
engine: str # "deterministic" | "catalog" | "ai"
|
||||
reason: str = ""
|
||||
# Set by a "fast" pass that couldn't decide — the client should follow up with
|
||||
# a "full" call once the audio tier is known.
|
||||
needs_ai: bool = False
|
||||
|
||||
|
||||
# --- Routes ---
|
||||
|
||||
|
||||
@router.get("/artists", response_model=list[ArtistResult])
|
||||
async def search_artists(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Artist name search query"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for artists by name. Returns canonical MusicBrainz/Spotify artist data
|
||||
for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_artists
|
||||
results = await search_artists(q, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.get("/tracks", response_model=list[TrackResult])
|
||||
async def search_tracks(
|
||||
request: Request,
|
||||
q: str = Query(..., min_length=2, description="Track name search query"),
|
||||
artist: Optional[str] = Query(None, description="Filter by artist name"),
|
||||
limit: int = Query(10, ge=1, le=50, description="Maximum results"),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Search for tracks by name, optionally filtered by artist.
|
||||
Returns canonical Spotify track data for autocomplete in job creation forms.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.catalog_proxy_service import search_tracks
|
||||
results = await search_tracks(q, artist, limit)
|
||||
return results
|
||||
|
||||
|
||||
@router.post("/community-check", response_model=CommunityCheckResponse)
|
||||
async def check_community_versions(
|
||||
request: Request,
|
||||
body: CommunityCheckRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Check if a song already has community-approved karaoke versions on YouTube
|
||||
via karaokenerds.com. Non-blocking — just informs the user.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.karaokenerds_service import check_community_versions
|
||||
result = await check_community_versions(body.artist, body.title)
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/match-judge", response_model=MatchJudgeResponse)
|
||||
async def match_judge(
|
||||
request: Request,
|
||||
body: MatchJudgeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""
|
||||
Judge a typed artist/title: decide official formatting + whether it matches a
|
||||
real song. Deterministic + catalog layers run first; a light AI model fires
|
||||
only when those aren't confident. Never blocks the submission flow — failures
|
||||
degrade to a "none" verdict.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
_check_rate_limit(auth_result.user_email or auth_result.token_id or "unknown", locale)
|
||||
|
||||
from backend.services.fake_workers import fake_workers_enabled
|
||||
if fake_workers_enabled():
|
||||
# Local E2E stack: scripted verdict (per-user fake config), no catalog/Gemini.
|
||||
from backend.services.fake_workers.judge import fake_match_verdict
|
||||
return fake_match_verdict(body.artist, body.title, auth_result.user_email, stage=body.stage).to_dict()
|
||||
|
||||
from backend.services.match_judge.service import judge_match
|
||||
verdict = await judge_match(
|
||||
body.artist, body.title,
|
||||
audio_tier=body.audio_confidence_tier,
|
||||
stage=body.stage,
|
||||
)
|
||||
logger.info(
|
||||
"match-judge stage=%s kind=%s engine=%s confident=%s needs_ai=%s",
|
||||
body.stage, verdict.kind, verdict.engine, verdict.confident, verdict.needs_ai,
|
||||
)
|
||||
return verdict.to_dict()
|
||||
@@ -1,136 +0,0 @@
|
||||
"""POST /api/client-errors — ingest frontend crash reports.
|
||||
|
||||
Unauthenticated (logged-out users' crashes still matter). Deduped and alerted
|
||||
via the existing error-monitor pipeline — see
|
||||
``backend/services/error_monitor``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Request, HTTPException
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from backend.services.error_monitor.firestore_adapter import ErrorPatternsAdapter
|
||||
from backend.services.error_monitor.frontend_ingestion import (
|
||||
FrontendErrorReport,
|
||||
RateLimiter,
|
||||
build_pattern_data,
|
||||
is_bot_user_agent,
|
||||
is_host_app_injection_error,
|
||||
is_opaque_script_error,
|
||||
sanitize_url,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/client-errors", tags=["client-errors"])
|
||||
|
||||
_limiter = RateLimiter(max_per_minute=60)
|
||||
_adapter_singleton: ErrorPatternsAdapter | None = None
|
||||
|
||||
|
||||
def _get_adapter() -> ErrorPatternsAdapter:
|
||||
global _adapter_singleton
|
||||
if _adapter_singleton is None:
|
||||
_adapter_singleton = ErrorPatternsAdapter()
|
||||
return _adapter_singleton
|
||||
|
||||
|
||||
class ClientErrorPayload(BaseModel):
|
||||
message: str = Field(..., min_length=1, max_length=4000)
|
||||
stack: Optional[str] = Field(None, max_length=1_000_000)
|
||||
url: str = Field("", max_length=2048)
|
||||
user_agent: str = Field("", max_length=1024)
|
||||
release: str = Field("", max_length=64)
|
||||
user_email: Optional[str] = Field(None, max_length=320)
|
||||
viewport: Optional[dict] = None
|
||||
locale: str = Field("en", max_length=10)
|
||||
source: str = Field("unknown", max_length=64)
|
||||
# optional free-form extra for debugging. Capped server-side.
|
||||
extra: Optional[dict] = None
|
||||
|
||||
@field_validator("message", "url", "user_agent", "locale", "source")
|
||||
@classmethod
|
||||
def strip(cls, v: str) -> str:
|
||||
return (v or "").strip()
|
||||
|
||||
|
||||
class ClientErrorResponse(BaseModel):
|
||||
pattern_id: str
|
||||
is_new: bool
|
||||
|
||||
|
||||
@router.post("", response_model=ClientErrorResponse, status_code=202)
|
||||
def report_client_error(payload: ClientErrorPayload, request: Request) -> ClientErrorResponse:
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
if not _limiter.allow(client_ip, time.monotonic()):
|
||||
raise HTTPException(status_code=429, detail="too many reports")
|
||||
|
||||
# Crawler/bot crashes are pure alert noise (bots run headless Chrome and trip
|
||||
# DOM errors no real user hits). Accept the report so the client doesn't retry,
|
||||
# but don't persist/alert on it.
|
||||
if is_bot_user_agent(payload.user_agent):
|
||||
# Log the bot UA (not sensitive) + sanitized URL (query/token stripped)
|
||||
# for visibility into what's being dropped, without persisting/alerting.
|
||||
logger.info(
|
||||
"frontend_crash_ignored_bot ua=%r url=%s source=%s",
|
||||
payload.user_agent[:120],
|
||||
sanitize_url(payload.url),
|
||||
payload.source,
|
||||
)
|
||||
return ClientErrorResponse(pattern_id="bot-ignored", is_new=False)
|
||||
|
||||
# Opaque cross-origin "Script error." — no file/line, not our code.
|
||||
if is_opaque_script_error(payload.message, payload.source, payload.extra):
|
||||
logger.info(
|
||||
"frontend_crash_ignored_opaque_script_error url=%s source=%s",
|
||||
sanitize_url(payload.url),
|
||||
payload.source,
|
||||
)
|
||||
return ClientErrorResponse(pattern_id="opaque-script-error-ignored", is_new=False)
|
||||
|
||||
# A script the in-app browser's host app injected (Zalo), not our code.
|
||||
if is_host_app_injection_error(payload.message, payload.user_agent):
|
||||
logger.info(
|
||||
"frontend_crash_ignored_host_app_injection url=%s source=%s",
|
||||
sanitize_url(payload.url),
|
||||
payload.source,
|
||||
)
|
||||
return ClientErrorResponse(pattern_id="host-app-injection-ignored", is_new=False)
|
||||
|
||||
report = FrontendErrorReport(
|
||||
message=payload.message,
|
||||
stack=payload.stack,
|
||||
url=payload.url,
|
||||
user_agent=payload.user_agent,
|
||||
release=payload.release,
|
||||
user_email=payload.user_email,
|
||||
viewport=payload.viewport,
|
||||
locale=payload.locale,
|
||||
extra=payload.extra,
|
||||
)
|
||||
pattern_data = build_pattern_data(report)
|
||||
|
||||
try:
|
||||
result = _get_adapter().upsert_pattern(pattern_data)
|
||||
except Exception: # pragma: no cover - Firestore transient errors
|
||||
logger.exception("Failed to upsert frontend error pattern")
|
||||
raise HTTPException(status_code=503, detail="storage unavailable")
|
||||
|
||||
logger.info(
|
||||
"frontend_crash_reported pattern_id=%s is_new=%s source=%s",
|
||||
result.pattern_id,
|
||||
result.is_new,
|
||||
payload.source,
|
||||
# Full client diagnostics + breadcrumbs, searchable in Cloud Logging
|
||||
# (the alert sample only carries a trimmed summary).
|
||||
extra={
|
||||
"error_pattern_id": result.pattern_id,
|
||||
"client_url": sanitize_url(payload.url),
|
||||
"client_release": payload.release,
|
||||
"client_extra": payload.extra,
|
||||
},
|
||||
)
|
||||
return ClientErrorResponse(pattern_id=result.pattern_id, is_new=result.is_new)
|
||||
@@ -1,212 +0,0 @@
|
||||
"""POST /api/client-events — persistent telemetry for user-visible degradation.
|
||||
|
||||
Records every time the frontend shows a user a degraded-service surface (the
|
||||
orange "trouble reaching our servers" banner, the stronger "temporarily
|
||||
unavailable" error, a failed lyrics load, a slow/failed waveform), with enough
|
||||
metadata to review frequency and context later. Two sinks per event:
|
||||
|
||||
1. A structured INFO log line (``client_event type=…``) — queryable in Cloud
|
||||
Logging / Log Analytics alongside backend symptoms from the same window.
|
||||
2. A Firestore doc in ``client_events`` — survives log retention, cheap to
|
||||
aggregate for "how often are users seeing this?" reviews.
|
||||
|
||||
Identity (2026-10-03): the client sends its session token (if any) and the
|
||||
server resolves ``user_email`` / ``is_admin`` / ``is_internal`` / ``is_test``
|
||||
from it — a client-supplied email is never trusted (and is ignored). Each
|
||||
event also carries the FingerprintJS ``device_fingerprint`` (same visitorId the
|
||||
magic-link abuse checks use), a per-tab ``tab_id`` and an ``episode_id`` that
|
||||
groups one degradation episode (first banner → ``banner_recovered``).
|
||||
|
||||
The watchdog in ``backend/services/loop_watchdog.py`` writes server-side
|
||||
``server_loop_stall`` docs into the same collection; clients cannot submit
|
||||
that type.
|
||||
|
||||
Unauthenticated (review-token users must be able to report) and rate-limited
|
||||
per IP, mirroring /api/client-errors. The limiter is process-local, so the
|
||||
effective ceiling is 120/min/IP × live instances — acceptable for this threat
|
||||
model (non-malicious browsers, 60s client-side per-type throttle); Cloudflare's
|
||||
zone-wide rate limit sits in front as a flood backstop. Events are dropped,
|
||||
never queued, when the limiter trips — telemetry, not a delivery guarantee.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Literal, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from backend.utils.test_data import is_internal_email, is_test_email
|
||||
from backend.services.error_monitor.frontend_ingestion import (
|
||||
RateLimiter,
|
||||
is_bot_user_agent,
|
||||
sanitize_url,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/client-events", tags=["client-events"])
|
||||
|
||||
# Sized for a legitimate worst case from ONE IP: a many-tab burst (e.g. 16 review
|
||||
# tabs at once) where every tab reports a banner + a waveform event — that must be
|
||||
# fully recorded, not self-suppressed. Each tab throttles per-type client-side.
|
||||
_limiter = RateLimiter(max_per_minute=120)
|
||||
_db_singleton = None
|
||||
|
||||
# The full closed set of reportable events. Reject anything else so the
|
||||
# collection stays reviewable (no free-form types accumulating).
|
||||
EventType = Literal[
|
||||
"banner_reconnecting",
|
||||
"banner_unavailable",
|
||||
"banner_waking",
|
||||
"banner_recovered",
|
||||
"lyrics_load_failed",
|
||||
"waveform_slow",
|
||||
"waveform_failed",
|
||||
]
|
||||
|
||||
|
||||
def _get_db():
|
||||
global _db_singleton
|
||||
if _db_singleton is None:
|
||||
from google.cloud import firestore # type: ignore[import]
|
||||
|
||||
_db_singleton = firestore.Client(project="nomadkaraoke")
|
||||
return _db_singleton
|
||||
|
||||
|
||||
class ClientEventPayload(BaseModel):
|
||||
type: EventType
|
||||
url: str = Field("", max_length=2048)
|
||||
job_id: Optional[str] = Field(None, max_length=64)
|
||||
# Legacy field from pre-2026-10-03 frontends — accepted but IGNORED (the
|
||||
# server resolves identity from the bearer token instead).
|
||||
user_email: Optional[str] = Field(None, max_length=320)
|
||||
device_fingerprint: Optional[str] = Field(None, max_length=128)
|
||||
tab_id: Optional[str] = Field(None, max_length=64)
|
||||
episode_id: Optional[str] = Field(None, max_length=64)
|
||||
locale: str = Field("en", max_length=10)
|
||||
release: str = Field("", max_length=64)
|
||||
# Small free-form context (stall age, http status, elapsed ms, probe state…).
|
||||
# Size-capped server-side; values are stored as-is.
|
||||
detail: Optional[dict] = None
|
||||
|
||||
@field_validator("url", "locale", "release")
|
||||
@classmethod
|
||||
def strip(cls, v: str) -> str:
|
||||
return (v or "").strip()
|
||||
|
||||
|
||||
def _capped_detail(detail: Optional[dict]) -> dict[str, Any]:
|
||||
"""Keep detail reviewable: at most 12 keys, scalar-ish values, short strings."""
|
||||
if not isinstance(detail, dict):
|
||||
return {}
|
||||
out: dict[str, Any] = {}
|
||||
for key, value in list(detail.items())[:12]:
|
||||
k = str(key)[:40]
|
||||
if isinstance(value, (int, float, bool)) or value is None:
|
||||
out[k] = value
|
||||
else:
|
||||
out[k] = str(value)[:200]
|
||||
return out
|
||||
|
||||
|
||||
_TENANT_HOST_EXCLUDE = {"gen", "www", "api", "app", "decide", "localhost"}
|
||||
|
||||
|
||||
def _tenant_from_url(url: str) -> Optional[str]:
|
||||
"""``randy-vild.nomadkaraoke.com/...`` → ``randy-vild``; consumer hosts → None."""
|
||||
try:
|
||||
host = (urlparse(url).hostname or "").lower()
|
||||
except ValueError:
|
||||
return None
|
||||
if not host.endswith(".nomadkaraoke.com"):
|
||||
return None
|
||||
sub = host[: -len(".nomadkaraoke.com")]
|
||||
if not sub or "." in sub or sub in _TENANT_HOST_EXCLUDE:
|
||||
return None
|
||||
return sub
|
||||
|
||||
|
||||
def _resolve_identity(request: Request) -> dict[str, Any]:
|
||||
"""Resolve the caller from its bearer token. Never raises; anonymous on failure."""
|
||||
identity: dict[str, Any] = {
|
||||
"user_email": None, "is_admin": False, "is_internal": False, "is_test": False,
|
||||
}
|
||||
auth = request.headers.get("authorization", "")
|
||||
if not auth.lower().startswith("bearer "):
|
||||
return identity
|
||||
token = auth[7:].strip()
|
||||
if not token:
|
||||
return identity
|
||||
try:
|
||||
from backend.services.auth_service import get_auth_service
|
||||
|
||||
result = get_auth_service().validate_token_full(token)
|
||||
except Exception:
|
||||
logger.debug("client_event token resolution failed", exc_info=True)
|
||||
return identity
|
||||
if not result.is_valid:
|
||||
return identity
|
||||
email = result.user_email
|
||||
identity["user_email"] = email
|
||||
identity["is_admin"] = bool(result.is_admin)
|
||||
identity["is_internal"] = bool(email and is_internal_email(email))
|
||||
identity["is_test"] = bool(email and is_test_email(email))
|
||||
return identity
|
||||
|
||||
|
||||
@router.post("", status_code=202)
|
||||
def report_client_event(payload: ClientEventPayload, request: Request) -> dict:
|
||||
client_ip = request.client.host if request.client else "unknown"
|
||||
if not _limiter.allow(client_ip, time.monotonic()):
|
||||
raise HTTPException(status_code=429, detail="too many events")
|
||||
|
||||
user_agent = request.headers.get("user-agent", "")[:1024]
|
||||
if is_bot_user_agent(user_agent):
|
||||
return {"status": "ignored"}
|
||||
|
||||
# Sync handler → runs in FastAPI's threadpool, so the token lookup
|
||||
# (Firestore) never touches the event loop.
|
||||
identity = _resolve_identity(request)
|
||||
|
||||
event = {
|
||||
"type": payload.type,
|
||||
"source": "client",
|
||||
"url": sanitize_url(payload.url),
|
||||
"tenant": _tenant_from_url(payload.url),
|
||||
"job_id": payload.job_id,
|
||||
**identity,
|
||||
"device_fingerprint": payload.device_fingerprint,
|
||||
"tab_id": payload.tab_id,
|
||||
"episode_id": payload.episode_id,
|
||||
"locale": payload.locale,
|
||||
"release": payload.release,
|
||||
"user_agent": user_agent,
|
||||
"detail": _capped_detail(payload.detail),
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
}
|
||||
|
||||
# Sink 1: structured log line — correlate with backend symptoms in Logging.
|
||||
logger.info(
|
||||
"client_event type=%s job_id=%s user=%s admin=%s fp=%s episode=%s url=%s release=%s detail=%s",
|
||||
event["type"],
|
||||
event["job_id"],
|
||||
event["user_email"],
|
||||
event["is_admin"],
|
||||
event["device_fingerprint"],
|
||||
event["episode_id"],
|
||||
event["url"],
|
||||
event["release"],
|
||||
event["detail"],
|
||||
)
|
||||
|
||||
# Sink 2: Firestore — best-effort; telemetry must never error back to users.
|
||||
try:
|
||||
_get_db().collection("client_events").add(event)
|
||||
except Exception:
|
||||
logger.exception("Failed to persist client event (non-fatal)")
|
||||
|
||||
return {"status": "recorded"}
|
||||
@@ -1,104 +0,0 @@
|
||||
"""
|
||||
Encoding worker lifecycle endpoints.
|
||||
|
||||
Provides warmup and heartbeat endpoints for the blue-green
|
||||
encoding worker VMs. Called by the lyrics-review frontend to ensure the
|
||||
primary VM is running before the reviewer previews a render.
|
||||
|
||||
Auth: these are gated on ``require_review_auth`` (review access to a specific
|
||||
job), NOT ``require_admin``. The callers are ordinary customers on the
|
||||
lyrics-review page — gating on admin makes every non-admin caller 403, which
|
||||
silently defeats the JIT pre-warm. Scoping to the job being reviewed keeps the
|
||||
abuse posture identical to the rest of the review surface: you can only warm
|
||||
the shared encoding VM if you already have review access to a real job.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Tuple
|
||||
from fastapi import APIRouter, Depends
|
||||
from backend.api.dependencies import require_review_auth
|
||||
from backend.services.encoding_worker_manager import EncodingWorkerManager
|
||||
from backend.services.encoding_errors import EncodingWorkerStartError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/internal/encoding-worker", tags=["encoding-worker"])
|
||||
|
||||
# Lazy-loaded singleton
|
||||
_manager = None
|
||||
|
||||
|
||||
def get_worker_manager():
|
||||
global _manager
|
||||
if _manager is None:
|
||||
from google.cloud import compute_v1, firestore
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
db = firestore.Client(project=settings.google_cloud_project)
|
||||
compute_client = compute_v1.InstancesClient()
|
||||
_manager = EncodingWorkerManager(
|
||||
db=db,
|
||||
compute_client=compute_client,
|
||||
project_id=settings.google_cloud_project,
|
||||
)
|
||||
return _manager
|
||||
|
||||
|
||||
@router.post("/warmup/{job_id}")
|
||||
async def warmup_encoding_worker(
|
||||
job_id: str,
|
||||
_auth: Tuple[str, str] = Depends(require_review_auth),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Start the preferred encoding worker if it's stopped (normally the GCE primary).
|
||||
|
||||
Called on lyrics-review page load so the VM is warm by the time the
|
||||
reviewer previews a render. ``job_id`` scopes auth to the job under review.
|
||||
"""
|
||||
try:
|
||||
# ensure_primary_running does Firestore reads + GCE compute-API calls
|
||||
# (instances.get / instances.start) that can take 10s+. Run it off the
|
||||
# event loop: on the single-process server a sync call here froze the
|
||||
# WHOLE instance for ~11s on every review-page open with a cold VM,
|
||||
# stalling that same page's audio/lyrics requests behind it
|
||||
# (2026-09-19, follow-up to the concurrent-load investigation).
|
||||
result = await asyncio.to_thread(manager.ensure_preferred_running)
|
||||
if result["started"]:
|
||||
logger.info(f"Started encoding worker VM: {result['vm_name']}")
|
||||
return result
|
||||
except EncodingWorkerStartError as e:
|
||||
# The primary VM couldn't start (capacity exhaustion or a transient
|
||||
# 503 SERVICE_UNAVAILABLE from the GCE backend). This is NOT fatal:
|
||||
# the encoding flow's own warmup (ensure_any_running) falls back to
|
||||
# the alternate-zone workers, so the job still completes. Log at
|
||||
# WARNING so it doesn't trip the production error monitor and page us
|
||||
# for self-healing capacity events.
|
||||
logger.warning(
|
||||
f"Primary encoding worker warmup failed ({e}); "
|
||||
f"encoding will fall back to an alternate-zone worker"
|
||||
)
|
||||
return {"started": False, "error": str(e)}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to warm up encoding worker: {e}")
|
||||
return {"started": False, "error": str(e)}
|
||||
|
||||
|
||||
@router.post("/heartbeat/{job_id}")
|
||||
async def heartbeat_encoding_worker(
|
||||
job_id: str,
|
||||
_auth: Tuple[str, str] = Depends(require_review_auth),
|
||||
manager=Depends(get_worker_manager),
|
||||
):
|
||||
"""Update activity timestamp to prevent idle shutdown.
|
||||
|
||||
Called periodically while the reviewer works. ``job_id`` scopes auth to the
|
||||
job under review.
|
||||
"""
|
||||
try:
|
||||
# Sync Firestore write — keep it off the event loop (see warmup above).
|
||||
await asyncio.to_thread(manager.update_activity)
|
||||
return {"status": "ok"}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update encoding worker heartbeat: {e}")
|
||||
return {"status": "error", "error": str(e)}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,681 +0,0 @@
|
||||
"""
|
||||
Health check routes.
|
||||
"""
|
||||
import asyncio
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter, Depends
|
||||
from typing import Dict, Any, Optional, Tuple
|
||||
|
||||
from backend.version import VERSION, COMMIT_SHA, PR_NUMBER, PR_TITLE, STARTUP_TIME
|
||||
from backend.api.dependencies import optional_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.encoding_worker_preference import infer_machine_type, PRIMARY_MACHINE_TYPE
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
from backend.services.job_health_service import check_job_consistency_detailed
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.models.job import JobStatus
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
import transmission_rpc
|
||||
host = os.environ.get("TRANSMISSION_HOST", "localhost")
|
||||
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
|
||||
|
||||
client = transmission_rpc.Client(host=host, port=port, timeout=5)
|
||||
stats = client.session_stats()
|
||||
|
||||
# Get detailed torrent info
|
||||
torrents = client.get_torrents()
|
||||
torrent_details = []
|
||||
for t in torrents:
|
||||
torrent_info = {
|
||||
"name": t.name,
|
||||
"progress": round(t.progress, 1),
|
||||
"status": str(t.status),
|
||||
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
|
||||
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
|
||||
}
|
||||
# Check if stalled (downloading but no progress and no peers)
|
||||
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
|
||||
torrent_info['stalled'] = True
|
||||
torrent_details.append(torrent_info)
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"host": host,
|
||||
"port": port,
|
||||
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
|
||||
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
|
||||
"torrents": torrent_details,
|
||||
}
|
||||
except ImportError as e:
|
||||
return {
|
||||
"available": False,
|
||||
"error": f"transmission_rpc not installed: {e}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"available": False,
|
||||
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
|
||||
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health_check() -> Dict[str, str]:
|
||||
"""Health check endpoint."""
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
|
||||
async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
"""Check if remote flacfetch service is available and healthy."""
|
||||
client = get_flacfetch_client()
|
||||
|
||||
if not client:
|
||||
return {
|
||||
"configured": False,
|
||||
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await client.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"available": True,
|
||||
"status": health.get("status"),
|
||||
"version": health.get("version"),
|
||||
"transmission": health.get("transmission", {}),
|
||||
"disk": health.get("disk", {}),
|
||||
"providers": health.get("providers", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = await asyncio.to_thread(get_encoding_service)
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/flacfetch")
|
||||
async def flacfetch_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check flacfetch service status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_flacfetch_service_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("version"),
|
||||
}
|
||||
|
||||
|
||||
def check_audio_separator_status() -> Dict[str, Any]:
|
||||
"""Report audio separation availability.
|
||||
|
||||
Separation runs in the ``audio-separation-job`` GPU Cloud Run Job (local
|
||||
GPU mode, models baked into the image). The standalone ``audio-separator``
|
||||
Cloud Run GPU service was shut down on 2026-10-01 (cost cut), so there is
|
||||
no remote service to probe — this reports the installed package version
|
||||
only and never calls out to anything.
|
||||
"""
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
pkg_version = version("audio-separator")
|
||||
except Exception:
|
||||
pkg_version = None
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"mode": "gpu_job",
|
||||
"version": pkg_version,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/audio-separator")
|
||||
def audio_separator_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Return audio separation status (installed audio-separator package version).
|
||||
|
||||
Does NOT call any remote service — separation runs in the
|
||||
audio-separation-job GPU Cloud Run Job. No authentication required.
|
||||
"""
|
||||
return check_audio_separator_status()
|
||||
|
||||
|
||||
def _get_encoding_worker_manager():
|
||||
"""Get the EncodingWorkerManager instance. Separated for testability."""
|
||||
try:
|
||||
encoding_service = get_encoding_service()
|
||||
if hasattr(encoding_service, '_worker_manager') and encoding_service._worker_manager:
|
||||
return encoding_service._worker_manager
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
@router.get("/health/system-status")
|
||||
async def system_status(
|
||||
auth: Optional[Tuple] = Depends(optional_auth),
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Aggregated system status for all services.
|
||||
|
||||
Returns basic info for everyone. Admin users get additional
|
||||
blue-green deployment details for the encoder.
|
||||
"""
|
||||
is_admin = auth is not None and len(auth) >= 2 and auth[1] == UserType.ADMIN
|
||||
|
||||
# Fetch all health statuses concurrently
|
||||
import asyncio
|
||||
encoding_task = check_encoding_worker_status()
|
||||
flacfetch_task = check_flacfetch_service_status()
|
||||
encoding_status, flacfetch_status = await asyncio.gather(
|
||||
encoding_task, flacfetch_task
|
||||
)
|
||||
separator_status = check_audio_separator_status()
|
||||
|
||||
# Build frontend service info (from env vars baked at build time — returned as-is)
|
||||
frontend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION, # Frontend and backend share the same version from pyproject.toml
|
||||
}
|
||||
|
||||
# Build backend service info
|
||||
backend_svc = {
|
||||
"status": "ok",
|
||||
"version": VERSION,
|
||||
"deployed_at": STARTUP_TIME,
|
||||
}
|
||||
if COMMIT_SHA:
|
||||
backend_svc["commit_sha"] = COMMIT_SHA
|
||||
if PR_NUMBER:
|
||||
backend_svc["pr_number"] = PR_NUMBER
|
||||
if PR_TITLE:
|
||||
backend_svc["pr_title"] = PR_TITLE
|
||||
|
||||
# Build encoder service info
|
||||
encoder_svc = {
|
||||
"status": "ok" if encoding_status.get("available") else "offline",
|
||||
"version": encoding_status.get("wheel_version"),
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
}
|
||||
|
||||
# Build flacfetch service info
|
||||
flacfetch_svc = {
|
||||
"status": "ok" if flacfetch_status.get("available") else "offline",
|
||||
"version": flacfetch_status.get("version"),
|
||||
}
|
||||
|
||||
# Build separator service info
|
||||
separator_svc = {
|
||||
"status": "ok" if separator_status.get("available") else "offline",
|
||||
"version": separator_status.get("version"),
|
||||
}
|
||||
|
||||
# Admin-only: blue-green encoder details from Firestore
|
||||
if is_admin:
|
||||
manager = _get_encoding_worker_manager()
|
||||
if manager:
|
||||
try:
|
||||
config = manager.get_config()
|
||||
|
||||
# Which VM is actually serving right now? When a capacity
|
||||
# fallback is active (e.g. a c4d stockout routed us to an
|
||||
# alternate family) the override points at the live VM;
|
||||
# otherwise it's the blue-green primary. The primary pair is
|
||||
# always c4d — fallbacks carry their family in the VM name
|
||||
# (…-fallback-c2df), which infer_machine_type
|
||||
# decodes when no explicit machine_type is stored on the config.
|
||||
on_fallback = bool(config.active_override_vm)
|
||||
active_vm = config.active_override_vm or config.primary_vm
|
||||
default_zone = getattr(manager, "_zone", "us-central1-c")
|
||||
active_zone = config.active_override_zone or default_zone
|
||||
if on_fallback:
|
||||
active_machine_type = infer_machine_type({"vm": active_vm})
|
||||
else:
|
||||
active_machine_type = (
|
||||
getattr(config, "primary_machine_type", None)
|
||||
or PRIMARY_MACHINE_TYPE
|
||||
)
|
||||
|
||||
encoder_svc["admin_details"] = {
|
||||
"primary_vm": config.primary_vm,
|
||||
"primary_ip": config.primary_ip,
|
||||
"primary_version": config.primary_version,
|
||||
"primary_deployed_at": config.primary_deployed_at,
|
||||
"secondary_vm": config.secondary_vm,
|
||||
"secondary_ip": config.secondary_ip,
|
||||
"secondary_version": config.secondary_version,
|
||||
"secondary_deployed_at": config.secondary_deployed_at,
|
||||
"last_swap_at": config.last_swap_at,
|
||||
"deploy_in_progress": config.deploy_in_progress,
|
||||
"active_jobs": encoding_status.get("active_jobs", 0),
|
||||
"queue_length": encoding_status.get("queue_length", 0),
|
||||
# Currently-serving worker (fallback-aware)
|
||||
"active_vm": active_vm,
|
||||
"active_zone": active_zone,
|
||||
"active_machine_type": active_machine_type,
|
||||
"on_fallback": on_fallback,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to get blue-green config: {e}")
|
||||
|
||||
# Add error details for offline services
|
||||
if not flacfetch_status.get("available"):
|
||||
flacfetch_svc["admin_details"] = {"error": flacfetch_status.get("error")}
|
||||
if not separator_status.get("available"):
|
||||
separator_svc["admin_details"] = {"error": separator_status.get("error")}
|
||||
if not encoding_status.get("available") and "admin_details" not in encoder_svc:
|
||||
encoder_svc["admin_details"] = {"error": encoding_status.get("error")}
|
||||
|
||||
return {
|
||||
"services": {
|
||||
"frontend": frontend_svc,
|
||||
"backend": backend_svc,
|
||||
"encoder": encoder_svc,
|
||||
"flacfetch": flacfetch_svc,
|
||||
"separator": separator_svc,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
email_status = {
|
||||
"configured": email_service.is_configured(),
|
||||
"provider": type(email_service.provider).__name__,
|
||||
}
|
||||
|
||||
# Check Stripe service
|
||||
stripe_service = get_stripe_service()
|
||||
stripe_status = {
|
||||
"configured": stripe_service.is_configured(),
|
||||
}
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
if not email_status["configured"]:
|
||||
logger.warning("Email service not configured - magic links will not work")
|
||||
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
"stripe": stripe_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/job-consistency")
|
||||
def job_consistency_check(
|
||||
status: Optional[str] = None,
|
||||
limit: int = 50
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Check job consistency across the system.
|
||||
|
||||
This endpoint identifies jobs with state inconsistencies that may indicate
|
||||
bugs in the state machine or trigger logic. Use this to find stuck jobs
|
||||
or jobs in invalid states.
|
||||
|
||||
Args:
|
||||
status: Optional filter by job status (e.g., "pending", "downloading")
|
||||
limit: Maximum number of jobs to check (default 50)
|
||||
|
||||
Returns:
|
||||
Dictionary with:
|
||||
- total_checked: Number of jobs checked
|
||||
- inconsistent_count: Number of jobs with issues
|
||||
- inconsistent_jobs: List of job details with their issues
|
||||
- summary: Summary of issue types found
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
|
||||
# Query jobs to check
|
||||
query_params = {}
|
||||
if status:
|
||||
try:
|
||||
job_status = JobStatus(status)
|
||||
query_params['status'] = job_status
|
||||
except ValueError:
|
||||
return {
|
||||
"error": f"Invalid status: {status}",
|
||||
"valid_statuses": [s.value for s in JobStatus]
|
||||
}
|
||||
|
||||
# Get jobs from Firestore
|
||||
jobs = job_manager.list_jobs(limit=limit, **query_params)
|
||||
|
||||
# Check each job for consistency
|
||||
inconsistent_jobs = []
|
||||
issue_summary = {}
|
||||
|
||||
for job in jobs:
|
||||
result = check_job_consistency_detailed(job)
|
||||
if not result['is_healthy']:
|
||||
inconsistent_jobs.append(result)
|
||||
# Count issue types
|
||||
for issue in result['issues']:
|
||||
issue_type = issue.split(':')[0] if ':' in issue else issue
|
||||
issue_summary[issue_type] = issue_summary.get(issue_type, 0) + 1
|
||||
|
||||
return {
|
||||
"total_checked": len(jobs),
|
||||
"inconsistent_count": len(inconsistent_jobs),
|
||||
"inconsistent_jobs": inconsistent_jobs,
|
||||
"summary": issue_summary,
|
||||
"status_filter": status,
|
||||
"limit": limit,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
return {
|
||||
"status": "ready",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -1,94 +0,0 @@
|
||||
"""POST /api/karaokehunt/request — intake for retired-KaraokeHunt-app requests.
|
||||
|
||||
A Cloudflare Worker on create.karaokehunt.com (the URL baked into the
|
||||
unchangeable app binary) answers the app instantly and forwards each payload
|
||||
here with a shared secret. This route validates + logs the payload, then runs
|
||||
the full conversion synchronously (see ``workers/karaokehunt_conversion``) —
|
||||
the Worker doesn't wait for us, so latency is a non-issue and uvicorn finishes
|
||||
the handler even if the forwarding subrequest is reaped.
|
||||
|
||||
Unauthenticated by necessity (the app can't auth), gated three ways: the
|
||||
shared secret (503 until configured — deploys dark), a per-IP rate limit, and
|
||||
the conversion worker's own daily job cap + per-song dedup.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import logging
|
||||
import time
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.config import get_settings
|
||||
from backend.services.error_monitor.frontend_ingestion import RateLimiter
|
||||
from backend.workers import karaokehunt_conversion
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/karaokehunt", tags=["karaokehunt"])
|
||||
|
||||
# ~1 legit request every 3 days historically; 10/min/IP is already generous.
|
||||
_limiter = RateLimiter(max_per_minute=10)
|
||||
|
||||
|
||||
def _client_ip(request: Request) -> str:
|
||||
# The Worker forwards the app's real IP; direct callers fall back to peer IP.
|
||||
return (request.headers.get("x-kh-client-ip")
|
||||
or (request.client.host if request.client else "unknown"))
|
||||
|
||||
|
||||
@router.post("/request", status_code=200)
|
||||
async def intake_karaokehunt_request(request: Request) -> dict:
|
||||
settings = get_settings()
|
||||
if not settings.karaokehunt_forwarder_secret:
|
||||
raise HTTPException(status_code=503, detail="not configured")
|
||||
|
||||
provided = request.headers.get("x-kh-forwarder-secret", "")
|
||||
if not hmac.compare_digest(provided, settings.karaokehunt_forwarder_secret):
|
||||
raise HTTPException(status_code=403, detail="forbidden")
|
||||
|
||||
if not _limiter.allow(_client_ip(request), time.monotonic()):
|
||||
raise HTTPException(status_code=429, detail="too many requests")
|
||||
|
||||
# The live app binary's exact schema is unverifiable (built from a
|
||||
# FlutterFlow export newer than any repo branch), so parse leniently:
|
||||
# any JSON object is accepted and logged; unusable ones become
|
||||
# outcome="invalid" audit rows rather than errors.
|
||||
try:
|
||||
payload = await request.json()
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("payload must be a JSON object")
|
||||
except Exception: # noqa: BLE001
|
||||
raise HTTPException(status_code=400, detail="invalid JSON")
|
||||
|
||||
doc = karaokehunt_conversion.create_intake(
|
||||
payload,
|
||||
client_ip=_client_ip(request),
|
||||
user_agent=request.headers.get("user-agent", "")[:512],
|
||||
)
|
||||
if doc["outcome"] == "invalid":
|
||||
return {"status": "success", "id": doc["id"]}
|
||||
|
||||
try:
|
||||
result = await karaokehunt_conversion.process_intake(doc["id"])
|
||||
logger.info("karaokehunt: processed %s -> %s", doc["id"], result.get("status"))
|
||||
except Exception: # noqa: BLE001 — the intake doc + reprocess endpoint carry retries
|
||||
logger.exception("karaokehunt: conversion failed for %s", doc["id"])
|
||||
try:
|
||||
karaokehunt_conversion._get_db().collection(
|
||||
karaokehunt_conversion.COLLECTION
|
||||
).document(doc["id"]).update({"outcome": "error", "error": "unhandled exception"})
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("karaokehunt: failed to mark %s as error", doc["id"])
|
||||
|
||||
# The app ignores the body; always succeed so nothing user-visible breaks.
|
||||
return {"status": "success", "id": doc["id"]}
|
||||
|
||||
|
||||
@router.post("/internal/reprocess/{doc_id}")
|
||||
async def reprocess_karaokehunt_request(
|
||||
doc_id: str, auth_result=Depends(require_admin)
|
||||
) -> dict:
|
||||
"""Admin retry for a failed/errored intake (idempotent markers prevent
|
||||
double credits/jobs). Also accepts force=true via query for terminal docs."""
|
||||
return await karaokehunt_conversion.process_intake(doc_id, force=True)
|
||||
@@ -1,625 +0,0 @@
|
||||
"""kjbox partner API — singer email-code sign-in + "free at the show" credits.
|
||||
|
||||
kjbox (the KJ device app at live karaoke nights) lets a patron verify their
|
||||
email inside the kjbox singer web page, then creates gen jobs AS that real gen
|
||||
user (with the returned session token) so they get normal gen delivery emails.
|
||||
|
||||
Every endpoint requires ``X-Kjbox-Secret`` (503 until ``KJBOX_PARTNER_SECRET``
|
||||
is configured — deploys dark; 403 if wrong). Contract:
|
||||
|
||||
- POST /api/kjbox/auth/send-code {email, locale?, venue?} → {"status": "sent"}
|
||||
- POST /api/kjbox/auth/verify-code {email, code} → {session_token, user,
|
||||
credits_granted, credit_status}
|
||||
- POST /api/kjbox/auth/google | apple {id_token, locale?, venue?} → same as
|
||||
verify-code (one-tap sign-in in the singer page)
|
||||
- POST /api/kjbox/credits/show-credit (+ Authorization: Bearer <session>)
|
||||
{idempotency_key, venue?, only_if_empty?}
|
||||
→ {granted, credits}
|
||||
|
||||
A venue is many singers behind one IP, so gen's 2-signups-per-IP cap does NOT
|
||||
apply here; instead a partner-wide rolling-24h cap on new VERIFIED accounts
|
||||
(accounts are only created at verify-code), a per-email code throttle, and a
|
||||
per-user daily show-credit cap bound abuse.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hmac
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request, Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.routes.users import (
|
||||
_build_user_public,
|
||||
_mask_email,
|
||||
_precompute_credit_eval,
|
||||
complete_verified_login,
|
||||
)
|
||||
from backend.config import get_settings, is_production
|
||||
from backend.i18n import SUPPORTED_LOCALES, t
|
||||
from backend.services.account_deletion_service import (
|
||||
AccountDeletionService,
|
||||
get_account_deletion_service,
|
||||
)
|
||||
from backend.services.email_service import EmailService, get_email_service
|
||||
from backend.services.email_validation_service import get_email_validation_service
|
||||
from backend.services.kjbox_partner_service import (
|
||||
CODE_EXPIRY_MINUTES,
|
||||
LOGIN_CODES_COLLECTION,
|
||||
KjboxPartnerService,
|
||||
CodeThrottled,
|
||||
ShowCreditCapReached,
|
||||
TransactionContention,
|
||||
VerifyOutcome,
|
||||
)
|
||||
from backend.services.user_service import UserService, get_user_service
|
||||
from backend.utils.request_helpers import get_client_ip
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/kjbox", tags=["kjbox"])
|
||||
|
||||
SIGNUP_SOURCE = "kjbox"
|
||||
_MAX_EMAIL_LEN = 254
|
||||
_MAX_VENUE_LEN = 200
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ auth gate
|
||||
|
||||
def require_kjbox_partner(x_kjbox_secret: Optional[str] = Header(default=None)) -> str:
|
||||
"""Shared-secret gate. Returns the secret (also used as the code-hash pepper)."""
|
||||
secret = get_settings().kjbox_partner_secret
|
||||
if not secret:
|
||||
raise HTTPException(status_code=503, detail="not configured")
|
||||
if not hmac.compare_digest((x_kjbox_secret or "").encode(), secret.encode()):
|
||||
raise HTTPException(status_code=403, detail="forbidden")
|
||||
return secret
|
||||
|
||||
|
||||
def require_resolve_partner(
|
||||
x_kjbox_secret: Optional[str] = Header(default=None),
|
||||
x_partner_secret: Optional[str] = Header(default=None),
|
||||
) -> str:
|
||||
"""Gate for ``catalog/resolve``: kjbox (``X-Kjbox-Secret``) or karaoke-decide
|
||||
(``X-Partner-Secret``). Returns the partner name ("kjbox" / "decide"), which
|
||||
picks the rate-budget bucket. 503 when neither secret is configured."""
|
||||
settings = get_settings()
|
||||
kjbox_secret = settings.kjbox_partner_secret
|
||||
decide_secret = getattr(settings, "decide_partner_secret", "")
|
||||
if not kjbox_secret and not decide_secret:
|
||||
raise HTTPException(status_code=503, detail="not configured")
|
||||
if kjbox_secret and x_kjbox_secret and hmac.compare_digest(
|
||||
x_kjbox_secret.encode(), kjbox_secret.encode()):
|
||||
return "kjbox"
|
||||
if decide_secret and x_partner_secret and hmac.compare_digest(
|
||||
x_partner_secret.encode(), decide_secret.encode()):
|
||||
return "decide"
|
||||
raise HTTPException(status_code=403, detail="forbidden")
|
||||
|
||||
|
||||
def get_kjbox_service(
|
||||
secret: str = Depends(require_kjbox_partner),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
) -> KjboxPartnerService:
|
||||
return KjboxPartnerService(user_service.db, pepper=secret)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- models
|
||||
|
||||
class SendCodeRequest(BaseModel):
|
||||
email: str
|
||||
locale: Optional[str] = None
|
||||
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
|
||||
|
||||
|
||||
class SendCodeResponse(BaseModel):
|
||||
status: str
|
||||
|
||||
|
||||
class VerifyCodeRequest(BaseModel):
|
||||
email: str
|
||||
code: str = Field(max_length=32)
|
||||
|
||||
|
||||
class SocialSignInRequest(BaseModel):
|
||||
# Google Identity Services / Sign in with Apple JS ID token from the singer page
|
||||
# (or ``fake:<email>`` on the local stack).
|
||||
id_token: str = Field(min_length=1, max_length=8192)
|
||||
locale: Optional[str] = None
|
||||
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
|
||||
|
||||
|
||||
class VerifyCodeResponse(BaseModel):
|
||||
session_token: str
|
||||
user: dict
|
||||
credits_granted: int
|
||||
credit_status: str
|
||||
|
||||
|
||||
class ShowCreditRequest(BaseModel):
|
||||
idempotency_key: str = Field(min_length=1, max_length=128)
|
||||
venue: Optional[str] = Field(default=None, max_length=_MAX_VENUE_LEN)
|
||||
# Grant only if the singer currently has no credits (< 1). The key is claimed
|
||||
# only when a credit is actually granted, so a later call with the same key
|
||||
# and only_if_empty=false grants iff this one didn't.
|
||||
only_if_empty: bool = False
|
||||
|
||||
|
||||
class ShowCreditResponse(BaseModel):
|
||||
granted: bool
|
||||
credits: int
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- helpers
|
||||
|
||||
def _normalize_email(raw: str) -> str:
|
||||
email = (raw or "").strip().lower()
|
||||
local, _, domain = email.partition("@")
|
||||
if (not local or "." not in domain or len(email) > _MAX_EMAIL_LEN
|
||||
or any(c.isspace() for c in email) or email.count("@") != 1):
|
||||
raise HTTPException(status_code=422, detail="invalid_email")
|
||||
return email
|
||||
|
||||
|
||||
def _split_locale(raw: Optional[str]) -> tuple[Optional[str], Optional[str]]:
|
||||
"""Return (email_locale in en/es/de or None, ui_locale primary subtag or None)."""
|
||||
if not raw:
|
||||
return None, None
|
||||
primary = raw.strip().replace("_", "-").split("-")[0].lower()
|
||||
if not primary.isalpha() or len(primary) < 2:
|
||||
return None, None
|
||||
return (primary if primary in SUPPORTED_LOCALES else None), primary
|
||||
|
||||
|
||||
def _pretend_sent() -> SendCodeResponse:
|
||||
return SendCodeResponse(status="sent")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ endpoints
|
||||
|
||||
@router.post("/auth/send-code", response_model=SendCodeResponse)
|
||||
def send_code(
|
||||
body: SendCodeRequest,
|
||||
http_request: Request,
|
||||
svc: KjboxPartnerService = Depends(get_kjbox_service),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
settings = get_settings()
|
||||
email = _normalize_email(body.email)
|
||||
email_locale, ui_locale = _split_locale(body.locale)
|
||||
locale = email_locale or "en"
|
||||
venue = (body.venue or "").strip() or None
|
||||
|
||||
email_validation = get_email_validation_service()
|
||||
if email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"kjbox: blocked disposable email {_mask_email(email)}")
|
||||
raise HTTPException(status_code=422, detail=t(locale, "users.disposableEmailBlocked"))
|
||||
if email_validation.is_email_blocked(email):
|
||||
logger.warning(f"kjbox: blocked email {_mask_email(email)} — pretending success")
|
||||
return _pretend_sent()
|
||||
ip_address = get_client_ip(http_request)
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"kjbox: blocked IP {ip_address} — pretending success")
|
||||
return _pretend_sent()
|
||||
|
||||
if not email_service.is_configured() and is_production():
|
||||
logger.error("kjbox: email service not configured - cannot send codes")
|
||||
raise HTTPException(status_code=503, detail="email not configured")
|
||||
|
||||
# No account is created here — only once the email is verified (verify-code),
|
||||
# so unverified addresses can neither burn the signup cap nor create accounts
|
||||
# for other people's emails. Venue/locale ride on the code record until then.
|
||||
user = user_service.get_user(email)
|
||||
try:
|
||||
issued = svc.issue_code(
|
||||
email,
|
||||
max_per_hour=settings.kjbox_codes_per_email_per_hour,
|
||||
needs_credit_eval=(
|
||||
not (user and getattr(user, "welcome_credits_granted", False))
|
||||
and not is_test_email(email)
|
||||
),
|
||||
locale=email_locale,
|
||||
ui_locale=ui_locale,
|
||||
venue=venue,
|
||||
ip_address=ip_address,
|
||||
user_agent=http_request.headers.get("user-agent"),
|
||||
)
|
||||
except (CodeThrottled, TransactionContention):
|
||||
# Contention on one email's code doc = a burst of parallel sends to the
|
||||
# same address, which is exactly what the throttle exists to stop.
|
||||
raise HTTPException(status_code=429, detail="too_many_codes")
|
||||
|
||||
sent = email_service.send_kjbox_login_code(
|
||||
email, issued.code, expiry_minutes=CODE_EXPIRY_MINUTES, locale=locale,
|
||||
)
|
||||
if not sent:
|
||||
if getattr(email_service, "last_send_suppressed", False):
|
||||
logger.info(f"kjbox: code recipient {_mask_email(email)} is Postmark-suppressed")
|
||||
else:
|
||||
logger.error(f"kjbox: failed to send code email to {_mask_email(email)}")
|
||||
|
||||
# Warm the welcome-credit AI evaluation while the singer reads their email,
|
||||
# so verify-code is instant (same optimisation as magic links).
|
||||
if issued.needs_credit_eval:
|
||||
threading.Thread(
|
||||
target=_precompute_credit_eval,
|
||||
args=(issued.doc_id, email, LOGIN_CODES_COLLECTION),
|
||||
daemon=True,
|
||||
).start()
|
||||
|
||||
return SendCodeResponse(status="sent")
|
||||
|
||||
|
||||
@router.post("/auth/verify-code", response_model=VerifyCodeResponse)
|
||||
def verify_code(
|
||||
body: VerifyCodeRequest,
|
||||
http_request: Request,
|
||||
svc: KjboxPartnerService = Depends(get_kjbox_service),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
settings = get_settings()
|
||||
email = _normalize_email(body.email)
|
||||
|
||||
# New accounts are created (and counted against the partner-wide cap) only
|
||||
# here, after the email is proven. Check the cap BEFORE consuming the code so
|
||||
# a capped singer can retry the same code later.
|
||||
existing = user_service.get_user(email)
|
||||
if existing is None and svc.count_recent_signups() >= settings.kjbox_signup_cap_per_24h:
|
||||
logger.warning("kjbox: partner-wide verified-signup cap reached")
|
||||
raise HTTPException(status_code=429, detail="signup_cap")
|
||||
|
||||
outcome, record = svc.verify_code(email, body.code)
|
||||
if outcome == VerifyOutcome.TOO_MANY_ATTEMPTS:
|
||||
raise HTTPException(status_code=429, detail="too_many_attempts")
|
||||
if outcome == VerifyOutcome.INVALID:
|
||||
raise HTTPException(status_code=401, detail="invalid_code")
|
||||
if outcome != VerifyOutcome.OK or record is None:
|
||||
raise HTTPException(status_code=401, detail="expired")
|
||||
|
||||
precomputed_eval = {
|
||||
k: record.get(k)
|
||||
for k in ("credit_eval_decision", "credit_eval_reasoning", "credit_eval_error")
|
||||
if record.get(k) is not None
|
||||
} or None
|
||||
return _finish_sign_in(
|
||||
email, http_request, svc=svc, user_service=user_service, email_service=email_service,
|
||||
venue=record.get("venue"), email_locale=record.get("locale"),
|
||||
ui_locale=record.get("ui_locale"), precomputed_eval=precomputed_eval, how="verified",
|
||||
)
|
||||
|
||||
|
||||
def _finish_sign_in(email, http_request, *, svc, user_service, email_service, venue,
|
||||
email_locale, ui_locale, precomputed_eval, how):
|
||||
"""A proven email (code, or a Google/Apple ID token) → the singer's gen session."""
|
||||
user = user_service.get_user(email)
|
||||
if user is None:
|
||||
# No signup_ip: the caller is the kjbox box, whose IP is the whole
|
||||
# venue's — recording it would trip gen's per-IP cap for everyone there.
|
||||
user = user_service.get_or_create_user(
|
||||
email, signup_source=SIGNUP_SOURCE, signup_venue=venue,
|
||||
)
|
||||
svc.record_signup(email, venue)
|
||||
logger.info(f"kjbox: created verified gen user {_mask_email(email)} (venue={venue!r}, via {how})")
|
||||
# Same first-login definition as magic-link verify (checked before last_login_at is set).
|
||||
is_first_login = user.total_jobs_created == 0 and not user.last_login_at
|
||||
user = user_service.update_user(
|
||||
email, email_verified=True, last_login_at=datetime.utcnow(),
|
||||
) or user
|
||||
|
||||
result = complete_verified_login(
|
||||
user,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
is_first_login=is_first_login,
|
||||
precomputed_eval=precomputed_eval,
|
||||
grant_welcome_credit=True,
|
||||
referral_code=None,
|
||||
locale=email_locale,
|
||||
ui_locale=ui_locale,
|
||||
ip_address=get_client_ip(http_request),
|
||||
user_agent=http_request.headers.get("user-agent"),
|
||||
tenant_id=None,
|
||||
device_fingerprint=None,
|
||||
email_locale=email_locale or user.locale or "en",
|
||||
)
|
||||
logger.info(f"kjbox: {how} {_mask_email(email)} (credit_status={result.credit_status})")
|
||||
|
||||
return VerifyCodeResponse(
|
||||
session_token=result.session.token,
|
||||
user=_build_user_public(result.user).model_dump(mode="json"),
|
||||
credits_granted=result.credits_granted,
|
||||
credit_status=result.credit_status,
|
||||
)
|
||||
|
||||
|
||||
def _social_sign_in(provider, body, http_request, svc, user_service, email_service):
|
||||
"""Google / Apple one-tap: the provider has proven the email, so no code.
|
||||
Same venue rules as verify-code: no per-IP signup cap, the partner-wide 24h
|
||||
cap on new accounts instead. Apple "Hide My Email" relays are their own account."""
|
||||
from backend.api.routes.login_code import APPLE_RELAY_DOMAIN, _verify_identity
|
||||
|
||||
identity = _verify_identity(provider, body.id_token) # 401 invalid_token / 503
|
||||
email = _normalize_email(identity.email)
|
||||
email_locale, ui_locale = _split_locale(body.locale)
|
||||
|
||||
# The same eligibility checks as send-code. There's no email to hide a
|
||||
# refusal behind, so (like the app's social sign-in) refusals are explicit.
|
||||
email_validation = get_email_validation_service()
|
||||
is_relay = provider == "apple" and email.endswith("@" + APPLE_RELAY_DOMAIN)
|
||||
if not is_relay and email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"kjbox: {provider} sign-in refused, disposable email {_mask_email(email)}")
|
||||
raise HTTPException(status_code=422, detail=t(email_locale or "en", "users.disposableEmailBlocked"))
|
||||
ip_address = get_client_ip(http_request)
|
||||
if email_validation.is_email_blocked(email) or (ip_address and email_validation.is_ip_blocked(ip_address)):
|
||||
logger.warning(f"kjbox: {provider} sign-in refused, blocked email/IP {_mask_email(email)}")
|
||||
raise HTTPException(status_code=422, detail="sign_in_not_allowed")
|
||||
if user_service.get_user(email) is None and \
|
||||
svc.count_recent_signups() >= get_settings().kjbox_signup_cap_per_24h:
|
||||
logger.warning("kjbox: partner-wide verified-signup cap reached")
|
||||
raise HTTPException(status_code=429, detail="signup_cap")
|
||||
return _finish_sign_in(
|
||||
email, http_request, svc=svc, user_service=user_service, email_service=email_service,
|
||||
venue=(body.venue or None), email_locale=email_locale, ui_locale=ui_locale,
|
||||
precomputed_eval=None, how=f"{provider} sign-in",
|
||||
)
|
||||
|
||||
|
||||
@router.post("/auth/google", response_model=VerifyCodeResponse)
|
||||
def google_sign_in(
|
||||
body: SocialSignInRequest,
|
||||
http_request: Request,
|
||||
svc: KjboxPartnerService = Depends(get_kjbox_service),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Google ID token (audience ``GOOGLE_OAUTH_CLIENT_IDS``) → singer session."""
|
||||
return _social_sign_in("google", body, http_request, svc, user_service, email_service)
|
||||
|
||||
|
||||
@router.post("/auth/apple", response_model=VerifyCodeResponse)
|
||||
def apple_sign_in(
|
||||
body: SocialSignInRequest,
|
||||
http_request: Request,
|
||||
svc: KjboxPartnerService = Depends(get_kjbox_service),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Apple ID token (audience ``APPLE_SIGNIN_AUDIENCES``, incl. the web Services ID) → singer session."""
|
||||
return _social_sign_in("apple", body, http_request, svc, user_service, email_service)
|
||||
|
||||
|
||||
@router.post("/credits/show-credit", response_model=ShowCreditResponse)
|
||||
def show_credit(
|
||||
body: ShowCreditRequest,
|
||||
svc: KjboxPartnerService = Depends(get_kjbox_service),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
authorization: Optional[str] = Header(default=None),
|
||||
):
|
||||
"""Quietly grant +1 credit so a singer's make-it job is free at the show.
|
||||
|
||||
No "credits added" email. Idempotent per ``idempotency_key`` (claimed only
|
||||
when a credit is granted); ``only_if_empty`` grants only at a 0 balance;
|
||||
capped per user per rolling 24h. All decisions + the increment are one
|
||||
Firestore transaction.
|
||||
"""
|
||||
settings = get_settings()
|
||||
scheme, _, token = (authorization or "").partition(" ")
|
||||
token = token.strip()
|
||||
if scheme.lower() != "bearer" or not token:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
valid, user, _msg = user_service.validate_session(token)
|
||||
if not valid or not user:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
|
||||
email = user.email.lower()
|
||||
venue = (body.venue or "").strip() or None
|
||||
user_ref = user_service.get_user_doc_ref(email)
|
||||
if user_ref is None:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
|
||||
try:
|
||||
result = svc.grant_show_credit(
|
||||
user_ref,
|
||||
email,
|
||||
body.idempotency_key,
|
||||
venue=venue,
|
||||
only_if_empty=body.only_if_empty,
|
||||
max_per_24h=settings.kjbox_show_credits_per_user_per_24h,
|
||||
)
|
||||
except ShowCreditCapReached:
|
||||
raise HTTPException(status_code=429, detail="show_credit_cap")
|
||||
except LookupError:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
except TransactionContention:
|
||||
# Nothing was written; kjbox retries with the same idempotency key.
|
||||
raise HTTPException(status_code=409, detail="busy_retry")
|
||||
|
||||
if result.granted:
|
||||
logger.info(f"kjbox: show credit granted to {_mask_email(email)} (venue={venue!r}) → {result.credits}")
|
||||
return ShowCreditResponse(granted=result.granted, credits=result.credits)
|
||||
|
||||
|
||||
# ------------------------------------------------------------- review link
|
||||
|
||||
class ReviewLinkRequest(BaseModel):
|
||||
locale: Optional[str] = None
|
||||
|
||||
|
||||
class ReviewLinkResponse(BaseModel):
|
||||
url: str
|
||||
status: str
|
||||
# "admin" = the KJ / support already opened the review; "owner" = the
|
||||
# singer did; None = nobody has opened it yet.
|
||||
review_started_by: Optional[str] = None
|
||||
|
||||
|
||||
_REVIEW_LINK_STATUSES = ("awaiting_review", "in_review")
|
||||
|
||||
|
||||
# ------------------------------------------------- account deletion queue
|
||||
|
||||
@router.get("/account-deletions")
|
||||
def list_account_deletions(
|
||||
since: Optional[datetime] = Query(default=None),
|
||||
_secret: str = Depends(require_kjbox_partner),
|
||||
deletion: AccountDeletionService = Depends(get_account_deletion_service),
|
||||
):
|
||||
"""Deleted gen accounts the box still has to erase (it pulls: it may be offline).
|
||||
|
||||
Only the sha256 of the email is sent; the box hashes the ``gen_email`` values
|
||||
it stores to match. Lists entries still pending for kjbox, created after ``since``.
|
||||
"""
|
||||
return {"deletions": deletion.list_kjbox_pending(since)}
|
||||
|
||||
|
||||
@router.post("/account-deletions/{entry_id}/ack", status_code=204)
|
||||
def ack_account_deletion(
|
||||
entry_id: str,
|
||||
_secret: str = Depends(require_kjbox_partner),
|
||||
deletion: AccountDeletionService = Depends(get_account_deletion_service),
|
||||
):
|
||||
"""The box erased this account's data; stop listing it. ``entry_id`` = sha256(email)."""
|
||||
if not deletion.ack_kjbox(entry_id):
|
||||
raise HTTPException(status_code=404, detail="not_found")
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/review-link", response_model=ReviewLinkResponse)
|
||||
async def review_link(
|
||||
job_id: str,
|
||||
body: ReviewLinkRequest,
|
||||
_secret: str = Depends(require_kjbox_partner),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
authorization: Optional[str] = Header(default=None),
|
||||
):
|
||||
"""One-click sign-in link straight to the singer's own job's lyrics review.
|
||||
|
||||
Backs the "Tap here to review it yourself" link on the kjbox singer page:
|
||||
the same single-use ``job_review:<id>`` magic link the kjbox review email
|
||||
carries, minted on demand. Only the job's owner (the Bearer session) may
|
||||
get one, and only while the job is in review (409 ``not_in_review``).
|
||||
"""
|
||||
scheme, _, token = (authorization or "").partition(" ")
|
||||
token = token.strip()
|
||||
if scheme.lower() != "bearer" or not token:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
locale = body.locale if body.locale in SUPPORTED_LOCALES else "en"
|
||||
# Session check, job read and token mint are blocking Firestore calls —
|
||||
# keep them off the event loop (it serves every in-flight request).
|
||||
url, status, started_by = await asyncio.to_thread(
|
||||
_mint_review_link, user_service, token, job_id, locale)
|
||||
return ReviewLinkResponse(url=url, status=status, review_started_by=started_by)
|
||||
|
||||
|
||||
def get_owned_review_job(user_service: UserService, session_token: str, job_id: str):
|
||||
"""Blocking: the session owner's job, checked to be in review → (job, status).
|
||||
|
||||
Shared by kjbox's review link and the mobile app's web handoff
|
||||
(``POST /api/users/auth/handoff-token``). Raises 401 ``invalid_session``,
|
||||
404 ``not_found`` (missing or not yours — same answer, so other users' job
|
||||
ids aren't confirmed) or 409 ``not_in_review``.
|
||||
"""
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
valid, user, _msg = user_service.validate_session(session_token)
|
||||
if not valid or not user:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
job = JobManager().get_job(job_id)
|
||||
if not job or (job.user_email or "").lower() != user.email.lower():
|
||||
raise HTTPException(status_code=404, detail="not_found")
|
||||
status = str(getattr(job.status, "value", job.status))
|
||||
if status not in _REVIEW_LINK_STATUSES:
|
||||
raise HTTPException(status_code=409, detail="not_in_review")
|
||||
return job, status
|
||||
|
||||
|
||||
def _mint_review_link(user_service: UserService, token: str, job_id: str, locale: str):
|
||||
"""Blocking half of ``review_link`` → (url, status, review_started_by)."""
|
||||
from backend.services.job_notification_service import build_review_login_url
|
||||
|
||||
job, status = get_owned_review_job(user_service, token, job_id)
|
||||
url = build_review_login_url(job.job_id, job.user_email, locale=locale)
|
||||
if not url:
|
||||
raise HTTPException(status_code=502, detail="link_unavailable")
|
||||
started_by = (job.state_data or {}).get("review_started_by") if status == "in_review" else None
|
||||
return url, status, started_by
|
||||
|
||||
|
||||
# ---------------------------------------------------------- search resolve
|
||||
|
||||
class ResolveRequest(BaseModel):
|
||||
query: str = Field(min_length=1, max_length=200)
|
||||
|
||||
|
||||
_RESOLVE_CACHE_MAX = 2000
|
||||
_RESOLVE_PER_MINUTE = 120
|
||||
_resolve_cache: "dict[str, dict]" = {}
|
||||
_RESOLVE_DECIDE_PER_MINUTE = 60 # karaoke-decide's own bucket, so kjbox nights are never starved
|
||||
_resolve_calls: "list[float]" = []
|
||||
_resolve_calls_decide: "list[float]" = []
|
||||
_resolve_lock = threading.Lock()
|
||||
|
||||
|
||||
def _resolve_budget_ok(partner: str = "kjbox") -> bool:
|
||||
"""Per-partner in-process cap on AI calls (cache hits are free)."""
|
||||
import time
|
||||
now = time.monotonic()
|
||||
if partner == "decide":
|
||||
calls, limit = _resolve_calls_decide, _RESOLVE_DECIDE_PER_MINUTE
|
||||
else:
|
||||
calls, limit = _resolve_calls, _RESOLVE_PER_MINUTE
|
||||
with _resolve_lock:
|
||||
while calls and calls[0] < now - 60:
|
||||
calls.pop(0)
|
||||
if len(calls) >= limit:
|
||||
return False
|
||||
calls.append(now)
|
||||
return True
|
||||
|
||||
|
||||
@router.post("/catalog/resolve")
|
||||
async def catalog_resolve(body: ResolveRequest, partner: str = Depends(require_resolve_partner)):
|
||||
"""Split + auto-correct a singer's free-text song search.
|
||||
|
||||
kjbox calls this only when its own catalogue search found nothing, then
|
||||
shows gen's AudioSourceStep "Tidied to … / keep what I typed" (re-searching
|
||||
when the song changed, and pre-filling its generate form). The AI splits the
|
||||
one-line query; the job flow's judge_match catalog pass then tidies it, so
|
||||
the canonical artist/title match what gen's own job form would use.
|
||||
Response = the match judge verdict shape + typed_artist / typed_title.
|
||||
Never 5xx on model trouble: kind "none".
|
||||
"""
|
||||
from backend.services.match_judge.free_text import _none, resolve_and_tidy
|
||||
|
||||
query = " ".join(body.query.split())
|
||||
if not query:
|
||||
raise HTTPException(status_code=422, detail="empty_query")
|
||||
from backend.services.fake_workers import fake_workers_enabled
|
||||
if fake_workers_enabled(get_settings()):
|
||||
# Local E2E stack: scripted verdict, no Gemini, no cache/budget (tests change the knobs).
|
||||
from backend.services.fake_workers.judge import fake_resolve
|
||||
return fake_resolve(query)
|
||||
key = query.casefold() # shared by both partners: same query -> same verdict
|
||||
with _resolve_lock:
|
||||
hit = _resolve_cache.get(key)
|
||||
if hit is not None:
|
||||
# Same query modulo case/spacing: the verdict applies, but echo THIS
|
||||
# request's text where the verdict fell back to the typed query.
|
||||
if not hit.get("typed_artist"):
|
||||
hit = {**hit, "typed_title": query}
|
||||
return hit
|
||||
if not _resolve_budget_ok(partner):
|
||||
raise HTTPException(status_code=429, detail="rate_limited")
|
||||
verdict = await resolve_and_tidy(query)
|
||||
if verdict.get("reason") != "unavailable": # don't cache transient failures
|
||||
with _resolve_lock:
|
||||
if len(_resolve_cache) >= _RESOLVE_CACHE_MAX:
|
||||
_resolve_cache.pop(next(iter(_resolve_cache)))
|
||||
_resolve_cache[key] = verdict
|
||||
return verdict or _none(query)
|
||||
@@ -1,628 +0,0 @@
|
||||
"""gen app sign-in: email code (``send-code`` + ``verify-code``), Google / Apple
|
||||
ID tokens, and the app → web handoff link.
|
||||
|
||||
An alternative to the magic link for clients that can't follow an emailed link
|
||||
back into the same session (the mobile app, a second device, local E2E). The
|
||||
email carries BOTH a 6-digit code and a regular magic link; either signs in.
|
||||
|
||||
- POST /api/users/auth/send-code {email, locale?, device_fingerprint?,
|
||||
referral_code?, attribution?} → 202 {"status": "sent"}
|
||||
- POST /api/users/auth/verify-code {email, code} → VerifyMagicLinkResponse
|
||||
- POST /api/users/auth/google | apple {id_token, locale?, device_fingerprint?,
|
||||
referral_code?, attribution?} → VerifyMagicLinkResponse
|
||||
- POST /api/users/auth/handoff-token {target, job_id, locale?} (Bearer)
|
||||
→ {url, expires_in_seconds}: the mobile app's
|
||||
2-minute, single-use signed-in web link
|
||||
|
||||
Code mechanics come from :class:`LoginCodeStore` (shared with kjbox): HMAC-hashed
|
||||
at rest with a server pepper, 10-minute TTL, 5 verify attempts then 429,
|
||||
single-use marker, and per-email + per-IP rolling-hour send throttles checked in
|
||||
the same Firestore transaction as the code write.
|
||||
|
||||
Abuse checks mirror ``POST /auth/magic-link``: disposable domain → 422 (unless a
|
||||
tenant explicitly allowlisted the address), blocked email / IP → pretend
|
||||
success, new-user per-IP/fingerprint signup cap → pretend success, tenant
|
||||
email-domain restriction → pretend success, no email provider in production →
|
||||
503. Pretend-success responses are indistinguishable from a real send.
|
||||
|
||||
Account creation: the email's magic link is minted with the normal
|
||||
``UserService.create_magic_link`` machinery, which (exactly like the magic-link
|
||||
endpoint) creates the account at send time, recording signup IP, device
|
||||
fingerprint and first-touch attribution. That keeps magic-link parity (same
|
||||
signup caps, same audit fields) instead of inventing a lazily-minted link.
|
||||
verify-code still re-creates the account from the code record if it's gone.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import threading
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Annotated, Dict, Literal, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, HTTPException, Request
|
||||
from pydantic import BaseModel, Field, StringConstraints
|
||||
|
||||
from backend.api.routes.kjbox import _normalize_email, _split_locale, get_owned_review_job
|
||||
from backend.api.routes.users import (
|
||||
_build_user_public,
|
||||
_mask_email,
|
||||
_precompute_credit_eval,
|
||||
complete_verified_login,
|
||||
)
|
||||
from backend.config import get_settings, is_production
|
||||
from backend.i18n import WEB_LOCALES, get_full_locale_from_request, get_locale_from_request, t
|
||||
from backend.models.user import VerifyMagicLinkResponse
|
||||
from backend.services.email_service import EmailService, get_email_service
|
||||
from backend.services.email_validation_service import get_email_validation_service
|
||||
from backend.services.fake_workers import fake_workers_enabled
|
||||
from backend.services import social_identity
|
||||
from backend.services.login_code_store import (
|
||||
CODE_EXPIRY_MINUTES,
|
||||
CodeThrottled,
|
||||
LoginCodeStore,
|
||||
TransactionContention,
|
||||
VerifyOutcome,
|
||||
)
|
||||
from backend.services.user_service import MAGIC_LINKS_COLLECTION, UserService, get_user_service
|
||||
from backend.utils.request_helpers import get_client_ip
|
||||
from backend.utils.test_data import is_test_email
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/users", tags=["users"])
|
||||
|
||||
LOGIN_CODES_COLLECTION = "login_codes"
|
||||
LOGIN_CODE_USES_COLLECTION = "login_code_uses"
|
||||
LOGIN_CODE_IP_COLLECTION = "login_code_ip_sends"
|
||||
|
||||
# Used only outside production when neither LOGIN_CODE_PEPPER nor an admin token
|
||||
# is configured (local dev / unit tests). Never used in production.
|
||||
_DEV_PEPPER = "gen-login-code-dev-pepper"
|
||||
|
||||
|
||||
def _resolve_pepper() -> Optional[str]:
|
||||
"""HMAC pepper for stored codes.
|
||||
|
||||
Order: ``LOGIN_CODE_PEPPER`` → sha256("gen-login-code-pepper:" + first admin
|
||||
token) → (non-production only) a fixed dev pepper. Returns None in production
|
||||
with nothing configured (endpoints then 503). Rotating the first admin token
|
||||
without setting LOGIN_CODE_PEPPER invalidates outstanding codes (≤10 min).
|
||||
"""
|
||||
settings = get_settings()
|
||||
if settings.login_code_pepper:
|
||||
return settings.login_code_pepper
|
||||
tokens = [tok.strip() for tok in (settings.admin_tokens or "").split(",") if tok.strip()]
|
||||
if tokens:
|
||||
return hashlib.sha256(f"gen-login-code-pepper:{tokens[0]}".encode("utf-8")).hexdigest()
|
||||
if not is_production():
|
||||
return _DEV_PEPPER
|
||||
return None
|
||||
|
||||
|
||||
def get_login_code_store(user_service: UserService = Depends(get_user_service)) -> LoginCodeStore:
|
||||
pepper = _resolve_pepper()
|
||||
if not pepper:
|
||||
logger.error("login-code: no LOGIN_CODE_PEPPER / admin token configured in production")
|
||||
raise HTTPException(status_code=503, detail="not configured")
|
||||
return LoginCodeStore(
|
||||
user_service.db,
|
||||
pepper,
|
||||
LOGIN_CODES_COLLECTION,
|
||||
LOGIN_CODE_USES_COLLECTION,
|
||||
ip_collection=LOGIN_CODE_IP_COLLECTION,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- models
|
||||
|
||||
class SendLoginCodeRequest(BaseModel):
|
||||
email: str
|
||||
locale: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = Field(default=None, max_length=256)
|
||||
referral_code: Optional[str] = Field(default=None, max_length=64)
|
||||
# First-touch attribution, same shape/limits as SendMagicLinkRequest.attribution.
|
||||
attribution: Optional[
|
||||
Dict[Annotated[str, StringConstraints(max_length=32)], Optional[Annotated[str, StringConstraints(max_length=500)]]]
|
||||
] = Field(default=None, max_length=16)
|
||||
|
||||
|
||||
class SendLoginCodeResponse(BaseModel):
|
||||
status: str
|
||||
|
||||
|
||||
class VerifyLoginCodeRequest(BaseModel):
|
||||
email: str
|
||||
code: str = Field(max_length=32)
|
||||
|
||||
|
||||
def _sent() -> SendLoginCodeResponse:
|
||||
return SendLoginCodeResponse(status="sent")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ endpoints
|
||||
|
||||
@router.post("/auth/send-code", response_model=SendLoginCodeResponse, status_code=202)
|
||||
def send_login_code(
|
||||
body: SendLoginCodeRequest,
|
||||
http_request: Request,
|
||||
store: LoginCodeStore = Depends(get_login_code_store),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Email a 6-digit sign-in code (plus a magic link) to ``email``."""
|
||||
from backend.middleware.tenant import get_tenant_config_from_request, get_tenant_from_request
|
||||
|
||||
settings = get_settings()
|
||||
email = _normalize_email(body.email)
|
||||
email_locale, ui_locale = _split_locale(body.locale)
|
||||
has_accept_language = bool(http_request.headers.get("accept-language"))
|
||||
locale = email_locale or get_locale_from_request(http_request)
|
||||
ui_locale = ui_locale or get_full_locale_from_request(http_request)
|
||||
# Locale persisted on the user at verify: only an explicit choice (body or
|
||||
# Accept-Language), never the "en" fallback — that would clobber a stored one.
|
||||
persist_locale = email_locale or (locale if has_accept_language else None)
|
||||
|
||||
tenant_id = get_tenant_from_request(http_request)
|
||||
tenant_config = get_tenant_config_from_request(http_request)
|
||||
ip_address = get_client_ip(http_request)
|
||||
device_fingerprint = body.device_fingerprint
|
||||
|
||||
check = _sign_in_checks(
|
||||
email, http_request, user_service,
|
||||
device_fingerprint=device_fingerprint, tenant_config=tenant_config, log_prefix="login-code",
|
||||
)
|
||||
if check.refusal == "disposable":
|
||||
raise HTTPException(status_code=422, detail=t(locale, "users.disposableEmailBlocked"))
|
||||
if check.refusal == "tenant_disallowed":
|
||||
logger.warning(f"login-code: email not allowed for tenant {tenant_id}: {_mask_email(email)}")
|
||||
return _sent()
|
||||
if check.refusal: # blocked email / IP, new-user signup cap: indistinguishable from a send
|
||||
return _sent()
|
||||
existing_user = check.existing_user
|
||||
|
||||
if not email_service.is_configured() and is_production():
|
||||
logger.error("login-code: email service not configured - cannot send codes")
|
||||
raise HTTPException(status_code=503, detail=t(locale, "users.emailServiceNotConfigured"))
|
||||
|
||||
user_agent = http_request.headers.get("user-agent")
|
||||
try:
|
||||
issued = store.issue_code(
|
||||
email,
|
||||
max_per_hour=settings.login_codes_per_email_per_hour,
|
||||
needs_credit_eval=(
|
||||
not (existing_user and getattr(existing_user, "welcome_credits_granted", False))
|
||||
and not is_test_email(email)
|
||||
and not tenant_id
|
||||
),
|
||||
fields={
|
||||
"locale": persist_locale,
|
||||
"email_locale": locale,
|
||||
"ui_locale": ui_locale,
|
||||
"ip_address": ip_address,
|
||||
"user_agent": user_agent,
|
||||
"device_fingerprint": device_fingerprint,
|
||||
"tenant_id": tenant_id,
|
||||
"referral_code": body.referral_code,
|
||||
},
|
||||
throttle_ip=ip_address,
|
||||
max_per_ip_per_hour=settings.login_codes_per_ip_per_hour,
|
||||
)
|
||||
except (CodeThrottled, TransactionContention):
|
||||
# Contention on one email's code doc = a burst of parallel sends to the
|
||||
# same address, which is exactly what the throttle exists to stop.
|
||||
raise HTTPException(status_code=429, detail="too_many_codes")
|
||||
|
||||
# Same machinery as POST /auth/magic-link (creates the account at send time,
|
||||
# recording signup IP / fingerprint / attribution) — see module docstring.
|
||||
magic_link = user_service.create_magic_link(
|
||||
email,
|
||||
ip_address=ip_address,
|
||||
user_agent=user_agent,
|
||||
tenant_id=tenant_id,
|
||||
device_fingerprint=device_fingerprint,
|
||||
referral_code=body.referral_code,
|
||||
attribution=body.attribution,
|
||||
)
|
||||
magic_link_url = email_service.build_magic_link_url(
|
||||
magic_link.token,
|
||||
locale=locale,
|
||||
tenant_frontend_url=tenant_config.get_frontend_url() if tenant_config else None,
|
||||
referral_code=body.referral_code,
|
||||
)
|
||||
sent = email_service.send_login_code(
|
||||
email,
|
||||
issued.code,
|
||||
magic_link_url=magic_link_url,
|
||||
expiry_minutes=CODE_EXPIRY_MINUTES,
|
||||
locale=locale,
|
||||
sender_email=tenant_config.get_sender_email() if tenant_config else None,
|
||||
tenant_name=tenant_config.name if tenant_config else None,
|
||||
)
|
||||
if not sent:
|
||||
if getattr(email_service, "last_send_suppressed", False):
|
||||
logger.info(f"login-code: recipient {_mask_email(email)} is Postmark-suppressed")
|
||||
else:
|
||||
logger.error(f"login-code: failed to send code email to {_mask_email(email)}")
|
||||
# Don't reveal failure (anti-enumeration).
|
||||
|
||||
# Warm the welcome-credit AI evaluation while the user reads their email, on
|
||||
# both sign-in docs (code record + magic link) so either path is instant.
|
||||
if issued.needs_credit_eval:
|
||||
threading.Thread(
|
||||
target=_precompute_credit_eval,
|
||||
args=(issued.doc_id, email, LOGIN_CODES_COLLECTION, ((MAGIC_LINKS_COLLECTION, magic_link.token),)),
|
||||
daemon=True,
|
||||
).start()
|
||||
|
||||
return _sent()
|
||||
|
||||
|
||||
@router.post("/auth/verify-code", response_model=VerifyMagicLinkResponse)
|
||||
def verify_login_code(
|
||||
body: VerifyLoginCodeRequest,
|
||||
http_request: Request,
|
||||
store: LoginCodeStore = Depends(get_login_code_store),
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Exchange an emailed code for a session (same response as GET /auth/verify)."""
|
||||
email = _normalize_email(body.email)
|
||||
|
||||
settings = get_settings()
|
||||
demo_email = (settings.app_review_demo_email or "").strip().lower()
|
||||
demo_code = settings.app_review_demo_code or ""
|
||||
if demo_email and demo_code and email == demo_email:
|
||||
# App Review demo account: reviewers can't receive email, so this one
|
||||
# address signs in with the fixed code. Attempt limits still apply.
|
||||
outcome, record = store.verify_fixed_code(email, body.code, demo_code)
|
||||
if outcome == VerifyOutcome.OK:
|
||||
logger.info("app review demo login")
|
||||
else:
|
||||
outcome, record = store.verify_code(email, body.code)
|
||||
if outcome == VerifyOutcome.TOO_MANY_ATTEMPTS:
|
||||
raise HTTPException(status_code=429, detail="too_many_attempts")
|
||||
if outcome == VerifyOutcome.INVALID:
|
||||
raise HTTPException(status_code=401, detail="invalid_code")
|
||||
if outcome != VerifyOutcome.OK or record is None:
|
||||
raise HTTPException(status_code=401, detail="expired")
|
||||
|
||||
tenant_id = record.get("tenant_id")
|
||||
user = user_service.get_user(email)
|
||||
if user is None:
|
||||
# Normally created at send time by create_magic_link; recreate from the
|
||||
# code record (same signup audit fields) if it was removed meanwhile.
|
||||
user = user_service.get_or_create_user(
|
||||
email,
|
||||
tenant_id=tenant_id,
|
||||
signup_ip=record.get("ip_address"),
|
||||
device_fingerprint=record.get("device_fingerprint"),
|
||||
)
|
||||
precomputed_eval = {
|
||||
k: record.get(k)
|
||||
for k in ("credit_eval_decision", "credit_eval_reasoning", "credit_eval_error")
|
||||
if record.get(k) is not None
|
||||
} or None
|
||||
persist_locale = record.get("locale")
|
||||
return finish_verified_sign_in(
|
||||
user,
|
||||
http_request,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
tenant_id=tenant_id,
|
||||
precomputed_eval=precomputed_eval,
|
||||
referral_code=http_request.headers.get("x-referral-code") or record.get("referral_code"),
|
||||
persist_locale=persist_locale,
|
||||
ui_locale=record.get("ui_locale"),
|
||||
email_locale=persist_locale or record.get("email_locale"),
|
||||
device_fingerprint=record.get("device_fingerprint"),
|
||||
log_prefix="login-code",
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------- shared sign-in steps
|
||||
|
||||
@dataclass
|
||||
class SignInCheck:
|
||||
"""Outcome of :func:`_sign_in_checks`; ``refusal`` is None when sign-in may go ahead."""
|
||||
refusal: Optional[str] # disposable | blocked_email | blocked_ip | signup_limited | tenant_disallowed
|
||||
existing_user: object = None
|
||||
|
||||
|
||||
APPLE_RELAY_DOMAIN = "privaterelay.appleid.com"
|
||||
|
||||
|
||||
def _sign_in_checks(
|
||||
email: str,
|
||||
http_request: Request,
|
||||
user_service: UserService,
|
||||
*,
|
||||
device_fingerprint: Optional[str],
|
||||
tenant_config,
|
||||
log_prefix: str,
|
||||
allow_apple_relay: bool = False,
|
||||
) -> SignInCheck:
|
||||
"""Abuse / eligibility checks shared by every sign-in that can create an account
|
||||
(email code, Google, Apple), mirroring ``POST /auth/magic-link``: disposable
|
||||
domain (unless a tenant explicitly allowlisted the address, or it's an Apple
|
||||
"Hide My Email" relay on the Apple route), blocked email, blocked IP, new-user
|
||||
per-IP/fingerprint signup cap, tenant email-domain restriction. Blocking
|
||||
Firestore / DeBounce calls: run from sync routes only.
|
||||
"""
|
||||
email_validation = get_email_validation_service()
|
||||
vouched = bool(tenant_config and tenant_config.is_explicitly_allowlisted(email))
|
||||
if allow_apple_relay and email.rsplit("@", 1)[-1] == APPLE_RELAY_DOMAIN:
|
||||
vouched = True # Apple-issued forwarding address tied to a real Apple ID
|
||||
if not vouched and email_validation.is_disposable_domain(email):
|
||||
logger.warning(f"{log_prefix}: blocked disposable email {_mask_email(email)}")
|
||||
return SignInCheck("disposable")
|
||||
if email_validation.is_email_blocked(email):
|
||||
logger.warning(f"{log_prefix}: blocked email {_mask_email(email)}")
|
||||
return SignInCheck("blocked_email")
|
||||
ip_address = get_client_ip(http_request)
|
||||
if ip_address and email_validation.is_ip_blocked(ip_address):
|
||||
logger.warning(f"{log_prefix}: blocked IP {ip_address}")
|
||||
return SignInCheck("blocked_ip")
|
||||
existing_user = user_service.get_user(email)
|
||||
if existing_user is None and user_service.is_signup_rate_limited(
|
||||
ip_address=ip_address, device_fingerprint=device_fingerprint,
|
||||
):
|
||||
logger.warning(f"{log_prefix}: signup rate limit hit: IP={ip_address} for {_mask_email(email)}")
|
||||
return SignInCheck("signup_limited")
|
||||
if tenant_config and not tenant_config.is_email_allowed(email):
|
||||
return SignInCheck("tenant_disallowed")
|
||||
return SignInCheck(None, existing_user)
|
||||
|
||||
|
||||
def finish_verified_sign_in(
|
||||
user,
|
||||
http_request: Request,
|
||||
*,
|
||||
user_service: UserService,
|
||||
email_service: EmailService,
|
||||
tenant_id: Optional[str],
|
||||
precomputed_eval: Optional[dict],
|
||||
referral_code: Optional[str],
|
||||
persist_locale: Optional[str],
|
||||
ui_locale: Optional[str],
|
||||
email_locale: Optional[str],
|
||||
device_fingerprint: Optional[str],
|
||||
log_prefix: str,
|
||||
) -> VerifyMagicLinkResponse:
|
||||
"""After an identity proof (email code, Google or Apple ID token): mark the
|
||||
user verified, then the shared ``complete_verified_login`` (welcome credit,
|
||||
referral, locale, session, welcome email) and the ``VerifyMagicLinkResponse``.
|
||||
|
||||
``user`` must already exist. Tenant portal sign-ins never touch consumer
|
||||
credits (as in magic-link verify).
|
||||
"""
|
||||
email = user.email
|
||||
# Same first-login definition as magic-link verify (checked before last_login_at is set).
|
||||
is_first_login = user.total_jobs_created == 0 and not user.last_login_at
|
||||
user = user_service.update_user(
|
||||
email, email_verified=True, last_login_at=datetime.utcnow(),
|
||||
) or user
|
||||
email_locale = email_locale or user.locale or "en"
|
||||
|
||||
result = complete_verified_login(
|
||||
user,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
is_first_login=is_first_login,
|
||||
precomputed_eval=precomputed_eval,
|
||||
grant_welcome_credit=not tenant_id,
|
||||
referral_code=referral_code if not tenant_id else None,
|
||||
locale=persist_locale,
|
||||
ui_locale=ui_locale,
|
||||
ip_address=get_client_ip(http_request),
|
||||
user_agent=http_request.headers.get("user-agent"),
|
||||
tenant_id=tenant_id,
|
||||
device_fingerprint=device_fingerprint,
|
||||
email_locale=email_locale,
|
||||
)
|
||||
logger.info(f"{log_prefix}: verified {_mask_email(email)} (credit_status={result.credit_status})")
|
||||
|
||||
tenant_subdomain = None
|
||||
if tenant_id:
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
tenant_cfg = get_tenant_service().get_tenant_config(tenant_id)
|
||||
if tenant_cfg:
|
||||
tenant_subdomain = tenant_cfg.subdomain
|
||||
|
||||
return VerifyMagicLinkResponse(
|
||||
status="success",
|
||||
session_token=result.session.token,
|
||||
user=_build_user_public(result.user),
|
||||
message=t(email_locale, "users.magicLinkSuccess"),
|
||||
tenant_subdomain=tenant_subdomain,
|
||||
credits_granted=result.credits_granted,
|
||||
credit_status=result.credit_status,
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------- web handoff
|
||||
|
||||
# App → web handoff links carry a real sign-in token in the URL, so they live
|
||||
# only long enough for the app to open the browser, and are single-use (the
|
||||
# magic-link verify transaction marks them used).
|
||||
HANDOFF_TOKEN_TTL_SECONDS = 120
|
||||
|
||||
|
||||
class HandoffTokenRequest(BaseModel):
|
||||
# Allowlist of web destinations; Phase 1 needs only the job's lyrics review.
|
||||
target: Literal["job_review"]
|
||||
job_id: str = Field(min_length=1, max_length=64, pattern=r"^[A-Za-z0-9_-]+$")
|
||||
locale: Optional[str] = None
|
||||
|
||||
|
||||
class HandoffTokenResponse(BaseModel):
|
||||
url: str
|
||||
expires_in_seconds: int
|
||||
|
||||
|
||||
def _bearer_token(authorization: Optional[str]) -> str:
|
||||
scheme, _, token = (authorization or "").partition(" ")
|
||||
token = token.strip()
|
||||
if scheme.lower() != "bearer" or not token:
|
||||
raise HTTPException(status_code=401, detail="invalid_session")
|
||||
return token
|
||||
|
||||
|
||||
@router.post("/auth/handoff-token", response_model=HandoffTokenResponse)
|
||||
def create_handoff_token(
|
||||
body: HandoffTokenRequest,
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
authorization: Optional[str] = Header(default=None),
|
||||
):
|
||||
"""Signed-in web link for the app's own job review (the mobile "Review on web").
|
||||
|
||||
The URL is ``{FRONTEND_URL}/{locale}/auth/verify?token=…&handoff=1``: a
|
||||
``job_review:<id>`` magic-link token valid for ``HANDOFF_TOKEN_TTL_SECONDS``
|
||||
and usable once. ``handoff=1`` makes the verify page sign in without the
|
||||
email-scanner "Complete Sign-In" tap; a used or expired link shows the
|
||||
normal "link expired" state. 401 ``invalid_session``, 404 ``not_found``
|
||||
(missing or not the caller's job), 409 ``not_in_review``.
|
||||
"""
|
||||
from backend.services.job_notification_service import build_review_login_url
|
||||
|
||||
token = _bearer_token(authorization)
|
||||
# Any of the web app's 33 locales (a French app user lands on /fr/…/review).
|
||||
locale = body.locale if body.locale in WEB_LOCALES else "en"
|
||||
# Sync route (threadpool): session check, job read and mint are blocking Firestore calls.
|
||||
job, _status = get_owned_review_job(user_service, token, body.job_id)
|
||||
url = build_review_login_url(
|
||||
job.job_id, job.user_email, locale=locale, handoff_seconds=HANDOFF_TOKEN_TTL_SECONDS,
|
||||
)
|
||||
if not url:
|
||||
raise HTTPException(status_code=502, detail="link_unavailable")
|
||||
return HandoffTokenResponse(url=url, expires_in_seconds=HANDOFF_TOKEN_TTL_SECONDS)
|
||||
|
||||
|
||||
# ------------------------------------------------------- Google / Apple sign-in
|
||||
|
||||
class SocialSignInRequest(BaseModel):
|
||||
# The ID token from the native Google / Apple SDK (or ``fake:<email>`` on the local stack).
|
||||
id_token: str = Field(min_length=1, max_length=8192)
|
||||
locale: Optional[str] = None
|
||||
device_fingerprint: Optional[str] = Field(default=None, max_length=256)
|
||||
referral_code: Optional[str] = Field(default=None, max_length=64)
|
||||
attribution: Optional[
|
||||
Dict[Annotated[str, StringConstraints(max_length=32)], Optional[Annotated[str, StringConstraints(max_length=500)]]]
|
||||
] = Field(default=None, max_length=16)
|
||||
|
||||
|
||||
def _verify_identity(provider: str, id_token: str) -> "social_identity.VerifiedIdentity":
|
||||
"""ID token → verified identity, or 401 ``invalid_token`` / 503 ``google_not_configured``."""
|
||||
settings = get_settings()
|
||||
try:
|
||||
if id_token.startswith(social_identity.FAKE_TOKEN_PREFIX):
|
||||
# Local stack only (FAKE_WORKERS, refused in production); never a network call.
|
||||
if not fake_workers_enabled():
|
||||
raise social_identity.InvalidIdentityToken("fake token outside fake mode")
|
||||
return social_identity.fake_identity(id_token, provider)
|
||||
if provider == "google":
|
||||
audiences = social_identity.parse_audiences(getattr(settings, "google_oauth_client_ids", ""))
|
||||
if not audiences:
|
||||
raise HTTPException(status_code=503, detail="google_not_configured")
|
||||
return social_identity.verify_google_id_token(id_token, audiences)
|
||||
audiences = social_identity.parse_audiences(getattr(settings, "apple_signin_audiences", ""))
|
||||
return social_identity.verify_apple_id_token(id_token, audiences)
|
||||
except social_identity.InvalidIdentityToken as e:
|
||||
logger.warning(f"{provider}-signin: {e}")
|
||||
raise HTTPException(status_code=401, detail="invalid_token")
|
||||
except social_identity.IdentityProviderUnavailable as e:
|
||||
logger.error(f"{provider}-signin: {e}")
|
||||
raise HTTPException(status_code=503, detail="identity_provider_unavailable")
|
||||
|
||||
|
||||
def _social_sign_in(
|
||||
provider: str,
|
||||
body: SocialSignInRequest,
|
||||
http_request: Request,
|
||||
user_service: UserService,
|
||||
email_service: EmailService,
|
||||
) -> VerifyMagicLinkResponse:
|
||||
"""Verify the ID token, apply the same abuse checks as send-code, find or
|
||||
create the user BY EMAIL, then the same session path as verify-code."""
|
||||
from backend.middleware.tenant import get_tenant_config_from_request, get_tenant_from_request
|
||||
|
||||
log_prefix = f"{provider}-signin"
|
||||
identity = _verify_identity(provider, body.id_token)
|
||||
email = identity.email
|
||||
|
||||
email_locale, ui_locale = _split_locale(body.locale)
|
||||
request_locale = get_locale_from_request(http_request)
|
||||
has_accept_language = bool(http_request.headers.get("accept-language"))
|
||||
persist_locale = email_locale or (request_locale if has_accept_language else None)
|
||||
ui_locale = ui_locale or get_full_locale_from_request(http_request)
|
||||
message_locale = email_locale or request_locale
|
||||
|
||||
tenant_id = get_tenant_from_request(http_request)
|
||||
tenant_config = get_tenant_config_from_request(http_request)
|
||||
check = _sign_in_checks(
|
||||
email, http_request, user_service,
|
||||
device_fingerprint=body.device_fingerprint, tenant_config=tenant_config,
|
||||
log_prefix=log_prefix, allow_apple_relay=provider == "apple",
|
||||
)
|
||||
# Unlike send-code there's no email to hide behind, so refusals are explicit.
|
||||
if check.refusal == "disposable":
|
||||
raise HTTPException(status_code=422, detail=t(message_locale, "users.disposableEmailBlocked"))
|
||||
if check.refusal == "signup_limited":
|
||||
raise HTTPException(status_code=429, detail="signup_rate_limited")
|
||||
if check.refusal: # blocked email / IP, tenant email restriction
|
||||
raise HTTPException(status_code=422, detail="sign_in_not_allowed")
|
||||
|
||||
user = check.existing_user or user_service.get_or_create_user_with_attribution(
|
||||
email,
|
||||
tenant_id=tenant_id,
|
||||
signup_ip=get_client_ip(http_request),
|
||||
device_fingerprint=body.device_fingerprint,
|
||||
attribution=body.attribution,
|
||||
)
|
||||
return finish_verified_sign_in(
|
||||
user,
|
||||
http_request,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
tenant_id=tenant_id,
|
||||
precomputed_eval=None,
|
||||
referral_code=http_request.headers.get("x-referral-code") or body.referral_code,
|
||||
persist_locale=persist_locale,
|
||||
ui_locale=ui_locale,
|
||||
email_locale=persist_locale,
|
||||
device_fingerprint=body.device_fingerprint,
|
||||
log_prefix=log_prefix,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/auth/google", response_model=VerifyMagicLinkResponse)
|
||||
def google_sign_in(
|
||||
body: SocialSignInRequest,
|
||||
http_request: Request,
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Exchange a Google ID token for a gen session (same response as verify-code).
|
||||
|
||||
Audience = ``GOOGLE_OAUTH_CLIENT_IDS``. 401 ``invalid_token`` (bad signature,
|
||||
issuer, audience, expiry, or unverified email), 503 ``google_not_configured``,
|
||||
503 ``identity_provider_unavailable`` (couldn't fetch the provider's keys;
|
||||
retryable), 422 disposable / blocked, 429 ``signup_rate_limited`` (new accounts only).
|
||||
"""
|
||||
return _social_sign_in("google", body, http_request, user_service, email_service)
|
||||
|
||||
|
||||
@router.post("/auth/apple", response_model=VerifyMagicLinkResponse)
|
||||
def apple_sign_in(
|
||||
body: SocialSignInRequest,
|
||||
http_request: Request,
|
||||
user_service: UserService = Depends(get_user_service),
|
||||
email_service: EmailService = Depends(get_email_service),
|
||||
):
|
||||
"""Exchange an Apple ID token for a gen session (same response as verify-code).
|
||||
|
||||
Audience = ``APPLE_SIGNIN_AUDIENCES``. "Hide My Email" relay addresses
|
||||
(``@privaterelay.appleid.com``) are accepted as their own account. Errors as
|
||||
for ``/auth/google`` (no 503: the Apple audience has a default).
|
||||
"""
|
||||
return _social_sign_in("apple", body, http_request, user_service, email_service)
|
||||
@@ -1,52 +0,0 @@
|
||||
"""POST /api/parse-karaoke-titles — batch karaoke-filename → artist/title.
|
||||
|
||||
Internal admin endpoint used by kjbox to canonicalise downloaded-file names.
|
||||
Reuses the Vertex Gemini parser; never blocks the caller (kjbox degrades to a
|
||||
deterministic guess when this is unavailable).
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.parse_titles import parse_titles
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(tags=["parse-titles"])
|
||||
|
||||
|
||||
class ParseItem(BaseModel):
|
||||
id: str
|
||||
filename: str
|
||||
channel: Optional[str] = None
|
||||
source: Optional[str] = None
|
||||
|
||||
|
||||
class ParseRequest(BaseModel):
|
||||
items: list[ParseItem]
|
||||
|
||||
|
||||
class ParseResult(BaseModel):
|
||||
id: str
|
||||
artist: str
|
||||
title: str
|
||||
confidence: float
|
||||
|
||||
|
||||
class ParseResponse(BaseModel):
|
||||
results: list[ParseResult]
|
||||
|
||||
|
||||
@router.post("/parse-karaoke-titles", response_model=ParseResponse)
|
||||
async def parse_karaoke_titles(
|
||||
body: ParseRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
items = [i.model_dump() for i in body.items]
|
||||
results = await parse_titles(items)
|
||||
logger.info("parse-karaoke-titles: %d items", len(items))
|
||||
return {"results": results}
|
||||
@@ -1,304 +0,0 @@
|
||||
"""
|
||||
Push Notification API routes.
|
||||
|
||||
Provides endpoints for managing Web Push notification subscriptions:
|
||||
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
|
||||
- POST /api/push/subscribe: Register a push subscription
|
||||
- POST /api/push/unsubscribe: Remove a push subscription
|
||||
- POST/DELETE /api/push/expo-token: Register/remove an Expo push token (mobile app)
|
||||
- GET /api/push/subscriptions: List user's subscriptions
|
||||
- POST /api/push/test: Send a test notification (admin only)
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
from typing import Literal, Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
from backend.services.expo_push_service import get_expo_push_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/push", tags=["push"])
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
|
||||
class VapidPublicKeyResponse(BaseModel):
|
||||
"""Response containing VAPID public key."""
|
||||
enabled: bool
|
||||
vapid_public_key: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeRequest(BaseModel):
|
||||
"""Request to subscribe to push notifications."""
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
tenant_id: Optional[str] = None # Tenant scope for notification isolation
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
"""Response after subscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnsubscribeRequest(BaseModel):
|
||||
"""Request to unsubscribe from push notifications."""
|
||||
endpoint: str
|
||||
|
||||
|
||||
class UnsubscribeResponse(BaseModel):
|
||||
"""Response after unsubscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class SubscriptionInfo(BaseModel):
|
||||
"""Information about a push subscription."""
|
||||
endpoint: str
|
||||
device_name: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_used_at: Optional[str] = None
|
||||
|
||||
|
||||
class SubscriptionsListResponse(BaseModel):
|
||||
"""Response containing user's subscriptions."""
|
||||
subscriptions: List[SubscriptionInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
"""Request to send a test notification."""
|
||||
title: Optional[str] = "Test Notification"
|
||||
body: Optional[str] = "This is a test push notification from Karaoke Generator"
|
||||
|
||||
|
||||
class TestNotificationResponse(BaseModel):
|
||||
"""Response after sending test notification."""
|
||||
status: str
|
||||
sent_count: int
|
||||
message: str
|
||||
|
||||
|
||||
class ExpoTokenRequest(BaseModel):
|
||||
"""Register an Expo push token for one device of the mobile app."""
|
||||
token: str = Field(pattern=r"^Expo(nent)?PushToken\[[^\]\s]+\]$")
|
||||
platform: Literal["ios", "android"]
|
||||
device_id: str = Field(min_length=1, max_length=200)
|
||||
|
||||
|
||||
class ExpoTokenDeleteRequest(BaseModel):
|
||||
token: str
|
||||
|
||||
|
||||
# Routes
|
||||
|
||||
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
|
||||
def get_vapid_public_key():
|
||||
"""
|
||||
Get the VAPID public key for push subscription.
|
||||
|
||||
This endpoint is public - no authentication required.
|
||||
Returns the public key needed for client-side PushManager.subscribe().
|
||||
"""
|
||||
settings = get_settings()
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
if not settings.enable_push_notifications:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
public_key = push_service.get_public_key()
|
||||
if not public_key:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
return VapidPublicKeyResponse(
|
||||
enabled=True,
|
||||
vapid_public_key=public_key
|
||||
)
|
||||
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
subscribe_request: SubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Register a push notification subscription for the current user.
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
|
||||
Subscriptions are scoped by tenant_id to prevent cross-tenant
|
||||
notification leakage. The tenant is resolved from:
|
||||
1. Request body tenant_id field (explicit from frontend)
|
||||
2. Tenant middleware (X-Tenant-ID header or subdomain detection)
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in subscribe_request.keys or "auth" not in subscribe_request.keys:
|
||||
raise HTTPException(status_code=400, detail=t(locale, "push.missingEncryptionKeys"))
|
||||
|
||||
# Resolve tenant_id: prefer explicit from body, fall back to middleware
|
||||
tenant_id = subscribe_request.tenant_id
|
||||
if tenant_id is None:
|
||||
tenant_id = getattr(http_request.state, "tenant_id", None)
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=subscribe_request.endpoint,
|
||||
keys=subscribe_request.keys,
|
||||
device_name=subscribe_request.device_name,
|
||||
tenant_id=tenant_id or None,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail=t(locale, "push.subscriptionSaveFailed"))
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionRegistered")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Remove a push notification subscription.
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
success = await push_service.remove_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint
|
||||
)
|
||||
|
||||
if not success:
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionNotFound")
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message=t(locale, "push.subscriptionRemoved")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/expo-token", status_code=204)
|
||||
async def register_expo_token(
|
||||
body: ExpoTokenRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Register this device's Expo token. Moves it off any other account that held it."""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
await asyncio.to_thread(
|
||||
get_expo_push_service().register_token,
|
||||
auth_result.user_email, body.token, body.platform, body.device_id,
|
||||
)
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.delete("/expo-token", status_code=204)
|
||||
async def delete_expo_token(
|
||||
body: ExpoTokenDeleteRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Remove one of the caller's Expo tokens (the app calls this before sign-out)."""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
await asyncio.to_thread(get_expo_push_service().remove_tokens, auth_result.user_email, [body.token])
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
request: Request,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
List all push notification subscriptions for the current user.
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
locale = get_locale_from_request(request)
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
|
||||
|
||||
return SubscriptionsListResponse(
|
||||
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
|
||||
count=len(subscriptions)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
http_request: Request,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Send a test push notification to the current user's devices.
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
locale = get_locale_from_request(http_request)
|
||||
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail=t(locale, "push.disabled"))
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail=t(locale, "push.userEmailNotAvailable"))
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
sent_count = await push_service.send_push(
|
||||
user_email=auth_result.user_email,
|
||||
title=request.title or "Test Notification",
|
||||
body=request.body or "This is a test push notification",
|
||||
url="/app/",
|
||||
tag="test"
|
||||
)
|
||||
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=t(locale, "push.testNotificationSent", sent_count=sent_count)
|
||||
)
|
||||
@@ -1,346 +0,0 @@
|
||||
"""
|
||||
Admin API routes for blocklist and YouTube queue management.
|
||||
|
||||
Handles:
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- YouTube upload queue management
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class YouTubeQueueEntry(BaseModel):
|
||||
"""A queued YouTube upload entry."""
|
||||
job_id: str
|
||||
status: str
|
||||
reason: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
brand_code: Optional[str] = None
|
||||
queued_at: Optional[datetime] = None
|
||||
attempts: int = 0
|
||||
max_attempts: int = 5
|
||||
last_error: Optional[str] = None
|
||||
youtube_url: Optional[str] = None
|
||||
notification_sent: bool = False
|
||||
|
||||
|
||||
class YouTubeQueueListResponse(BaseModel):
|
||||
"""List of YouTube upload queue entries."""
|
||||
entries: List[YouTubeQueueEntry]
|
||||
stats: dict
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data with domain source separation."""
|
||||
external_domains: List[str]
|
||||
manual_domains: List[str]
|
||||
allowlisted_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
last_sync_at: Optional[datetime] = None
|
||||
last_sync_count: Optional[int] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
updated_by: Optional[str] = None
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
"""Request to add/remove a domain."""
|
||||
domain: str
|
||||
|
||||
|
||||
class EmailRequest(BaseModel):
|
||||
"""Request to add/remove an email."""
|
||||
email: str
|
||||
|
||||
|
||||
class IPRequest(BaseModel):
|
||||
"""Request to add/remove an IP."""
|
||||
ip_address: str
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/blocklists", response_model=BlocklistsResponse)
|
||||
def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all blocklist data with domain source separation."""
|
||||
email_validation = get_email_validation_service()
|
||||
raw_data = email_validation.get_blocklist_raw_data()
|
||||
return BlocklistsResponse(**raw_data)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
def add_disposable_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.domainAddedToBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
|
||||
def remove_disposable_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.domainRemovedFromBl", domain=domain)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/allowlisted-domains", response_model=SuccessResponse)
|
||||
def add_allowlisted_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the allowlist (overrides external blocklist)."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidDomainFormat"))
|
||||
email_validation.add_allowlisted_domain(domain, auth_result.user_email)
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainAddedToAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.delete("/blocklists/allowlisted-domains/{domain}", response_model=SuccessResponse)
|
||||
def remove_allowlisted_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the allowlist."""
|
||||
email_validation = get_email_validation_service()
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_allowlisted_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.domainNotFoundInAllowlist"))
|
||||
return SuccessResponse(success=True, message=t("en", "admin.domainRemovedFromAllowlist", domain=domain))
|
||||
|
||||
|
||||
@router.post("/blocklists/sync", response_model=SuccessResponse)
|
||||
async def trigger_sync(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Manually trigger a sync of the external disposable domain blocklist."""
|
||||
import asyncio
|
||||
from backend.services.disposable_domain_sync_service import (
|
||||
fetch_external_blocklist, sync_disposable_domains
|
||||
)
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
|
||||
domains = await fetch_external_blocklist()
|
||||
db = get_firestore_client()
|
||||
result = await asyncio.to_thread(sync_disposable_domains, db, domains)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.syncComplete", external_count=result['external_count'], added=result['added'], removed=result['removed'])
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
|
||||
def add_blocked_email(
|
||||
request: EmailRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an email to the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidEmailFormat"))
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.emailAddedToBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
|
||||
def remove_blocked_email(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.emailNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.emailRemovedFromBl", email=email)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
|
||||
def add_blocked_ip(
|
||||
request: IPRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail=t("en", "admin.invalidIPAddress"))
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.ipAddedToBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
|
||||
def remove_blocked_ip(
|
||||
ip_address: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail=t("en", "admin.ipNotFoundInBl"))
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.ipRemovedFromBl", ip_address=ip_address)
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# YouTube Upload Queue Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/youtube-queue", response_model=YouTubeQueueListResponse)
|
||||
def get_youtube_queue(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get YouTube upload queue entries and stats.
|
||||
|
||||
Returns all queue entries (any status) and aggregate statistics.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
entries = queue_service.get_all_queue_entries(limit=50)
|
||||
stats = queue_service.get_queue_stats()
|
||||
|
||||
return YouTubeQueueListResponse(
|
||||
entries=[YouTubeQueueEntry(**entry) for entry in entries],
|
||||
stats=stats,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/youtube-queue/{job_id}/retry", response_model=SuccessResponse)
|
||||
def retry_youtube_upload(
|
||||
job_id: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Manually retry a failed YouTube upload.
|
||||
|
||||
Resets the entry back to 'queued' status with 0 attempts.
|
||||
"""
|
||||
from backend.services.youtube_upload_queue_service import get_youtube_upload_queue_service
|
||||
|
||||
queue_service = get_youtube_upload_queue_service()
|
||||
|
||||
if not queue_service.retry_upload(job_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=t("en", "admin.youtubeUploadNotFound", job_id=job_id)
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.youtubeUploadQueuedForRetry", job_id=job_id)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/youtube-queue/process", response_model=SuccessResponse)
|
||||
async def trigger_youtube_queue_processing(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Manually trigger YouTube upload queue processing.
|
||||
|
||||
Same as the scheduled processor but triggered from the admin dashboard.
|
||||
Runs in the background and returns immediately.
|
||||
"""
|
||||
from backend.workers.youtube_queue_processor import process_youtube_upload_queue
|
||||
import asyncio
|
||||
|
||||
async def _process():
|
||||
try:
|
||||
result = await process_youtube_upload_queue()
|
||||
logger.info(f"Manual YouTube queue processing complete: {result}")
|
||||
except Exception as e:
|
||||
logger.exception(f"Manual YouTube queue processing failed: {e}")
|
||||
|
||||
# Fire and forget
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
loop.create_task(_process())
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=t("en", "admin.youtubeQueueProcessingStarted")
|
||||
)
|
||||
@@ -1,461 +0,0 @@
|
||||
"""
|
||||
Referral API routes.
|
||||
|
||||
Handles:
|
||||
- Public referral interstitial (click tracking)
|
||||
- Authenticated referral dashboard
|
||||
- Stripe Connect onboarding
|
||||
- Admin vanity link creation and link management
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||
from fastapi.responses import Response
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.models.referral import (
|
||||
CreateReferralLinkRequest,
|
||||
UpdateReferralLinkRequest,
|
||||
ReferralInterstitialResponse,
|
||||
VanityRequestResponse,
|
||||
)
|
||||
from backend.services.flyer_service import get_flyer_service, FlyerError
|
||||
from backend.services.referral_service import get_referral_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/referrals", tags=["referrals"])
|
||||
|
||||
|
||||
class GenerateFlyerRequest(BaseModel):
|
||||
theme: str = Field(..., pattern="^(light|dark)$")
|
||||
qr_data_url: str = Field(..., max_length=500_000)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Public endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/r/{code}", response_model=ReferralInterstitialResponse)
|
||||
def get_referral_interstitial(code: str, no_track: bool = False):
|
||||
"""
|
||||
Public endpoint: get interstitial data for a referral link.
|
||||
|
||||
Increments click count unless no_track=true. Returns valid=False if code not found.
|
||||
"""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
return ReferralInterstitialResponse(valid=False)
|
||||
|
||||
# Track the click (skip when fetching for display purposes)
|
||||
if not no_track:
|
||||
try:
|
||||
service.increment_clicks(code)
|
||||
except Exception:
|
||||
logger.warning("Failed to increment clicks for code %s", code, exc_info=True)
|
||||
|
||||
return ReferralInterstitialResponse(
|
||||
valid=True,
|
||||
referral_code=link.code,
|
||||
referrer_display_name=link.display_name,
|
||||
discount_percent=link.discount_percent,
|
||||
discount_duration_days=link.discount_duration_days,
|
||||
custom_message=link.custom_message,
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Authenticated user endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.get("/me")
|
||||
def get_my_dashboard(auth=Depends(require_auth)):
|
||||
"""Get referral dashboard data for the authenticated user."""
|
||||
service = get_referral_service()
|
||||
return service.get_dashboard_data(auth.user_email)
|
||||
|
||||
|
||||
@router.put("/me")
|
||||
def update_my_link(
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Update display_name and custom_message on the user's own referral link."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
update_fields = {}
|
||||
if updates.display_name is not None:
|
||||
update_fields["display_name"] = updates.display_name
|
||||
if updates.custom_message is not None:
|
||||
update_fields["custom_message"] = updates.custom_message
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(link.code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/me/connect")
|
||||
def start_stripe_connect(auth=Depends(require_auth)):
|
||||
"""Start Stripe Connect onboarding for the authenticated user."""
|
||||
# Check if user already has a Connect account
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(auth.user_email)
|
||||
if user and user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="Stripe Connect account already configured")
|
||||
|
||||
service = get_referral_service()
|
||||
account_id, onboarding_url = service.create_connect_account(auth.user_email)
|
||||
|
||||
if not account_id:
|
||||
raise HTTPException(status_code=500, detail="Failed to create Stripe Connect account")
|
||||
|
||||
user_service.update_user(auth.user_email, stripe_connect_account_id=account_id)
|
||||
|
||||
return {"account_id": account_id, "onboarding_url": onboarding_url}
|
||||
|
||||
|
||||
@router.post("/me/connect/dashboard-link")
|
||||
def get_connect_dashboard_link(auth=Depends(require_auth)):
|
||||
"""Get a login link to the Stripe Express dashboard for managing payout settings."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_login_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create dashboard link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
@router.post("/me/connect/update-link")
|
||||
def get_connect_update_link(auth=Depends(require_auth)):
|
||||
"""Get a link to update Stripe Connect account info (re-onboarding)."""
|
||||
from backend.services.user_service import get_user_service
|
||||
user = get_user_service().get_user(auth.user_email)
|
||||
if not user or not user.stripe_connect_account_id:
|
||||
raise HTTPException(status_code=400, detail="No Stripe Connect account configured")
|
||||
|
||||
service = get_referral_service()
|
||||
url = service.create_connect_update_link(user.stripe_connect_account_id)
|
||||
if not url:
|
||||
raise HTTPException(status_code=500, detail="Failed to create update link")
|
||||
|
||||
return {"url": url}
|
||||
|
||||
|
||||
class VanityRequest(BaseModel):
|
||||
desired_code: str = Field(..., min_length=2, max_length=30, pattern="^[a-z0-9-]+$")
|
||||
|
||||
|
||||
@router.post("/me/vanity-request")
|
||||
def request_vanity_url(
|
||||
request: VanityRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Request a custom vanity referral URL. Sends email to admin for review."""
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
service = get_referral_service()
|
||||
|
||||
# Validate up-front against the same rules approval enforces (reserved words,
|
||||
# vanity pattern) so a request that would later be auto-rejected never lands
|
||||
# in the admin queue. Do this before get_or_create_link so a rejected request
|
||||
# from a brand-new user doesn't leave an orphaned auto-generated link behind.
|
||||
valid, msg = service._validate_vanity_code(request.desired_code)
|
||||
if not valid:
|
||||
raise HTTPException(status_code=400, detail=msg)
|
||||
|
||||
# Check if code is already taken
|
||||
existing = service.get_link_by_code(request.desired_code)
|
||||
if existing:
|
||||
raise HTTPException(status_code=409, detail="That code is already taken. Please try another.")
|
||||
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
# Persist the request so it shows up as a pending item in the admin dashboard.
|
||||
service.create_vanity_request(
|
||||
owner_email=auth.user_email,
|
||||
current_code=link.code,
|
||||
desired_code=request.desired_code,
|
||||
)
|
||||
|
||||
email_service = get_email_service()
|
||||
email_service.provider.send_email(
|
||||
to_email="andrew@nomadkaraoke.com",
|
||||
subject=f"Vanity referral URL request: {request.desired_code}",
|
||||
html_content=f"""
|
||||
<h2>Vanity Referral URL Request</h2>
|
||||
<p><strong>User:</strong> {auth.user_email}</p>
|
||||
<p><strong>Current code:</strong> {link.code}</p>
|
||||
<p><strong>Desired vanity code:</strong> {request.desired_code}</p>
|
||||
<p>Approve or deny it under "Pending Vanity Requests" in the <a href="https://gen.nomadkaraoke.com/admin/referrals">admin dashboard</a>.</p>
|
||||
""",
|
||||
text_content=f"Vanity URL request from {auth.user_email}: {request.desired_code} (current: {link.code})",
|
||||
)
|
||||
|
||||
logger.info("Vanity URL request from %s: %s", auth.user_email, request.desired_code)
|
||||
return {"ok": True, "message": "Request sent"}
|
||||
|
||||
|
||||
@router.post("/me/flyer")
|
||||
def generate_flyer(
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_auth),
|
||||
):
|
||||
"""Generate a personalized referral flyer PDF."""
|
||||
service = get_referral_service()
|
||||
link = service.get_or_create_link(auth.user_email)
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": 'attachment; filename="nomad-karaoke-referral-flyer.pdf"',
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Admin endpoints
|
||||
# ============================================================================
|
||||
|
||||
|
||||
@router.post("/admin/vanity")
|
||||
def create_vanity_link(
|
||||
request: CreateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: create a vanity referral link."""
|
||||
if not request.vanity_code:
|
||||
raise HTTPException(status_code=400, detail="vanity_code is required")
|
||||
|
||||
service = get_referral_service()
|
||||
success, link, message = service.create_vanity_link(
|
||||
code=request.vanity_code,
|
||||
owner_email=request.owner_email,
|
||||
display_name=request.display_name,
|
||||
custom_message=request.custom_message,
|
||||
discount_percent=request.discount_percent,
|
||||
kickback_percent=request.kickback_percent,
|
||||
discount_duration_days=request.discount_duration_days,
|
||||
earning_duration_days=request.earning_duration_days,
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "code": link.code, "message": message}
|
||||
|
||||
|
||||
@router.get("/admin/vanity-requests")
|
||||
def list_vanity_requests(
|
||||
status: str = Query("pending", pattern="^(pending|approved|denied)$"),
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: list vanity code requests (pending by default)."""
|
||||
service = get_referral_service()
|
||||
requests = service.list_vanity_requests(status=status)
|
||||
return {
|
||||
"requests": [
|
||||
VanityRequestResponse(**r.model_dump()).model_dump(mode="json")
|
||||
for r in requests
|
||||
],
|
||||
"count": len(requests),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/admin/vanity-requests/{request_id}/approve")
|
||||
def approve_vanity_request(
|
||||
request_id: str,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: approve a vanity request by renaming the owner's link, then notify them."""
|
||||
from backend.services.email_service import get_email_service
|
||||
|
||||
service = get_referral_service()
|
||||
req = service.get_vanity_request(request_id)
|
||||
if not req:
|
||||
raise HTTPException(status_code=404, detail="Request not found")
|
||||
|
||||
success, link, message = service.approve_vanity_request(
|
||||
request_id, resolved_by=auth.user_email
|
||||
)
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
# Notify the requester that their vanity URL is live.
|
||||
try:
|
||||
referral_url = f"https://nomadkaraoke.com/r/{link.code}"
|
||||
get_email_service().provider.send_email(
|
||||
to_email=req.owner_email,
|
||||
subject="Your custom referral link is live 🎉",
|
||||
html_content=f"""
|
||||
<h2>Your custom referral link is ready</h2>
|
||||
<p>Good news — your requested referral code <strong>{link.code}</strong> is now active.</p>
|
||||
<p>Share it anywhere: <a href="{referral_url}">{referral_url}</a></p>
|
||||
<p>All your existing clicks, signups, and earnings carried over automatically.</p>
|
||||
""",
|
||||
text_content=f"Your custom referral link is live: {referral_url}",
|
||||
)
|
||||
except Exception:
|
||||
logger.warning("Failed to send vanity-approved email to %s", req.owner_email, exc_info=True)
|
||||
|
||||
return {"ok": True, "code": link.code, "message": message}
|
||||
|
||||
|
||||
@router.post("/admin/vanity-requests/{request_id}/deny")
|
||||
def deny_vanity_request(
|
||||
request_id: str,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: deny a vanity request."""
|
||||
service = get_referral_service()
|
||||
success, message = service.deny_vanity_request(request_id, resolved_by=auth.user_email)
|
||||
if not success:
|
||||
raise HTTPException(status_code=400, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.get("/admin/links")
|
||||
def list_links(
|
||||
limit: int = Query(50, ge=1, le=200),
|
||||
offset: int = Query(0, ge=0),
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: list all referral links."""
|
||||
service = get_referral_service()
|
||||
links = service.list_links(limit=limit, offset=offset)
|
||||
return {
|
||||
"links": [link.model_dump(mode="json") for link in links],
|
||||
"count": len(links),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/admin/apply-discount")
|
||||
async def admin_apply_discount(
|
||||
request: Request,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: apply a referral discount to any user, even without a referral link."""
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
body = await request.json()
|
||||
user_email = body.get("email", "").lower()
|
||||
discount_percent = body.get("discount_percent", 10)
|
||||
duration_days = body.get("duration_days", 30)
|
||||
referral_code = body.get("referral_code", "admin-grant")
|
||||
|
||||
if not user_email:
|
||||
raise HTTPException(status_code=400, detail="Email is required")
|
||||
|
||||
user_service = get_user_service()
|
||||
user = user_service.get_user(user_email)
|
||||
if not user:
|
||||
raise HTTPException(status_code=404, detail="User not found")
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
now = datetime.utcnow()
|
||||
|
||||
user_service.update_user(
|
||||
user_email,
|
||||
referred_by_code=referral_code,
|
||||
referred_at=now,
|
||||
referral_discount_expires_at=now + timedelta(days=duration_days),
|
||||
)
|
||||
|
||||
# Create/ensure the Stripe coupon exists for this discount percent
|
||||
service = get_referral_service()
|
||||
service.get_or_create_stripe_coupon(discount_percent)
|
||||
|
||||
logger.info(f"Admin applied {discount_percent}% discount for {duration_days} days to {user_email}")
|
||||
|
||||
return {
|
||||
"ok": True,
|
||||
"message": f"Applied {discount_percent}% discount for {duration_days} days",
|
||||
"discount_expires_at": (now + timedelta(days=duration_days)).isoformat(),
|
||||
}
|
||||
|
||||
|
||||
@router.put("/admin/links/{code}")
|
||||
def update_link(
|
||||
code: str,
|
||||
updates: UpdateReferralLinkRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: update any referral link's settings."""
|
||||
service = get_referral_service()
|
||||
|
||||
update_fields = {
|
||||
k: v for k, v in updates.model_dump().items() if v is not None
|
||||
}
|
||||
|
||||
if not update_fields:
|
||||
return {"ok": True, "message": "No changes"}
|
||||
|
||||
success, message = service.update_link(code, **update_fields)
|
||||
if not success:
|
||||
raise HTTPException(status_code=404, detail=message)
|
||||
|
||||
return {"ok": True, "message": message}
|
||||
|
||||
|
||||
@router.post("/admin/links/{code}/flyer")
|
||||
def admin_generate_flyer(
|
||||
code: str,
|
||||
request: GenerateFlyerRequest,
|
||||
auth=Depends(require_admin),
|
||||
):
|
||||
"""Admin: generate a personalized referral flyer PDF for any code."""
|
||||
service = get_referral_service()
|
||||
link = service.get_link_by_code(code)
|
||||
|
||||
if not link:
|
||||
raise HTTPException(status_code=404, detail=f"Referral link '{code}' not found")
|
||||
|
||||
try:
|
||||
flyer_service = get_flyer_service()
|
||||
pdf_bytes = flyer_service.generate_pdf(
|
||||
theme=request.theme,
|
||||
referral_code=link.code,
|
||||
discount_percent=link.discount_percent,
|
||||
qr_data_url=request.qr_data_url,
|
||||
)
|
||||
except FlyerError as e:
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type="application/pdf",
|
||||
headers={
|
||||
"Content-Disposition": f'attachment; filename="nomad-karaoke-flyer-{link.code}.pdf"',
|
||||
},
|
||||
)
|
||||
@@ -1,208 +0,0 @@
|
||||
"""
|
||||
Public song-request voting board API (requests.nomadkaraoke.com).
|
||||
|
||||
Anyone signed in with an email magic link can submit a song request (auto-corrected
|
||||
artist/title) and cast one vote per calendar day. The board itself (GET /requests) is
|
||||
readable without auth so it can be linked from YouTube descriptions and browsed freely.
|
||||
|
||||
Phase 2 (daily auto-picker, free-credit grant, ownership handoff, voter emails) is not
|
||||
here — see docs/archive/2026-09-02-requests-voting-board-plan.md.
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
|
||||
from backend.api.dependencies import require_auth, security, get_token_from_request
|
||||
from backend.services.auth_service import AuthResult, get_auth_service
|
||||
from backend.services.match_judge.classifier import normalize_for_match
|
||||
from backend.models.song_request import (
|
||||
BoardResponse,
|
||||
DailyVoteStatus,
|
||||
SongRequest,
|
||||
SongRequestPublic,
|
||||
SubmitRequestBody,
|
||||
SubmitResponse,
|
||||
VoteBody,
|
||||
)
|
||||
from backend.services.song_request_service import (
|
||||
InvalidSubmitter,
|
||||
RequestNotFound,
|
||||
SubmissionRateLimited,
|
||||
VoteAlreadyUsed,
|
||||
WouldDeleteRequest,
|
||||
get_song_request_service,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/requests-board", tags=["requests-board"])
|
||||
|
||||
|
||||
async def optional_user_email(
|
||||
request: Request,
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
) -> Optional[str]:
|
||||
"""Resolve the caller's email if a valid token is present, else None.
|
||||
|
||||
Unlike require_auth this never raises — the board is publicly viewable, we just
|
||||
annotate the viewer's own vote when they happen to be signed in.
|
||||
"""
|
||||
token = await get_token_from_request(request, credentials, None)
|
||||
if not token:
|
||||
return None
|
||||
try:
|
||||
result = get_auth_service().validate_token_full(token)
|
||||
return result.user_email if result.is_valid else None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _was_corrected(req: SongRequest) -> bool:
|
||||
return (
|
||||
normalize_for_match(req.artist_raw) != normalize_for_match(req.artist)
|
||||
or normalize_for_match(req.title_raw) != normalize_for_match(req.title)
|
||||
)
|
||||
|
||||
|
||||
def _to_public(req: SongRequest, your_vote: Optional[int] = None) -> SongRequestPublic:
|
||||
return SongRequestPublic(
|
||||
id=req.id,
|
||||
artist=req.artist,
|
||||
title=req.title,
|
||||
status=req.status,
|
||||
source=req.source,
|
||||
vote_count=req.vote_count,
|
||||
created_at=req.created_at.isoformat(),
|
||||
youtube_url=req.youtube_url,
|
||||
was_corrected=_was_corrected(req),
|
||||
your_vote=your_vote,
|
||||
)
|
||||
|
||||
|
||||
@router.get("/requests", response_model=BoardResponse)
|
||||
def list_requests(
|
||||
viewer_email: Optional[str] = Depends(optional_user_email),
|
||||
):
|
||||
"""The ranked board: active requests + recently published, with the caller's vote."""
|
||||
service = get_song_request_service()
|
||||
active = service.list_active()
|
||||
published = service.list_published()
|
||||
|
||||
daily_vote = service.get_daily_vote(viewer_email) if viewer_email else None
|
||||
voted_request_id = daily_vote.request_id if daily_vote else None
|
||||
voted_value = daily_vote.value if daily_vote else None
|
||||
|
||||
return BoardResponse(
|
||||
requests=[
|
||||
_to_public(r, your_vote=voted_value if r.id == voted_request_id else None)
|
||||
for r in active
|
||||
],
|
||||
published=[_to_public(r) for r in published],
|
||||
voted_today=bool(daily_vote) if viewer_email is not None else None,
|
||||
your_vote_request_id=voted_request_id,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/requests", response_model=SubmitResponse)
|
||||
async def submit_request(
|
||||
body: SubmitRequestBody,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Submit a song request (requires sign-in). Auto-corrects artist/title and dedupes."""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=403, detail="A signed-in email is required to submit.")
|
||||
|
||||
service = get_song_request_service()
|
||||
try:
|
||||
request, already_existed, canonical_artist, canonical_title = await service.submit_request(
|
||||
auth_result.user_email, body.artist, body.title
|
||||
)
|
||||
except SubmissionRateLimited:
|
||||
raise HTTPException(
|
||||
status_code=429,
|
||||
detail={
|
||||
"code": "submission_limit",
|
||||
"message": "You can request one song per day — please come back tomorrow.",
|
||||
},
|
||||
)
|
||||
except VoteAlreadyUsed:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"code": "vote_already_used",
|
||||
"message": "Requesting a song uses your one vote for today, and you've already "
|
||||
"voted on another request. Remove that vote first, or come back tomorrow.",
|
||||
},
|
||||
)
|
||||
except InvalidSubmitter:
|
||||
raise HTTPException(status_code=403, detail="A signed-in email is required to submit.")
|
||||
|
||||
daily = service.get_daily_vote(auth_result.user_email)
|
||||
your_vote = daily.value if daily and daily.request_id == request.id else None
|
||||
return SubmitResponse(
|
||||
status="already_exists" if already_existed else "created",
|
||||
request=_to_public(request, your_vote=your_vote),
|
||||
canonical_artist=canonical_artist,
|
||||
canonical_title=canonical_title,
|
||||
was_corrected=_was_corrected(request),
|
||||
)
|
||||
|
||||
|
||||
@router.post("/requests/{request_id}/vote", response_model=SongRequestPublic)
|
||||
def vote(
|
||||
request_id: str,
|
||||
body: VoteBody,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Cast/move/undo your single daily vote on a request."""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=403, detail="A signed-in email is required to vote.")
|
||||
|
||||
service = get_song_request_service()
|
||||
try:
|
||||
service.cast_vote(
|
||||
auth_result.user_email, request_id, body.direction, confirm_delete=body.confirm_delete
|
||||
)
|
||||
except RequestNotFound:
|
||||
raise HTTPException(status_code=404, detail="That request no longer exists.")
|
||||
except WouldDeleteRequest as exc:
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"code": "would_delete_request",
|
||||
# The request that would be deleted (not necessarily the one being voted on:
|
||||
# moving your only up-vote away deletes the OLD request).
|
||||
"request_id": exc.request_id,
|
||||
"message": "Removing your vote will delete this request from the board "
|
||||
"(nobody else is backing it). Continue?",
|
||||
},
|
||||
)
|
||||
except InvalidSubmitter:
|
||||
raise HTTPException(status_code=403, detail="A signed-in email is required to vote.")
|
||||
|
||||
updated = service.get_request(request_id)
|
||||
if not updated:
|
||||
# Its last up-vote was withdrawn (confirmed), so the request was deleted.
|
||||
return SongRequestPublic(
|
||||
id=request_id, artist="", title="", status="open", source="human",
|
||||
vote_count=0, created_at="", request_deleted=True,
|
||||
)
|
||||
daily = service.get_daily_vote(auth_result.user_email)
|
||||
your_vote = daily.value if daily and daily.request_id == request_id else None
|
||||
return _to_public(updated, your_vote=your_vote)
|
||||
|
||||
|
||||
@router.get("/me", response_model=DailyVoteStatus)
|
||||
def my_daily_status(
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""The caller's daily-vote status (has today's vote been used, and on what)."""
|
||||
service = get_song_request_service()
|
||||
daily = service.get_daily_vote(auth_result.user_email) if auth_result.user_email else None
|
||||
return DailyVoteStatus(
|
||||
voted_today=bool(daily),
|
||||
request_id=daily.request_id if daily else None,
|
||||
value=daily.value if daily else None,
|
||||
)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,121 +0,0 @@
|
||||
"""
|
||||
Tenant API routes for white-label portal configuration.
|
||||
|
||||
These endpoints are public (no auth required) since the frontend needs
|
||||
to fetch tenant branding before the user logs in.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.i18n import t, get_locale_from_request
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
|
||||
|
||||
|
||||
@router.get("/config", response_model=TenantConfigResponse)
|
||||
def get_tenant_config(
|
||||
request: Request,
|
||||
tenant: Optional[str] = Query(
|
||||
None,
|
||||
description="Tenant ID override (for development). If not provided, detected from subdomain.",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Get tenant configuration for the current portal.
|
||||
|
||||
This endpoint detects the tenant from:
|
||||
1. Query parameter `tenant` (for development/testing)
|
||||
2. X-Tenant-ID header (set by frontend)
|
||||
3. Host header subdomain
|
||||
|
||||
Returns the public tenant configuration (branding, features, defaults)
|
||||
or indicates this is the default Nomad Karaoke portal.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
|
||||
# Priority 1: Query parameter (for dev/testing)
|
||||
tenant_id = tenant
|
||||
|
||||
# Priority 2: X-Tenant-ID header
|
||||
if not tenant_id:
|
||||
tenant_id = request.headers.get("X-Tenant-ID")
|
||||
|
||||
# Priority 3: Detect from Host header
|
||||
if not tenant_id:
|
||||
host = request.headers.get("Host", "")
|
||||
# Check if this is a tenant subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
if host and "nomadkaraoke.com" in host.lower():
|
||||
parts = host.lower().split(".")
|
||||
# Skip known non-tenant subdomains
|
||||
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
|
||||
potential_tenant = parts[0]
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
tenant_id = potential_tenant
|
||||
|
||||
# If no tenant detected, return default config
|
||||
if not tenant_id:
|
||||
logger.debug("No tenant detected, returning default config")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Load tenant config
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
logger.warning(f"Tenant not found: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
logger.info(f"Returning config for tenant: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
|
||||
def get_tenant_config_by_id(tenant_id: str):
|
||||
"""
|
||||
Get tenant configuration by explicit tenant ID.
|
||||
|
||||
This is useful for admin tools or debugging.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Check if tenant is active
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/asset/{tenant_id}/{asset_name}")
|
||||
def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
This redirects to the signed GCS URL for the asset.
|
||||
"""
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
tenant_service = get_tenant_service()
|
||||
url = tenant_service.get_asset_url(tenant_id, asset_name)
|
||||
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail=t("en", "tenant.assetNotFound"))
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
@@ -1,375 +0,0 @@
|
||||
"""
|
||||
Admin API routes for provisioning white-label tenants.
|
||||
|
||||
These endpoints let an admin mint a tenant (branded portal + locked theme) from
|
||||
the UI, replacing the hand-run ``scripts/setup-*-tenant.py`` recipe. Once created,
|
||||
the tenant can be driven immediately via the admin preview path
|
||||
``?preview_tenant=<id>`` — including the tenant bulk-upload flow (folder of tracks,
|
||||
auto-paired mixed/instrumental, resumable uploads, locked theme on every job).
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, UploadFile
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from pydantic import BaseModel, ValidationError
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.models.tenant import TenantPublicConfig
|
||||
from backend.models.theme import ColorOverrides
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.tenant_admin_service import (
|
||||
IMAGE_CONTENT_TYPES,
|
||||
TenantConflictError,
|
||||
TenantNotFoundError,
|
||||
TenantProvisioningError,
|
||||
TenantValidationError,
|
||||
create_tenant,
|
||||
delete_tenant,
|
||||
get_default_style_params,
|
||||
get_tenant_detail,
|
||||
get_tenant_domain_status,
|
||||
list_tenants,
|
||||
provision_tenant_domain,
|
||||
update_tenant,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/admin/tenants", tags=["admin", "tenant"])
|
||||
|
||||
# Public app origin used to build the admin-preview link.
|
||||
APP_ORIGIN = "https://gen.nomadkaraoke.com"
|
||||
|
||||
# Reject oversized image uploads (backgrounds/logo).
|
||||
MAX_IMAGE_BYTES = 15 * 1024 * 1024
|
||||
|
||||
|
||||
class TenantSummary(BaseModel):
|
||||
id: Optional[str] = None
|
||||
name: Optional[str] = None
|
||||
subdomain: Optional[str] = None
|
||||
is_active: bool = True
|
||||
locked_theme: Optional[str] = None
|
||||
theme_id: Optional[str] = None
|
||||
dropbox_path: Optional[str] = None
|
||||
brand_prefix: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
|
||||
|
||||
class TenantListResponse(BaseModel):
|
||||
tenants: List[TenantSummary]
|
||||
|
||||
|
||||
class TenantCreateResponse(BaseModel):
|
||||
tenant: TenantPublicConfig
|
||||
preview_url: str
|
||||
subdomain_url: str
|
||||
# Cloudflare state of the portal subdomain: {hostname, state, dns_ok, pages_status}.
|
||||
# state is "active" once the certificate is issued (usually within ~1 min).
|
||||
domain: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class TenantDetailResponse(BaseModel):
|
||||
tenant: Dict[str, Any]
|
||||
theme_id: str
|
||||
style_params: Dict[str, Any]
|
||||
assets: List[str]
|
||||
preview_url: str
|
||||
domain: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class TenantDomainResponse(BaseModel):
|
||||
domain: Dict[str, Any]
|
||||
|
||||
|
||||
class TenantTemplateResponse(BaseModel):
|
||||
style_params: Dict[str, Any]
|
||||
|
||||
|
||||
class TenantUpdateResponse(BaseModel):
|
||||
tenant: TenantPublicConfig
|
||||
|
||||
|
||||
def _clean(value: Optional[str]) -> Optional[str]:
|
||||
"""Normalize empty/whitespace form values to None."""
|
||||
if value is None:
|
||||
return None
|
||||
value = value.strip()
|
||||
return value or None
|
||||
|
||||
|
||||
def _preview_url(tenant_id: str) -> str:
|
||||
return f"{APP_ORIGIN}/en/app?preview_tenant={tenant_id}"
|
||||
|
||||
|
||||
def _parse_json_field(raw: Optional[str], label: str) -> Optional[Any]:
|
||||
"""Parse an optional JSON string form field, mapping errors to 400."""
|
||||
if raw is None or not raw.strip():
|
||||
return None
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except (ValueError, TypeError) as exc:
|
||||
raise HTTPException(status_code=400, detail=f"{label}: invalid JSON ({exc}).")
|
||||
|
||||
|
||||
def _parse_domains(raw: Optional[str]) -> List[str]:
|
||||
if not raw:
|
||||
return []
|
||||
parts = raw.replace("\n", ",").replace(" ", ",").split(",")
|
||||
return [p.strip().lower() for p in parts if p.strip()]
|
||||
|
||||
|
||||
async def _read_image(upload: Optional[UploadFile], label: str) -> Optional[Tuple[bytes, str]]:
|
||||
"""Read+validate an uploaded image, returning (bytes, ext) or None."""
|
||||
if upload is None or not upload.filename:
|
||||
return None
|
||||
ext = upload.filename.rsplit(".", 1)[-1].lower() if "." in upload.filename else ""
|
||||
if ext not in IMAGE_CONTENT_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"{label}: unsupported image type '.{ext}'. Use PNG, JPG, GIF, or WEBP.",
|
||||
)
|
||||
data = await upload.read()
|
||||
if not data:
|
||||
return None
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"{label}: image is too large ({len(data) // (1024 * 1024)}MB). Max 15MB.",
|
||||
)
|
||||
return data, ext
|
||||
|
||||
|
||||
# Assets replaceable in the manage flow: images plus fonts.
|
||||
ASSET_EXTENSIONS = set(IMAGE_CONTENT_TYPES) | {"ttf", "otf", "woff", "woff2"}
|
||||
|
||||
|
||||
async def _read_asset(upload: Optional[UploadFile], label: str) -> Optional[Tuple[bytes, str]]:
|
||||
"""Read+validate an uploaded theme asset (image or font)."""
|
||||
if upload is None or not upload.filename:
|
||||
return None
|
||||
ext = upload.filename.rsplit(".", 1)[-1].lower() if "." in upload.filename else ""
|
||||
if ext not in ASSET_EXTENSIONS:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"{label}: unsupported asset type '.{ext}'. Use an image or font file.",
|
||||
)
|
||||
data = await upload.read()
|
||||
if not data:
|
||||
return None
|
||||
if len(data) > MAX_IMAGE_BYTES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"{label}: file is too large ({len(data) // (1024 * 1024)}MB). Max 15MB.",
|
||||
)
|
||||
return data, ext
|
||||
|
||||
|
||||
@router.get("", response_model=TenantListResponse)
|
||||
def admin_list_tenants(auth_data: AuthResult = Depends(require_admin)):
|
||||
"""List all white-label tenants."""
|
||||
return TenantListResponse(tenants=[TenantSummary(**t) for t in list_tenants()])
|
||||
|
||||
|
||||
@router.get("/_template", response_model=TenantTemplateResponse)
|
||||
def admin_tenant_theme_template(auth_data: AuthResult = Depends(require_admin)):
|
||||
"""Return the default Nomad theme's full style_params as an editing starting point."""
|
||||
try:
|
||||
return TenantTemplateResponse(style_params=get_default_style_params())
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=500, detail=str(exc))
|
||||
|
||||
|
||||
# Sync handlers: FastAPI runs them in its threadpool, so the blocking GCS and
|
||||
# Cloudflare calls don't stall the event loop (the admin UI polls this).
|
||||
@router.get("/{tenant_id}", response_model=TenantDetailResponse)
|
||||
def admin_get_tenant(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
|
||||
"""Return a tenant's full config, theme style_params, and asset list for editing."""
|
||||
try:
|
||||
detail = get_tenant_detail(tenant_id)
|
||||
except TenantNotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc))
|
||||
return TenantDetailResponse(
|
||||
preview_url=_preview_url(tenant_id),
|
||||
domain=get_tenant_domain_status(tenant_id),
|
||||
**detail,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/{tenant_id}/domain", response_model=TenantDomainResponse)
|
||||
def admin_provision_tenant_domain(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
|
||||
"""(Re)provision the tenant's {id}.nomadkaraoke.com subdomain (idempotent)."""
|
||||
try:
|
||||
domain = provision_tenant_domain(tenant_id)
|
||||
except TenantNotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc))
|
||||
except TenantConflictError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc))
|
||||
except TenantProvisioningError as exc:
|
||||
raise HTTPException(status_code=502, detail=str(exc))
|
||||
logger.info(f"Admin {auth_data.user_email or 'admin:unknown'} provisioned domain for tenant '{tenant_id}'")
|
||||
return TenantDomainResponse(domain=domain)
|
||||
|
||||
|
||||
@router.delete("/{tenant_id}", status_code=204)
|
||||
def admin_delete_tenant(tenant_id: str, auth_data: AuthResult = Depends(require_admin)):
|
||||
"""Delete a tenant: subdomain (Cloudflare), theme, and config. Jobs are kept."""
|
||||
try:
|
||||
delete_tenant(tenant_id)
|
||||
except TenantNotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc))
|
||||
except TenantProvisioningError as exc:
|
||||
raise HTTPException(status_code=502, detail=str(exc))
|
||||
logger.info(f"Admin {auth_data.user_email or 'admin:unknown'} deleted tenant '{tenant_id}'")
|
||||
|
||||
|
||||
@router.put("/{tenant_id}", response_model=TenantUpdateResponse)
|
||||
async def admin_update_tenant(
|
||||
tenant_id: str,
|
||||
auth_data: AuthResult = Depends(require_admin),
|
||||
config: Optional[str] = Form(None),
|
||||
style_params: Optional[str] = Form(None),
|
||||
logo: Optional[UploadFile] = File(None),
|
||||
assets: Optional[List[UploadFile]] = File(None),
|
||||
):
|
||||
"""
|
||||
Update a tenant: merge config fields (JSON), replace the full theme
|
||||
style_params (JSON), and/or add/replace theme assets (uploaded files keyed
|
||||
by their filename). Re-render jobs to apply an updated theme.
|
||||
"""
|
||||
config_updates = _parse_json_field(config, "config")
|
||||
if config_updates is not None and not isinstance(config_updates, dict):
|
||||
raise HTTPException(status_code=400, detail="config must be a JSON object.")
|
||||
style = _parse_json_field(style_params, "style_params")
|
||||
|
||||
asset_map: Dict[str, Tuple[bytes, str]] = {}
|
||||
for upload in assets or []:
|
||||
asset = await _read_asset(upload, f"asset '{upload.filename}'")
|
||||
if asset and upload.filename:
|
||||
asset_map[upload.filename] = asset
|
||||
logo_image = await _read_image(logo, "logo")
|
||||
|
||||
try:
|
||||
updated = await run_in_threadpool(
|
||||
update_tenant,
|
||||
tenant_id,
|
||||
config_updates=config_updates,
|
||||
style_params=style,
|
||||
assets=asset_map or None,
|
||||
logo=logo_image,
|
||||
)
|
||||
except TenantNotFoundError as exc:
|
||||
raise HTTPException(status_code=404, detail=str(exc))
|
||||
except TenantConflictError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc))
|
||||
except TenantValidationError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except TenantProvisioningError as exc:
|
||||
raise HTTPException(status_code=502, detail=str(exc))
|
||||
except ValidationError as exc:
|
||||
raise HTTPException(status_code=400, detail=f"Invalid config: {exc}")
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.exception("Failed to update tenant")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to update tenant: {exc}")
|
||||
|
||||
admin_id = auth_data.user_email or "admin:unknown"
|
||||
logger.info(f"Admin {admin_id} updated tenant '{tenant_id}'")
|
||||
return TenantUpdateResponse(tenant=TenantPublicConfig.from_config(updated))
|
||||
|
||||
|
||||
@router.post("", response_model=TenantCreateResponse, status_code=201)
|
||||
async def admin_create_tenant(
|
||||
auth_data: AuthResult = Depends(require_admin),
|
||||
name: str = Form(...),
|
||||
tenant_id: Optional[str] = Form(None),
|
||||
subdomain: Optional[str] = Form(None),
|
||||
allowed_email_domains: Optional[str] = Form(None),
|
||||
allowed_emails: Optional[str] = Form(None),
|
||||
artist_color: Optional[str] = Form(None),
|
||||
title_color: Optional[str] = Form(None),
|
||||
sung_lyrics_color: Optional[str] = Form(None),
|
||||
unsung_lyrics_color: Optional[str] = Form(None),
|
||||
tagline: Optional[str] = Form(None),
|
||||
distribution_mode: str = Form("download_only"),
|
||||
dropbox_path: Optional[str] = Form(None),
|
||||
brand_prefix: Optional[str] = Form(None),
|
||||
style_params: Optional[str] = Form(None),
|
||||
karaoke_background: Optional[UploadFile] = File(None),
|
||||
intro_background: Optional[UploadFile] = File(None),
|
||||
end_background: Optional[UploadFile] = File(None),
|
||||
logo: Optional[UploadFile] = File(None),
|
||||
):
|
||||
"""Provision a new white-label tenant (theme + config) from admin form input."""
|
||||
try:
|
||||
colors = ColorOverrides(
|
||||
artist_color=_clean(artist_color),
|
||||
title_color=_clean(title_color),
|
||||
sung_lyrics_color=_clean(sung_lyrics_color),
|
||||
unsung_lyrics_color=_clean(unsung_lyrics_color),
|
||||
)
|
||||
except ValidationError:
|
||||
raise HTTPException(
|
||||
status_code=400, detail="Colors must be hex values like #ff5bb8."
|
||||
)
|
||||
|
||||
backgrounds = {}
|
||||
for field, upload in (
|
||||
("karaoke_background", karaoke_background),
|
||||
("intro_background", intro_background),
|
||||
("end_background", end_background),
|
||||
):
|
||||
image = await _read_image(upload, field)
|
||||
if image:
|
||||
backgrounds[field] = image
|
||||
logo_image = await _read_image(logo, "logo")
|
||||
|
||||
if distribution_mode not in ("all", "download_only", "cloud_only"):
|
||||
raise HTTPException(status_code=400, detail="Invalid distribution_mode.")
|
||||
|
||||
style_override = _parse_json_field(style_params, "style_params")
|
||||
if style_override is not None and not isinstance(style_override, dict):
|
||||
raise HTTPException(status_code=400, detail="style_params must be a JSON object.")
|
||||
|
||||
try:
|
||||
config = await run_in_threadpool(
|
||||
create_tenant,
|
||||
name=name,
|
||||
tenant_id=_clean(tenant_id),
|
||||
subdomain=_clean(subdomain),
|
||||
allowed_email_domains=_parse_domains(allowed_email_domains),
|
||||
allowed_emails=_parse_domains(allowed_emails),
|
||||
colors=colors,
|
||||
style_params_override=style_override,
|
||||
tagline=_clean(tagline),
|
||||
distribution_mode=distribution_mode,
|
||||
dropbox_path=_clean(dropbox_path),
|
||||
brand_prefix=_clean(brand_prefix),
|
||||
backgrounds=backgrounds,
|
||||
logo=logo_image,
|
||||
)
|
||||
except TenantConflictError as exc:
|
||||
raise HTTPException(status_code=409, detail=str(exc))
|
||||
except TenantValidationError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except TenantProvisioningError as exc:
|
||||
raise HTTPException(status_code=502, detail=str(exc))
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
except Exception as exc: # pragma: no cover - defensive
|
||||
logger.exception("Failed to create tenant")
|
||||
raise HTTPException(status_code=500, detail=f"Failed to create tenant: {exc}")
|
||||
|
||||
admin_id = auth_data.user_email or "admin:unknown"
|
||||
logger.info(f"Admin {admin_id} created tenant '{config.id}'")
|
||||
|
||||
return TenantCreateResponse(
|
||||
tenant=TenantPublicConfig.from_config(config),
|
||||
preview_url=_preview_url(config.id),
|
||||
subdomain_url=config.get_frontend_url(),
|
||||
domain=await run_in_threadpool(get_tenant_domain_status, config.id),
|
||||
)
|
||||
@@ -1,89 +0,0 @@
|
||||
"""Authenticated tenant bulk-upload endpoints.
|
||||
|
||||
Separate from ``tenant.py`` (which is public config only): these require a
|
||||
tenant session and gate on the ``bulk_upload`` feature flag.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from backend.services.auth_service import AuthResult
|
||||
from pathlib import Path
|
||||
|
||||
from backend.services.tenant_bulk import analyze_filenames
|
||||
from backend.services.tenant_bulk.analyze import AUDIO_EXTENSIONS, default_generate
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant/bulk", tags=["tenant-bulk"])
|
||||
|
||||
# Mirror consumer Bulk Mode's cap to bound LLM cost and upload load. Counts
|
||||
# audio files (candidate tracks), not incidental non-audio files a folder pick
|
||||
# may include (cover art, etc.), so an image-heavy folder isn't falsely blocked.
|
||||
MAX_AUDIO_FILES = 100
|
||||
# Absolute payload guard so a single request can't send an unbounded list.
|
||||
MAX_FILENAMES = 2000
|
||||
|
||||
|
||||
class BulkAnalyzeRequest(BaseModel):
|
||||
filenames: List[str] = Field(
|
||||
..., description="Filenames from the operator's folder pick (no paths needed)"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/analyze")
|
||||
async def analyze_bulk_filenames(
|
||||
request: Request,
|
||||
body: BulkAnalyzeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Analyse a list of filenames into proposed Mixed/Instrumental job rows.
|
||||
|
||||
Pure analysis: filenames only, no uploads, no audio, no state written.
|
||||
Returns ``{rows, unpaired, ignored}`` for an editable review table.
|
||||
"""
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if not tenant_config:
|
||||
raise HTTPException(
|
||||
status_code=403, detail="Bulk upload is only available on tenant portals"
|
||||
)
|
||||
if not getattr(tenant_config.features, "bulk_upload", False):
|
||||
raise HTTPException(
|
||||
status_code=403, detail="Bulk upload is not enabled for this portal"
|
||||
)
|
||||
|
||||
filenames = [f for f in (body.filenames or []) if f and f.strip()]
|
||||
if not filenames:
|
||||
raise HTTPException(status_code=400, detail="No filenames provided")
|
||||
if len(filenames) > MAX_FILENAMES:
|
||||
raise HTTPException(status_code=400, detail="Too many files in request")
|
||||
audio_count = sum(
|
||||
1 for f in filenames if Path(f).suffix.lower() in AUDIO_EXTENSIONS
|
||||
)
|
||||
if audio_count > MAX_AUDIO_FILES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Too many tracks ({audio_count}). Please submit at most {MAX_AUDIO_FILES} at a time.",
|
||||
)
|
||||
|
||||
# The LLM pass is a blocking Vertex call; run it off the event loop. It only
|
||||
# fires for files the regex could not pair, and analyze_filenames falls back
|
||||
# to the regex result if the model errors — so this never hard-fails.
|
||||
def _run() -> dict:
|
||||
return analyze_filenames(filenames, generate=default_generate).to_dict()
|
||||
|
||||
analysis = await asyncio.to_thread(_run)
|
||||
logger.info(
|
||||
"tenant-bulk analyze: tenant=%s files=%d rows=%d unpaired=%d ignored=%d",
|
||||
tenant_config.id,
|
||||
len(filenames),
|
||||
len(analysis["rows"]),
|
||||
len(analysis["unpaired"]),
|
||||
len(analysis["ignored"]),
|
||||
)
|
||||
return analysis
|
||||
@@ -1,298 +0,0 @@
|
||||
"""
|
||||
Tenant self-service theme editor API (``/api/tenant/theme``).
|
||||
|
||||
Any user allowed on a tenant's portal can view, preview and save that tenant's
|
||||
theme. Tenant context comes from the request (subdomain or the client-controlled
|
||||
``X-Tenant-ID`` header), so every endpoint re-checks membership against the
|
||||
tenant's allowlist (admins always pass) — a header alone grants nothing.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from typing import Any, Dict, List, Literal, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, File, Form, HTTPException, Request, UploadFile
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from pydantic import BaseModel, Field, constr
|
||||
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from backend.models.tenant import TenantConfig
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.tenant_admin_service import TenantValidationError
|
||||
from backend.services.tenant_theme_service import (
|
||||
ThemeNotEditableError,
|
||||
ThemeNotFoundError,
|
||||
get_theme_for_editor,
|
||||
prepare_preview_styles,
|
||||
save_tenant_theme,
|
||||
store_uploaded_asset,
|
||||
)
|
||||
from backend.services.theme_change_service import outdated_jobs, refresh_inflight_jobs
|
||||
from backend.services.theme_preview_service import ThemePreviewError, render_theme_preview
|
||||
from backend.services.tenant_admin_service import _theme_id_for
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant/theme", tags=["tenant-theme"])
|
||||
|
||||
MAX_UPLOAD_BYTES = 15 * 1024 * 1024
|
||||
|
||||
# Previews run ffmpeg/PIL on the API instance — bound concurrency like the review
|
||||
# preview path so a burst of edits can't starve request handling.
|
||||
_PREVIEW_SEMAPHORE = asyncio.Semaphore(2)
|
||||
_PREVIEW_EXECUTOR = ThreadPoolExecutor(max_workers=2, thread_name_prefix="theme-preview")
|
||||
PREVIEW_TIMEOUT_S = 45
|
||||
|
||||
|
||||
def require_tenant_member(
|
||||
request: Request, auth_result: AuthResult = Depends(require_auth)
|
||||
) -> TenantConfig:
|
||||
"""The request's tenant, iff the authenticated user is allowed on it."""
|
||||
config = get_tenant_config_from_request(request)
|
||||
if config is None or not config.is_active:
|
||||
raise HTTPException(status_code=404, detail="Theme editing is only available on a tenant portal.")
|
||||
if auth_result.tenant_id and auth_result.tenant_id != config.id:
|
||||
raise HTTPException(status_code=403, detail="Your sign-in belongs to a different portal.")
|
||||
if not auth_result.user_email or not config.is_email_allowed(auth_result.user_email):
|
||||
raise HTTPException(status_code=403, detail="You don't have access to this portal's theme.")
|
||||
return config
|
||||
|
||||
|
||||
def _raise_http(exc: TenantValidationError) -> None:
|
||||
if isinstance(exc, ThemeNotFoundError):
|
||||
raise HTTPException(status_code=404, detail=str(exc))
|
||||
if isinstance(exc, ThemeNotEditableError):
|
||||
raise HTTPException(status_code=403, detail=str(exc))
|
||||
raise HTTPException(status_code=400, detail=str(exc))
|
||||
|
||||
|
||||
class ThemeResponse(BaseModel):
|
||||
theme_id: str
|
||||
style_params: Dict[str, Any]
|
||||
images: List[str]
|
||||
fonts: List[str]
|
||||
# The images that fit the 300x216 CD+G screen (offered on the CD+G tab).
|
||||
cdg_images: List[str] = Field(default_factory=list)
|
||||
# Built-in CD+G reveal transitions (greyscale draw-order masks).
|
||||
cdg_transitions: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class PreviewSample(BaseModel):
|
||||
artist: str = Field("Artist Name", max_length=80)
|
||||
title: str = Field("Song Title", max_length=80)
|
||||
lyrics: Optional[List[constr(max_length=120)]] = Field(None, max_length=4)
|
||||
|
||||
|
||||
class PreviewRequest(BaseModel):
|
||||
style_params: Dict[str, Any]
|
||||
sample: PreviewSample = Field(default_factory=PreviewSample)
|
||||
# Also render the CD+G up card + a lyrics page (slower; the editor asks on its CD+G tab).
|
||||
include_cdg: bool = False
|
||||
# MP4 screens to render (default all): the editor asks only for the tab's screen.
|
||||
kinds: Optional[List[Literal["title_card", "karaoke_frame", "end_screen"]]] = None
|
||||
|
||||
|
||||
class PreviewResponse(BaseModel):
|
||||
# data:image/jpeg;base64,... — null for a screen the theme leaves out of videos.
|
||||
title_card: Optional[str] = None
|
||||
karaoke_frame: Optional[str] = None
|
||||
end_screen: Optional[str] = None
|
||||
# data:image/png;base64,... — only when include_cdg was requested.
|
||||
cdg_title_card: Optional[str] = None
|
||||
cdg_lyrics: Optional[str] = None
|
||||
cdg_instrumental: Optional[str] = None
|
||||
cdg_outro: Optional[str] = None
|
||||
|
||||
|
||||
class SaveRequest(BaseModel):
|
||||
style_params: Dict[str, Any]
|
||||
|
||||
|
||||
class AssetResponse(BaseModel):
|
||||
name: str
|
||||
|
||||
|
||||
@router.get("", response_model=ThemeResponse)
|
||||
def get_theme(config: TenantConfig = Depends(require_tenant_member)):
|
||||
try:
|
||||
return ThemeResponse(**get_theme_for_editor(config))
|
||||
except TenantValidationError as exc:
|
||||
_raise_http(exc)
|
||||
|
||||
|
||||
@router.post("/assets", response_model=AssetResponse)
|
||||
async def upload_asset(
|
||||
file: UploadFile = File(...),
|
||||
purpose: Optional[str] = Form(None),
|
||||
config: TenantConfig = Depends(require_tenant_member),
|
||||
):
|
||||
if purpose not in (None, "", "cdg"):
|
||||
raise HTTPException(status_code=400, detail="Unknown upload purpose.")
|
||||
# Read in chunks and stop as soon as the limit is exceeded, so an oversized
|
||||
# upload never gets materialised in memory in full.
|
||||
chunks, size = [], 0
|
||||
while True:
|
||||
chunk = await file.read(1024 * 1024)
|
||||
if not chunk:
|
||||
break
|
||||
size += len(chunk)
|
||||
if size > MAX_UPLOAD_BYTES:
|
||||
raise HTTPException(status_code=400, detail="The file is too large (max 15 MB).")
|
||||
chunks.append(chunk)
|
||||
data = b"".join(chunks)
|
||||
if not data:
|
||||
raise HTTPException(status_code=400, detail="The file is empty.")
|
||||
try:
|
||||
name = await run_in_threadpool(
|
||||
lambda: store_uploaded_asset(config, file.filename or "asset", data, purpose=purpose or None)
|
||||
)
|
||||
except TenantValidationError as exc:
|
||||
_raise_http(exc)
|
||||
return AssetResponse(name=name)
|
||||
|
||||
|
||||
@router.post("/preview", response_model=PreviewResponse)
|
||||
async def preview_theme(body: PreviewRequest, config: TenantConfig = Depends(require_tenant_member)):
|
||||
try:
|
||||
styles = await run_in_threadpool(prepare_preview_styles, config, body.style_params)
|
||||
except TenantValidationError as exc:
|
||||
_raise_http(exc)
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
await _PREVIEW_SEMAPHORE.acquire()
|
||||
future = loop.run_in_executor(
|
||||
_PREVIEW_EXECUTOR,
|
||||
lambda: render_theme_preview(
|
||||
_theme_id_for(config),
|
||||
styles,
|
||||
artist=body.sample.artist,
|
||||
title=body.sample.title,
|
||||
lyrics=body.sample.lyrics,
|
||||
include_cdg=body.include_cdg,
|
||||
kinds=body.kinds,
|
||||
),
|
||||
)
|
||||
# The slot is released only when the render really finishes — a timed-out
|
||||
# render keeps running in its thread, so releasing early would let a burst
|
||||
# queue unbounded work behind it.
|
||||
future.add_done_callback(lambda _f: _PREVIEW_SEMAPHORE.release())
|
||||
try:
|
||||
images = await asyncio.wait_for(asyncio.shield(future), timeout=PREVIEW_TIMEOUT_S)
|
||||
except ThemePreviewError as exc:
|
||||
raise HTTPException(status_code=422, detail=str(exc))
|
||||
except asyncio.TimeoutError:
|
||||
raise HTTPException(status_code=504, detail="The preview took too long to render. Try again.")
|
||||
return PreviewResponse(**images.as_data_urls())
|
||||
|
||||
|
||||
class SaveResponse(ThemeResponse):
|
||||
# In-progress tracks switched to the new theme by this save.
|
||||
refreshed_jobs: int = 0
|
||||
# The caller's finished tracks made with an older theme (re-render to update).
|
||||
outdated_job_ids: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
class OutdatedJobsResponse(BaseModel):
|
||||
theme_updated_at: Optional[str] = None
|
||||
job_ids: List[str]
|
||||
|
||||
|
||||
class RerenderOutdatedResponse(BaseModel):
|
||||
started: List[str]
|
||||
failed: Dict[str, str]
|
||||
# Outdated tracks beyond this call's cap (ask again to start them).
|
||||
remaining: List[str] = Field(default_factory=list)
|
||||
|
||||
|
||||
# Tracks started per bulk call: bounds the request time and the burst of
|
||||
# render/encode work it queues. The UI offers the remainder again.
|
||||
MAX_BULK_RERENDER = 20
|
||||
|
||||
|
||||
def _owner_scope(auth_result: AuthResult) -> Optional[str]:
|
||||
"""Every caller here is a portal member (require_tenant_member), and tenant
|
||||
portals are shared workspaces (backend/services/job_access.py), so theme
|
||||
refresh/rerender covers all of the portal's tracks, whoever submitted them."""
|
||||
return None
|
||||
|
||||
|
||||
@router.put("", response_model=SaveResponse)
|
||||
def save_theme(
|
||||
body: SaveRequest,
|
||||
config: TenantConfig = Depends(require_tenant_member),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
try:
|
||||
save_tenant_theme(config, body.style_params)
|
||||
theme = get_theme_for_editor(config)
|
||||
except TenantValidationError as exc:
|
||||
_raise_http(exc)
|
||||
theme_id = _theme_id_for(config)
|
||||
# Never fail the save over the follow-up work: the theme is already stored.
|
||||
refreshed = 0
|
||||
try:
|
||||
refreshed = refresh_inflight_jobs(config.id, theme_id)["updated"]
|
||||
except Exception:
|
||||
logger.exception(f"Tenant '{config.id}': refreshing in-progress tracks after theme save failed")
|
||||
outdated: List[str] = []
|
||||
try:
|
||||
outdated = outdated_jobs(config.id, theme_id, _owner_scope(auth_result))["job_ids"]
|
||||
except Exception:
|
||||
logger.exception(f"Tenant '{config.id}': listing outdated tracks after theme save failed")
|
||||
return SaveResponse(**theme, refreshed_jobs=refreshed, outdated_job_ids=outdated)
|
||||
|
||||
|
||||
@router.get("/outdated-jobs", response_model=OutdatedJobsResponse)
|
||||
def get_outdated_jobs(
|
||||
config: TenantConfig = Depends(require_tenant_member),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Finished tracks made with an older version of the theme (re-render to update)."""
|
||||
return OutdatedJobsResponse(**outdated_jobs(config.id, _theme_id_for(config), _owner_scope(auth_result)))
|
||||
|
||||
|
||||
@router.post("/rerender-outdated", response_model=RerenderOutdatedResponse)
|
||||
async def rerender_outdated(
|
||||
config: TenantConfig = Depends(require_tenant_member),
|
||||
auth_result: AuthResult = Depends(require_auth),
|
||||
):
|
||||
"""Re-render every outdated finished track with the current theme.
|
||||
|
||||
Quiet: no per-track "your video is ready" email/push (one per track would
|
||||
flood the inbox); the portal shows progress on each track.
|
||||
"""
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.theme_rerender_service import RerenderError, ThemeRerenderService
|
||||
|
||||
theme_id = _theme_id_for(config)
|
||||
job_ids = (await run_in_threadpool(
|
||||
outdated_jobs, config.id, theme_id, _owner_scope(auth_result)
|
||||
))["job_ids"]
|
||||
batch, remaining = job_ids[:MAX_BULK_RERENDER], job_ids[MAX_BULK_RERENDER:]
|
||||
job_manager = JobManager()
|
||||
service = ThemeRerenderService(job_manager)
|
||||
started: List[str] = []
|
||||
failed: Dict[str, str] = {}
|
||||
for job_id in batch:
|
||||
job = await run_in_threadpool(job_manager.get_job, job_id)
|
||||
if job is None:
|
||||
continue
|
||||
try:
|
||||
await service.start(
|
||||
job, theme_id=theme_id,
|
||||
requested_by=auth_result.user_email or "unknown",
|
||||
notify_customer=False,
|
||||
)
|
||||
started.append(job_id)
|
||||
except RerenderError as e:
|
||||
failed[job_id] = str(e)
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Bulk theme re-render failed to start")
|
||||
failed[job_id] = "Couldn't start the re-render."
|
||||
logger.info(
|
||||
f"Tenant '{config.id}': bulk re-render started {len(started)}, failed {len(failed)}, "
|
||||
f"remaining {len(remaining)}"
|
||||
)
|
||||
return RerenderOutdatedResponse(started=started, failed=failed, remaining=remaining)
|
||||
Loaded 100 of 2766 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user