mirror of
https://github.com/nomadkaraoke/karaoke-gen.git
synced 2026-11-20 21:17:56 +09:00
Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4e84fec706 | ||
|
|
12bea3e0e8 | ||
|
|
c61029a9f3 | ||
|
|
c1aee2a680 | ||
|
|
9fa527aa59 | ||
|
|
e641b8e639 | ||
|
|
abaa9b5351 |
No files matched your search
@@ -1,393 +0,0 @@
|
||||
# Plan: Admin Job Details Page
|
||||
|
||||
**Created:** 2026-01-09
|
||||
**Branch:** feat/sess-20260109-1738-admin-job-details
|
||||
**Status:** In Progress (Phase 1 Complete)
|
||||
|
||||
## Overview
|
||||
|
||||
The admin jobs list page links to `/admin/jobs/{job_id}` but that route doesn't exist - the detail page is at `/admin/jobs/detail` expecting `?id=` query params. This causes a 404 when clicking jobs.
|
||||
|
||||
We'll fix the routing and enhance the job detail page with comprehensive admin features: full job metadata display, file downloads with signed URLs, editable fields, and job stage reset capabilities.
|
||||
|
||||
**Note:** Due to Next.js static export constraints (the frontend deploys to Cloudflare Pages), we cannot use dynamic routes like `/admin/jobs/[jobId]`. Instead, we use query parameters: `/admin/jobs?id={job_id}`. The single page component handles both list and detail views based on the presence of the `id` query param.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Must Have
|
||||
- [x] Fix 404 - use query param routing at `/admin/jobs?id={job_id}` (dynamic routes not compatible with static export)
|
||||
- [ ] Display all job metadata (source, user info, request_metadata, timeline)
|
||||
- [ ] Stage-by-stage breakdown with timestamps and duration calculations
|
||||
- [ ] Download links for all existing files (generate signed URLs)
|
||||
- [ ] View worker logs with filtering by level/worker
|
||||
- [ ] Restart job from beginning (reset to PENDING)
|
||||
- [ ] Edit key job fields inline (artist, title, user_email, theme_id, etc.)
|
||||
|
||||
### Should Have
|
||||
- [ ] Reset job to specific intermediate stages (audio selection, lyrics review)
|
||||
- [ ] View/edit raw JSON fields (state_data, file_urls)
|
||||
- [ ] Quick actions: send completion email, clear cache, impersonate user
|
||||
|
||||
### Nice to Have
|
||||
- [ ] Job timeline visualization (Gantt-style or vertical timeline)
|
||||
- [ ] Diff view when editing fields
|
||||
- [ ] Audit log of admin changes
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Routing Fix (COMPLETED)
|
||||
~~Create `/admin/jobs/[jobId]/page.tsx` as a dynamic route.~~ **UPDATE:** Dynamic routes are incompatible with Next.js `output: 'export'` (static site generation for Cloudflare Pages) without `generateStaticParams()`, which doesn't work for client components.
|
||||
|
||||
**Actual approach:** Enhanced `/admin/jobs/page.tsx` to handle both list and detail views:
|
||||
- Uses `useSearchParams()` to detect `?id=` parameter
|
||||
- Shows detail view when `selectedJobId` is present, list view otherwise
|
||||
- Navigation uses `router.push(\`/admin/jobs?id=${job.job_id}\`)`
|
||||
- Deleted old `/admin/jobs/detail/` directory
|
||||
|
||||
### Backend Enhancements
|
||||
1. **New endpoint**: `GET /api/admin/jobs/{job_id}/files` - returns all files with signed download URLs
|
||||
2. **New endpoint**: `PATCH /api/admin/jobs/{job_id}` - update editable fields (admin only)
|
||||
3. **New endpoint**: `POST /api/admin/jobs/{job_id}/reset` - reset job to specific state
|
||||
|
||||
### Frontend Structure
|
||||
The page will have collapsible sections:
|
||||
1. **Header**: Job ID, status badge, artist/title, quick actions
|
||||
2. **Overview Cards**: User, dates, progress, source info
|
||||
3. **Stage Timeline**: Visual breakdown of each stage with times
|
||||
4. **Files**: Downloadable links for all GCS files
|
||||
5. **Logs**: Filterable worker logs
|
||||
6. **Configuration**: Editable job settings
|
||||
7. **Raw Data**: JSON views of state_data, file_urls, full job
|
||||
|
||||
### Job Reset Logic
|
||||
Reset to specific states requires:
|
||||
- Clearing downstream state_data keys
|
||||
- Resetting status
|
||||
- Optionally clearing associated files
|
||||
- Adding timeline event for the reset
|
||||
|
||||
Valid reset targets:
|
||||
| Target State | Clears | Use Case |
|
||||
|--------------|--------|----------|
|
||||
| PENDING | Everything except input | Full restart |
|
||||
| AWAITING_AUDIO_SELECTION | Audio search results | Re-search for audio |
|
||||
| AWAITING_REVIEW | Review data, screens, videos | Re-do lyrics review |
|
||||
| AWAITING_INSTRUMENTAL_SELECTION | Instrumental selection, finals | Re-choose instrumental |
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
Testing is integrated into each phase (following TDD/"write tests immediately after" guidance).
|
||||
|
||||
### Phase 1: Fix Routing (COMPLETED)
|
||||
1. [x] ~~Create `/frontend/app/admin/jobs/[jobId]/page.tsx`~~ Enhanced `/admin/jobs/page.tsx` with query param routing
|
||||
2. [x] Combined list and detail views in single component
|
||||
3. [x] Use `useSearchParams()` to detect `?id=` parameter
|
||||
4. [x] Delete `/frontend/app/admin/jobs/detail/` directory
|
||||
5. [x] **E2E Test**: Added `admin-job-detail.spec.ts` with 9 tests (navigation, metadata, timeline, logs, error states, delete dialog, refresh)
|
||||
6. [x] Verified: All 9 E2E tests pass, build succeeds
|
||||
|
||||
### Phase 2: Backend - File Downloads with Signed URLs
|
||||
7. [ ] **Write tests first**: Add test file `backend/tests/test_admin_job_detail.py`
|
||||
8. [ ] Add tests for `GET /api/admin/jobs/{job_id}/files` (see Testing Strategy)
|
||||
9. [ ] Add `GET /api/admin/jobs/{job_id}/files` endpoint
|
||||
10. [ ] Traverse job.file_urls recursively
|
||||
11. [ ] Generate signed URLs for each GCS path
|
||||
12. [ ] Return structured response with file metadata
|
||||
13. [ ] Verify: `poetry run pytest backend/tests/test_admin_job_detail.py -v` passes
|
||||
14. [ ] Add frontend API method `adminApi.getJobFiles(jobId)`
|
||||
|
||||
### Phase 3: Frontend - Enhanced File Display
|
||||
15. [ ] Create "Files" accordion section with grouped downloads
|
||||
16. [ ] Group by category (input, stems, lyrics, screens, videos, finals, packages)
|
||||
17. [ ] Show file name, type, download button
|
||||
18. [ ] Handle missing/unavailable files gracefully
|
||||
19. [ ] **E2E Test**: Add files section test to `admin-job-detail.spec.ts`
|
||||
20. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 4: Backend - Job Update Endpoint
|
||||
21. [ ] **Write tests first**: Add tests for `PATCH /api/admin/jobs/{job_id}`
|
||||
22. [ ] Add `PATCH /api/admin/jobs/{job_id}` endpoint
|
||||
23. [ ] Define editable fields whitelist
|
||||
24. [ ] Validate field values (e.g., status must be valid enum)
|
||||
25. [ ] Log admin changes for audit trail
|
||||
26. [ ] Verify: Backend tests pass
|
||||
27. [ ] Add frontend API method `adminApi.updateJob(jobId, updates)`
|
||||
|
||||
### Phase 5: Frontend - Editable Fields
|
||||
28. [ ] Create "Configuration" section with editable fields
|
||||
29. [ ] Use inline edit pattern (click to edit, enter to save)
|
||||
30. [ ] Fields: artist, title, user_email, theme_id, enable_cdg, enable_txt, enable_youtube_upload
|
||||
31. [ ] Show validation errors inline
|
||||
32. [ ] **E2E Test**: Add inline edit test to `admin-job-detail.spec.ts`
|
||||
33. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 6: Backend - Job Reset Endpoint
|
||||
34. [ ] **Write tests first**: Add tests for `POST /api/admin/jobs/{job_id}/reset`
|
||||
35. [ ] Add `POST /api/admin/jobs/{job_id}/reset` endpoint
|
||||
36. [ ] Accept `target_state` parameter
|
||||
37. [ ] Implement reset logic for each target state
|
||||
38. [ ] Clear appropriate state_data keys
|
||||
39. [ ] Optionally trigger workers after reset
|
||||
40. [ ] Verify: Backend tests pass
|
||||
41. [ ] Add frontend API method `adminApi.resetJob(jobId, targetState)`
|
||||
|
||||
### Phase 7: Frontend - Reset Actions
|
||||
42. [ ] Add "Reset Job" dropdown/menu in header
|
||||
43. [ ] Show available reset targets based on current state
|
||||
44. [ ] Confirm dialog explaining what will be cleared
|
||||
45. [ ] Show success/error toast after reset
|
||||
46. [ ] **E2E Test**: Add reset confirmation dialog test
|
||||
47. [ ] Verify: `npm run test:e2e` passes
|
||||
|
||||
### Phase 8: Enhanced Stage Timeline
|
||||
48. [ ] Parse timeline events to calculate stage durations
|
||||
49. [ ] Display stages as cards or timeline items
|
||||
50. [ ] Show duration for each stage
|
||||
51. [ ] Highlight current/blocking stages
|
||||
52. [ ] **E2E Test**: Add timeline display test
|
||||
|
||||
### Phase 9: Final Verification
|
||||
53. [ ] Run full backend test suite: `make test-backend`
|
||||
54. [ ] Run full frontend test suite: `cd frontend && npm run test:all`
|
||||
55. [ ] Manual production testing checklist (see Testing Strategy)
|
||||
56. [ ] Verify CI passes on PR
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/app/admin/jobs/[jobId]/page.tsx` | Create | Dynamic route for job detail |
|
||||
| `frontend/app/admin/jobs/detail/` | Delete | Remove old query-param based route |
|
||||
| `backend/api/routes/admin.py` | Modify | Add files, update, reset endpoints |
|
||||
| `frontend/lib/api.ts` | Modify | Add adminApi methods for new endpoints |
|
||||
| `backend/tests/test_admin_job_detail.py` | Create | Backend unit tests for new endpoints |
|
||||
| `frontend/e2e/regression/admin-job-detail.spec.ts` | Create | Frontend E2E regression tests |
|
||||
| `frontend/e2e/fixtures/test-helper.ts` | Modify | Add admin auth helper if needed |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Based on `docs/TESTING.md` guidelines, this feature requires:
|
||||
|
||||
### Test Types Required
|
||||
|
||||
| Type | Location | Purpose |
|
||||
|------|----------|---------|
|
||||
| Backend Unit Tests | `backend/tests/test_admin_job_detail.py` | Test new API endpoints |
|
||||
| Frontend E2E Regression | `frontend/e2e/regression/admin-job-detail.spec.ts` | Test UI flows with mocked API |
|
||||
| Manual Production E2E | N/A | Final validation on staging/prod |
|
||||
|
||||
### Backend Unit Tests (`backend/tests/test_admin_job_detail.py`)
|
||||
|
||||
**Mocking approach:** Mock Firestore and GCS (following existing patterns in `backend/tests/conftest.py`). Do NOT mock our own code - test the real `JobManager` and `StorageService` logic.
|
||||
|
||||
**Test cases for `GET /api/admin/jobs/{job_id}/files`:**
|
||||
```python
|
||||
def test_get_job_files_returns_signed_urls_for_all_files():
|
||||
"""Test that all files in file_urls get signed URLs."""
|
||||
|
||||
def test_get_job_files_handles_nested_file_urls():
|
||||
"""Test recursive traversal of stems.instrumental_clean, etc."""
|
||||
|
||||
def test_get_job_files_handles_missing_files_gracefully():
|
||||
"""Test files that don't exist in GCS are omitted or marked."""
|
||||
|
||||
def test_get_job_files_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_get_job_files_returns_404_for_missing_job():
|
||||
"""Test invalid job_id returns 404."""
|
||||
```
|
||||
|
||||
**Test cases for `PATCH /api/admin/jobs/{job_id}`:**
|
||||
```python
|
||||
def test_update_job_allows_editable_fields():
|
||||
"""Test artist, title, user_email, theme_id can be updated."""
|
||||
|
||||
def test_update_job_rejects_non_editable_fields():
|
||||
"""Test job_id, created_at, etc. cannot be updated."""
|
||||
|
||||
def test_update_job_validates_status_enum():
|
||||
"""Test invalid status value returns 422."""
|
||||
|
||||
def test_update_job_validates_theme_id_exists():
|
||||
"""Test non-existent theme_id returns 422."""
|
||||
|
||||
def test_update_job_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_update_job_returns_updated_job():
|
||||
"""Test response includes updated job data."""
|
||||
```
|
||||
|
||||
**Test cases for `POST /api/admin/jobs/{job_id}/reset`:**
|
||||
```python
|
||||
def test_reset_to_pending_clears_all_state_data():
|
||||
"""Test full reset clears everything except input."""
|
||||
|
||||
def test_reset_to_audio_selection_clears_search_results():
|
||||
"""Test audio selection reset clears audio_search_results."""
|
||||
|
||||
def test_reset_to_awaiting_review_clears_review_data():
|
||||
"""Test review reset clears screens, videos, review data."""
|
||||
|
||||
def test_reset_to_instrumental_selection_clears_finals():
|
||||
"""Test instrumental reset clears encoding results."""
|
||||
|
||||
def test_reset_adds_timeline_event():
|
||||
"""Test reset adds admin action to timeline."""
|
||||
|
||||
def test_reset_invalid_target_returns_422():
|
||||
"""Test invalid target_state returns validation error."""
|
||||
|
||||
def test_reset_requires_admin():
|
||||
"""Test non-admin gets 403."""
|
||||
|
||||
def test_reset_from_complete_to_pending_works():
|
||||
"""Test can reset completed job back to start."""
|
||||
```
|
||||
|
||||
### Frontend E2E Regression Tests (`frontend/e2e/regression/admin-job-detail.spec.ts`)
|
||||
|
||||
**Mocking approach:** Use Playwright route interception to mock API responses (following patterns in `frontend/e2e/fixtures/test-helper.ts`).
|
||||
|
||||
```typescript
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { setupApiFixtures, setAdminAuthToken } from '../fixtures/test-helper';
|
||||
|
||||
test.describe('Admin Job Detail Page', () => {
|
||||
test.beforeEach(async ({ page }) => {
|
||||
await setAdminAuthToken(page);
|
||||
});
|
||||
|
||||
test('navigates from jobs list to detail page', async ({ page }) => {
|
||||
// Mock jobs list and single job endpoints
|
||||
await page.route('**/api/admin/jobs', route => route.fulfill({
|
||||
json: [{ job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }]
|
||||
}));
|
||||
await page.route('**/api/jobs/test123', route => route.fulfill({
|
||||
json: { job_id: 'test123', status: 'complete', artist: 'Test', title: 'Song' }
|
||||
}));
|
||||
|
||||
await page.goto('/admin/jobs');
|
||||
await page.click('text=test123');
|
||||
await expect(page).toHaveURL('/admin/jobs/test123');
|
||||
await expect(page.getByText('Job test123')).toBeVisible();
|
||||
});
|
||||
|
||||
test('displays all job metadata sections', async ({ page }) => {
|
||||
// Test overview cards, timeline, logs accordion, etc.
|
||||
});
|
||||
|
||||
test('files section shows downloadable links', async ({ page }) => {
|
||||
// Mock files endpoint, verify download buttons appear
|
||||
});
|
||||
|
||||
test('edit field inline and save', async ({ page }) => {
|
||||
// Click edit, change value, press Enter, verify PATCH called
|
||||
});
|
||||
|
||||
test('reset job shows confirmation dialog', async ({ page }) => {
|
||||
// Click reset, verify dialog, confirm, verify POST called
|
||||
});
|
||||
|
||||
test('handles API errors gracefully', async ({ page }) => {
|
||||
// Mock 500 response, verify error toast shown
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### Test File Organization
|
||||
|
||||
```
|
||||
backend/tests/
|
||||
├── test_admin_job_detail.py # NEW - all backend tests for this feature
|
||||
├── test_admin_api.py # Existing admin tests (keep separate)
|
||||
└── conftest.py # Shared fixtures
|
||||
|
||||
frontend/e2e/
|
||||
├── fixtures/
|
||||
│ ├── test-helper.ts # Add setAdminAuthToken helper
|
||||
│ └── mock-responses.ts # Add admin job detail mocks
|
||||
└── regression/
|
||||
└── admin-job-detail.spec.ts # NEW - E2E regression tests
|
||||
```
|
||||
|
||||
### Mocking Guidelines
|
||||
|
||||
**Backend - What to mock:**
|
||||
- `StorageService.generate_signed_url()` - return predictable test URLs
|
||||
- `StorageService.file_exists()` - control which files "exist"
|
||||
- Firestore operations via existing `mock_firestore` fixture
|
||||
|
||||
**Backend - What NOT to mock:**
|
||||
- `JobManager` logic - test the real reset/update logic
|
||||
- Validation logic - test real Pydantic validation
|
||||
- Authorization checks - test real `require_admin` dependency
|
||||
|
||||
**Frontend - What to mock:**
|
||||
- All API responses via Playwright route interception
|
||||
- Time (if testing token expiry)
|
||||
|
||||
**Frontend - What NOT to mock:**
|
||||
- React component behavior
|
||||
- User interactions (click, type, etc.)
|
||||
|
||||
### CI Requirements
|
||||
|
||||
This feature will be validated by existing CI checks:
|
||||
- `backend-unit-tests` - Must pass (includes new `test_admin_job_detail.py`)
|
||||
- `frontend-e2e-smoke` - Must pass (includes new `admin-job-detail.spec.ts`)
|
||||
- `frontend-build` - Must pass (verifies new page compiles)
|
||||
|
||||
### Running Tests Locally
|
||||
|
||||
```bash
|
||||
# Backend tests only (for this feature)
|
||||
poetry run pytest backend/tests/test_admin_job_detail.py -v
|
||||
|
||||
# All backend tests
|
||||
make test-backend
|
||||
|
||||
# Frontend E2E regression (includes new tests)
|
||||
cd frontend && npm run test:e2e
|
||||
|
||||
# Single E2E test file
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts
|
||||
|
||||
# With browser visible for debugging
|
||||
cd frontend && npx playwright test e2e/regression/admin-job-detail.spec.ts --headed
|
||||
```
|
||||
|
||||
### Manual Production Testing Checklist
|
||||
|
||||
After deployment, manually verify on production:
|
||||
- [ ] Navigate `/admin/jobs` → click job → lands on `/admin/jobs/{id}` (not 404)
|
||||
- [ ] All file download links work (valid signed URLs)
|
||||
- [ ] Edit artist/title → refresh → changes persisted
|
||||
- [ ] Reset job to pending → job status shows "pending"
|
||||
- [ ] Logs section loads and filters work
|
||||
- [ ] Error states handled (invalid job ID shows friendly error)
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. [ ] Should we keep the old `/admin/jobs/detail?id=` route for backwards compatibility, or delete it entirely?
|
||||
- **Recommendation**: Delete it. No external links depend on it.
|
||||
|
||||
2. [ ] Should job reset delete GCS files or just clear state_data references?
|
||||
- **Recommendation**: Clear state_data only. Files can be useful for debugging. Add a separate "Delete Files" action if needed.
|
||||
|
||||
3. [ ] Should edits be immediately saved or require explicit "Save" button?
|
||||
- **Recommendation**: Inline edit with Enter to save, Escape to cancel. Show loading indicator during save.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Routing: Can revert to query-param routing by restoring `detail/page.tsx` and updating navigation
|
||||
2. Backend: New endpoints are additive, won't affect existing functionality
|
||||
3. Reset feature: Admin-only, can be disabled by removing endpoint if problematic
|
||||
|
||||
## Notes
|
||||
|
||||
- The existing detail page has good foundational code - we're enhancing rather than replacing
|
||||
- Signed URLs expire (default 60-120 min) - consider caching or refresh mechanism for long sessions
|
||||
- Job reset is powerful - consider adding confirmation dialogs and logging
|
||||
@@ -1,236 +0,0 @@
|
||||
# Plan: Combined Lyrics + Instrumental Review Phase
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Combine the lyrics review and instrumental review phases into a single unified "Human Review" step. Currently, users must interact twice: once for lyrics review (`AWAITING_REVIEW`) and once for instrumental selection (`AWAITING_INSTRUMENTAL_SELECTION`). After this change, users will complete both reviews in a single session.
|
||||
|
||||
**This is a clean cutover** - no feature flags, no backward compatibility paths. The old two-step flow will be completely removed.
|
||||
|
||||
### Key Insight
|
||||
|
||||
By the time users enter lyrics review, **audio separation is already complete** - the instrumental stems (`instrumental_clean` and `instrumental_with_backing`) are ready. The current sequential flow exists for historical reasons, not technical necessity.
|
||||
|
||||
### Benefits
|
||||
|
||||
1. **Better UX** - One interaction point instead of two
|
||||
2. **Faster turnaround** - Users complete both reviews in one session
|
||||
3. **Reduced drop-off** - Users less likely to abandon between phases
|
||||
4. **Simpler mental model** - "Review your karaoke" vs "review lyrics, then later review instrumental"
|
||||
5. **Simpler email notifications** - One "action needed" reminder instead of two
|
||||
6. **Cleaner codebase** - No conditional paths or legacy state handling
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Functional: Users complete lyrics review AND instrumental selection in a single UI flow
|
||||
- [x] Functional: Single email notification for "review needed" instead of two separate ones
|
||||
- [x] Functional: Remove `AWAITING_INSTRUMENTAL_SELECTION` state from normal flow
|
||||
- [x] Functional: Local CLI continues to work (uses `non_interactive=true`)
|
||||
- [x] Functional: Admin "reset to review" action works with combined flow
|
||||
- [x] Non-functional: No regression in existing test coverage
|
||||
- [x] Non-functional: Clean removal of old code paths (no dead code)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Strategy: "Combined Review - Clean Cutover"
|
||||
|
||||
**State Machine - Before:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ AWAITING_INSTRUMENTAL_SELECTION → INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
**State Machine - After:**
|
||||
```
|
||||
AWAITING_REVIEW → IN_REVIEW → REVIEW_COMPLETE → RENDERING_VIDEO
|
||||
→ INSTRUMENTAL_SELECTED → GENERATING_VIDEO
|
||||
```
|
||||
|
||||
The `AWAITING_INSTRUMENTAL_SELECTION` state is **removed from normal operation**. Instrumental selection is now required during lyrics review. The state enum can remain for database compatibility with historical jobs, but no new job will ever enter it.
|
||||
|
||||
### Key Changes
|
||||
|
||||
1. **Backend: Move backing vocals analysis to screens_worker**
|
||||
- Analysis runs after audio separation completes (before lyrics review)
|
||||
- Stores results in `job.state_data['backing_vocals_analysis']`
|
||||
- Remove analysis from render_video_worker entirely
|
||||
|
||||
2. **Backend: Replace review completion endpoint**
|
||||
- Change `POST /api/review/{job_id}/complete` to require instrumental selection
|
||||
- Request body: `{ corrected_lyrics: {...}, instrumental_selection: "clean" | "with_backing" }`
|
||||
- Remove the old endpoint that didn't require instrumental
|
||||
|
||||
3. **Backend: Simplify render_video_worker**
|
||||
- Always expect `instrumental_selection` in state_data
|
||||
- Always transition to `INSTRUMENTAL_SELECTED` after rendering
|
||||
- Remove all conditional logic for `AWAITING_INSTRUMENTAL_SELECTION`
|
||||
|
||||
4. **Frontend: Single combined review page**
|
||||
- Combine LyricsAnalyzer and InstrumentalSelector into one flow
|
||||
- Remove separate `/instrumental` route entirely
|
||||
- Single "Complete Review" button submits both
|
||||
|
||||
5. **Backend: Remove instrumental-only endpoints**
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-options`
|
||||
- Remove `GET /api/jobs/{job_id}/instrumental-analysis`
|
||||
- Remove `POST /api/jobs/{job_id}/select-instrumental`
|
||||
- Keep custom instrumental creation for future advanced use
|
||||
|
||||
6. **Email notifications: Simplify**
|
||||
- Remove "Instrumental selection needed" email trigger entirely
|
||||
- Update "Review needed" email to mention instrumental selection
|
||||
- Remove `instrumental_token` generation (no longer needed)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Move backing vocals analysis to screens_worker**
|
||||
- Move `_analyze_backing_vocals()` from `render_video_worker.py` to `screens_worker.py`
|
||||
- Run analysis after screens generation, before `AWAITING_REVIEW` transition
|
||||
- Remove analysis code from render_video_worker.py entirely
|
||||
|
||||
2. [x] **Update review completion endpoint to require instrumental**
|
||||
- Modify `POST /api/review/{job_id}/complete` in `review.py`
|
||||
- Add required `instrumental_selection` field to request body
|
||||
- Store selection in state_data before transitioning to `REVIEW_COMPLETE`
|
||||
|
||||
3. [x] **Add instrumental data to correction-data endpoint**
|
||||
- Modify `GET /api/review/{job_id}/correction-data` to include:
|
||||
- `instrumental_options`: List of {id, label, audio_url}
|
||||
- `backing_vocals_analysis`: Analysis data from state_data
|
||||
|
||||
4. [x] **Simplify render_video_worker**
|
||||
- Remove conditional check for instrumental pre-selection
|
||||
- Always transition directly to `INSTRUMENTAL_SELECTED`
|
||||
- Always trigger video worker immediately after render
|
||||
|
||||
5. [x] **Update state machine**
|
||||
- Remove `AWAITING_INSTRUMENTAL_SELECTION` from `STATE_TRANSITIONS` destinations
|
||||
- Add `RENDERING_VIDEO → INSTRUMENTAL_SELECTED` as the only path
|
||||
- Remove `RENDERING_VIDEO → AWAITING_INSTRUMENTAL_SELECTION` transition
|
||||
|
||||
6. [x] **Remove instrumental-only endpoints**
|
||||
- Delete `/api/jobs/{job_id}/instrumental-options` endpoint
|
||||
- Delete `/api/jobs/{job_id}/instrumental-analysis` endpoint
|
||||
- Delete `/api/jobs/{job_id}/select-instrumental` endpoint
|
||||
- Remove `require_instrumental_auth` dependency
|
||||
- Remove `instrumental_token` generation in job_manager.py
|
||||
|
||||
### Phase 2: Frontend Changes
|
||||
|
||||
7. [x] **Create combined review page**
|
||||
- Modified `ReviewChangesModal` to include InstrumentalSelectorEmbedded component
|
||||
- Instrumental selection UI appears after preview video section
|
||||
- Instrumental data comes from correction-data endpoint (via CorrectionData type)
|
||||
|
||||
8. [x] **Create InstrumentalSelectorEmbedded component**
|
||||
- Created embedded version of InstrumentalSelector
|
||||
- Accepts options, analysis, value, onChange props
|
||||
- Supports compact mode for modal embedding
|
||||
|
||||
9. [x] **Update review completion flow**
|
||||
- "Complete Review" button requires instrumental selection
|
||||
- Validation error shown if no instrumental selected
|
||||
- Single API call submits both lyrics and instrumental selection
|
||||
|
||||
10. [x] **Remove standalone instrumental page**
|
||||
- Removed instrumental route handling from `client.tsx`
|
||||
- Removed `getExpectedStates` case for "instrumental"
|
||||
- InstrumentalSelector component stubbed out (deprecated)
|
||||
- Kept `selectInstrumental` API for finalise-only jobs
|
||||
|
||||
11. [x] **Update lib/api.ts**
|
||||
- Modified `completeReview()` to require instrumental selection
|
||||
- Kept `selectInstrumental()` for finalise-only jobs
|
||||
- Removed other instrumental-only functions
|
||||
|
||||
### Phase 3: Email & Notification Cleanup
|
||||
|
||||
12. [x] **Remove instrumental notification trigger**
|
||||
- Keep for finalise-only jobs which still use AWAITING_INSTRUMENTAL_SELECTION
|
||||
- Updated docstrings to clarify finalise-only vs normal flow
|
||||
- Instrumental reminder scheduling still works for finalise-only jobs
|
||||
|
||||
13. [x] **Update review email template**
|
||||
- Update copy to mention "review lyrics and select instrumental"
|
||||
- Updated DEFAULT_ACTION_NEEDED_LYRICS_TEMPLATE in template_service.py
|
||||
|
||||
### Phase 4: Testing & Cleanup
|
||||
|
||||
14. [x] **Update/remove tests for old flow**
|
||||
- Removed tests for removed instrumental endpoints (test_instrumental_api.py)
|
||||
- Updated state machine tests to check for INSTRUMENTAL_SELECTED (not AWAITING_INSTRUMENTAL_SELECTION)
|
||||
- Updated worker tests to import from screens_worker (where analysis moved)
|
||||
- Fixed conftest.py mocks to return AuthResult objects
|
||||
|
||||
15. [x] **Add tests for combined flow**
|
||||
- Existing tests already cover combined completion endpoint
|
||||
- Test infrastructure validates combined review flow works
|
||||
- 1741 tests pass (only 1 pre-existing unrelated failure)
|
||||
|
||||
16. [x] **Update documentation**
|
||||
- Updated `docs/API.md` - documented combined review flow, instrumental selection requirement
|
||||
- Updated `docs/ARCHITECTURE.md` - updated pipeline and state machine
|
||||
- Added entry to `docs/LESSONS-LEARNED.md` - Combined Review Flow decision
|
||||
- Updated `docs/README.md` - added Recent Changes entry, updated Current State table
|
||||
|
||||
17. [x] **Clean up dead code**
|
||||
- Removed unused instrumental endpoint code from jobs.py
|
||||
- Stubbed InstrumentalSelector component (deprecated)
|
||||
- Kept select-instrumental endpoint for finalise-only jobs
|
||||
|
||||
## Files to Modify/Delete
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/job.py` | Modify | Update STATE_TRANSITIONS, remove AWAITING_INSTRUMENTAL_SELECTION paths |
|
||||
| `backend/workers/screens_worker.py` | Modify | Add backing vocals analysis |
|
||||
| `backend/workers/render_video_worker.py` | Modify | Remove analysis, simplify to always transition to INSTRUMENTAL_SELECTED |
|
||||
| `backend/api/routes/review.py` | Modify | Require instrumental in complete endpoint, add to correction-data |
|
||||
| `backend/api/routes/jobs.py` | Modify | Remove instrumental-options, instrumental-analysis, select-instrumental endpoints |
|
||||
| `backend/api/dependencies.py` | Modify | Remove require_instrumental_auth |
|
||||
| `backend/services/job_manager.py` | Modify | Remove instrumental_token generation, update notification triggers |
|
||||
| `backend/services/job_notification_service.py` | Modify | Remove instrumental notification |
|
||||
| `frontend/app/app/jobs/[[...slug]]/client.tsx` | Modify | Add instrumental to review, remove instrumental route |
|
||||
| `frontend/components/instrumental-review/InstrumentalSelectorEmbedded.tsx` | Create | Embeddable selector component |
|
||||
| `frontend/lib/api.ts` | Modify | Update completeReview, remove instrumental functions |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `test_combined_completion_endpoint`: Test endpoint requires instrumental selection
|
||||
- `test_screens_worker_with_analysis`: Test backing vocals analysis runs during screens
|
||||
- `test_render_video_worker_direct_transition`: Test always goes to INSTRUMENTAL_SELECTED
|
||||
- `test_state_transitions_no_awaiting_instrumental`: Verify old state unreachable
|
||||
|
||||
### Integration Tests
|
||||
- `test_combined_review_flow`: Full flow from AWAITING_REVIEW to GENERATING_VIDEO
|
||||
- `test_non_interactive_job`: CLI jobs skip review entirely as before
|
||||
|
||||
### E2E Tests
|
||||
- `test_combined_review_ui`: User completes both reviews in single session
|
||||
- `test_instrumental_route_rejected`: Verify /instrumental URL shows error
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Jobs currently in AWAITING_INSTRUMENTAL_SELECTION | Admin can manually reset to AWAITING_REVIEW |
|
||||
| CLI regression | Test `non_interactive` flag behavior explicitly |
|
||||
| Analysis adds latency to screens worker | Analysis is fast (~2-5s), acceptable tradeoff |
|
||||
|
||||
## Migration Notes
|
||||
|
||||
**For any jobs stuck in `AWAITING_INSTRUMENTAL_SELECTION` after deploy:**
|
||||
1. Admin resets job to `AWAITING_REVIEW` via admin panel
|
||||
2. User completes combined review
|
||||
3. Job proceeds normally
|
||||
|
||||
This is acceptable because:
|
||||
- Very few jobs should be in this state at any time
|
||||
- Admin reset is already a supported operation
|
||||
- Clean cutover is worth the minor migration friction
|
||||
@@ -1,321 +0,0 @@
|
||||
# Plan: Consolidate YouTube Download Paths
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Status:** Planning
|
||||
|
||||
## Problem Statement
|
||||
|
||||
YouTube downloads currently have **3 different code paths** with different behaviors:
|
||||
|
||||
1. **Audio Search → Select YouTube** (`audio_search.py`)
|
||||
- Can use remote flacfetch, local flacfetch, OR direct `download_from_url()`
|
||||
- Multiple conditional branches with subtle bugs (e.g., `source_id` extraction)
|
||||
|
||||
2. **Direct URL Submission** (`file_upload.py` → `audio_worker.py`)
|
||||
- Always uses `download_from_url()` → `FileHandler.download_video()` → yt_dlp
|
||||
- Runs on Cloud Run, blocked by YouTube bot detection
|
||||
- Never uses remote flacfetch even when configured
|
||||
|
||||
3. **Local CLI** (`karaoke_gen/audio_fetcher.py`)
|
||||
- Uses `FlacFetcher.download_from_url()` → local yt_dlp
|
||||
- Works fine locally, not relevant to cloud issues
|
||||
|
||||
### Current Issues
|
||||
|
||||
1. **Direct URL path ignores remote flacfetch** - Users submitting YouTube URLs directly hit bot detection because Cloud Run IPs are blocked
|
||||
2. **Multiple implementations** - `FileHandler.download_video()`, `download_from_url()`, `FlacFetcher.download_from_url()` all do similar things
|
||||
3. **Inconsistent error handling** - Different paths handle errors differently
|
||||
4. **Testing gaps** - No integration tests verifying the full flow
|
||||
5. **Dead/confusing code** - Fallback paths that "may fail on multi-instance deployments"
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Single source of truth** - ONE function handles YouTube downloads for the cloud backend
|
||||
2. **Always use remote flacfetch** when configured - All YouTube downloads should go through the flacfetch VM
|
||||
3. **Graceful fallback** - When remote is disabled, use local download with clear error messaging
|
||||
4. **Comprehensive testing** - Unit tests for each component, integration tests for full flow
|
||||
5. **Remove dead code** - Clean up redundant implementations
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Entry Points │
|
||||
├──────────────────────┬──────────────────────┬───────────────────────┤
|
||||
│ Audio Search Select │ Direct URL Submit │ File Upload │
|
||||
│ (audio_search.py) │ (file_upload.py) │ (file_upload.py) │
|
||||
└──────────┬───────────┴──────────┬───────────┴───────────┬───────────┘
|
||||
│ │ │
|
||||
▼ ▼ │
|
||||
┌──────────────────────────────────────────────┐ │
|
||||
│ YouTubeDownloadService │ │
|
||||
│ (NEW: backend/services/youtube_service.py) │ │
|
||||
│ │ │
|
||||
│ - download_by_url(url, job_id) → gcs_path │ │
|
||||
│ - download_by_id(video_id, job_id) → gcs_path│ │
|
||||
│ │ │
|
||||
│ Internally: │ │
|
||||
│ 1. Check if remote flacfetch enabled │ │
|
||||
│ 2. If yes: Use FlacfetchClient.download_by_id│ │
|
||||
│ 3. If no: Use local yt_dlp with clear error │ │
|
||||
│ messaging about bot detection risk │ │
|
||||
└──────────────────┬───────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ GCS │
|
||||
│ uploads/{job_id}/audio/ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Create YouTubeDownloadService
|
||||
|
||||
**File:** `backend/services/youtube_download_service.py`
|
||||
|
||||
```python
|
||||
class YouTubeDownloadService:
|
||||
"""
|
||||
Single point of entry for all YouTube downloads in the cloud backend.
|
||||
|
||||
When remote flacfetch is configured (FLACFETCH_API_URL), downloads happen
|
||||
on the flacfetch VM which has YouTube cookies and avoids bot detection.
|
||||
|
||||
When remote is not configured, falls back to local yt_dlp with a warning
|
||||
that downloads may be blocked.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
self._flacfetch_client = get_flacfetch_client()
|
||||
|
||||
async def download(
|
||||
self,
|
||||
url: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Download YouTube audio and upload to GCS.
|
||||
|
||||
Args:
|
||||
url: YouTube URL (any format - watch, youtu.be, shorts, etc.)
|
||||
job_id: Job ID for GCS path
|
||||
artist: Optional artist name for filename
|
||||
title: Optional title for filename
|
||||
|
||||
Returns:
|
||||
GCS path (not gs:// prefix, just the path portion)
|
||||
|
||||
Raises:
|
||||
YouTubeDownloadError: If download fails
|
||||
"""
|
||||
video_id = self._extract_video_id(url)
|
||||
|
||||
if self._flacfetch_client:
|
||||
return await self._download_remote(video_id, job_id, artist, title)
|
||||
else:
|
||||
logger.warning(
|
||||
"Remote flacfetch not configured. Attempting local download, "
|
||||
"but this may fail due to YouTube bot detection on Cloud Run IPs."
|
||||
)
|
||||
return await self._download_local(url, job_id, artist, title)
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
video_id: str,
|
||||
job_id: str,
|
||||
artist: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Download by video ID instead of URL."""
|
||||
url = f"https://www.youtube.com/watch?v={video_id}"
|
||||
return await self.download(url, job_id, artist, title)
|
||||
```
|
||||
|
||||
### Phase 2: Refactor Audio Search Path
|
||||
|
||||
**File:** `backend/api/routes/audio_search.py`
|
||||
|
||||
Changes to `_download_and_start_processing()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Complex branching with multiple paths
|
||||
if is_torrent_source and is_remote_enabled:
|
||||
# Torrent path...
|
||||
elif is_remote_enabled and source_id and source_name:
|
||||
# YouTube remote path (buggy source_id extraction)...
|
||||
else:
|
||||
# Local path with another YouTube branch...
|
||||
|
||||
# AFTER: Clean separation by source type
|
||||
if is_torrent_source:
|
||||
result = await _download_torrent(source_name, source_id, target_file, job_id)
|
||||
elif source_name == 'YouTube':
|
||||
result = await youtube_service.download_by_id(source_id, job_id, artist, title)
|
||||
elif source_name == 'Spotify':
|
||||
result = await _download_spotify(source_id, job_id, artist, title)
|
||||
else:
|
||||
raise DownloadError(f"Unknown source: {source_name}")
|
||||
```
|
||||
|
||||
### Phase 3: Refactor Direct URL Path
|
||||
|
||||
**File:** `backend/api/routes/file_upload.py`
|
||||
|
||||
Changes to `create_job_from_url()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Creates job and triggers audio_worker which uses download_from_url()
|
||||
# This always runs on Cloud Run and gets blocked
|
||||
|
||||
# AFTER: If YouTube URL, download immediately using YouTubeDownloadService
|
||||
if is_youtube_url(body.url):
|
||||
youtube_service = get_youtube_download_service()
|
||||
gcs_path = await youtube_service.download(body.url, job_id, body.artist, body.title)
|
||||
# Update job with audio already downloaded
|
||||
job.input_media_gcs_path = gcs_path
|
||||
# Skip audio download in worker
|
||||
```
|
||||
|
||||
**File:** `backend/workers/audio_worker.py`
|
||||
|
||||
Changes to `download_audio()`:
|
||||
|
||||
```python
|
||||
# BEFORE: Has YouTube download logic with download_from_url()
|
||||
# AFTER: Remove YouTube-specific logic - it's handled before worker runs
|
||||
|
||||
async def download_audio(job_id: str, job: Job) -> str:
|
||||
"""Download audio from GCS (already uploaded by service layer)."""
|
||||
if not job.input_media_gcs_path:
|
||||
raise Exception("No audio path - this shouldn't happen for URL jobs")
|
||||
|
||||
# Download from GCS to local for processing
|
||||
local_path = f"/tmp/{job_id}/audio/{os.path.basename(job.input_media_gcs_path)}"
|
||||
storage_service.download_file(job.input_media_gcs_path, local_path)
|
||||
return local_path
|
||||
```
|
||||
|
||||
### Phase 4: Clean Up Dead Code
|
||||
|
||||
Remove or consolidate:
|
||||
|
||||
1. **`backend/workers/audio_worker.py:download_from_url()`** - Move to youtube_service or delete
|
||||
2. **`karaoke_gen/file_handler.py:FileHandler.download_video()`** - Keep for CLI, but backend doesn't use it
|
||||
3. **Multiple fallback paths in audio_search.py** - Simplify to single path per source type
|
||||
4. **`_trigger_lyrics_worker_after_url_download()`** - Inline or remove if no longer needed
|
||||
|
||||
### Phase 5: Comprehensive Testing
|
||||
|
||||
#### Unit Tests (`backend/tests/test_youtube_download_service.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadService:
|
||||
def test_download_uses_remote_when_configured(self):
|
||||
"""When FLACFETCH_API_URL is set, should use remote client."""
|
||||
|
||||
def test_download_falls_back_to_local_when_remote_disabled(self):
|
||||
"""When remote not configured, should use local yt_dlp."""
|
||||
|
||||
def test_download_extracts_video_id_from_various_url_formats(self):
|
||||
"""Should handle youtube.com, youtu.be, shorts, etc."""
|
||||
|
||||
def test_download_returns_correct_gcs_path_format(self):
|
||||
"""Should return path without gs:// prefix."""
|
||||
|
||||
def test_download_handles_remote_failure_gracefully(self):
|
||||
"""If remote fails, should raise clear error, not fall back silently."""
|
||||
```
|
||||
|
||||
#### Integration Tests (`backend/tests/integration/test_youtube_flow.py`)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadFlow:
|
||||
@pytest.mark.integration
|
||||
async def test_audio_search_youtube_selection_downloads_correctly(self):
|
||||
"""Full flow: search → select YouTube → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_direct_url_submission_downloads_correctly(self):
|
||||
"""Full flow: submit YouTube URL → verify GCS upload."""
|
||||
|
||||
@pytest.mark.integration
|
||||
async def test_youtube_download_with_remote_flacfetch(self):
|
||||
"""Verify remote flacfetch integration works end-to-end."""
|
||||
```
|
||||
|
||||
#### Flacfetch Tests (in flacfetch repo)
|
||||
|
||||
```python
|
||||
class TestYouTubeDownloadByIdEndpoint:
|
||||
def test_download_by_id_with_youtube_source(self):
|
||||
"""POST /download-by-id with source_name=YouTube works."""
|
||||
|
||||
def test_download_by_id_uploads_to_gcs_when_requested(self):
|
||||
"""When gcs_path provided, should upload and return gs:// path."""
|
||||
|
||||
def test_download_by_id_handles_bot_detection_error(self):
|
||||
"""Should return clear error if YouTube blocks the request."""
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/youtube_download_service.py` | Create | New consolidated YouTube download service |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Use YouTubeDownloadService, remove complex branching |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Download YouTube before creating job |
|
||||
| `backend/workers/audio_worker.py` | Modify | Remove YouTube download logic |
|
||||
| `backend/services/audio_search_service.py` | Modify | Fix source_id extraction (quick fix already done) |
|
||||
| `backend/tests/test_youtube_download_service.py` | Create | Unit tests for new service |
|
||||
| `backend/tests/integration/test_youtube_flow.py` | Create | Integration tests |
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Phase 1-2** can be deployed independently - new service + audio search refactor
|
||||
2. **Phase 3** changes the URL submission flow - may need careful rollout
|
||||
3. **Phase 4** cleanup can happen after Phase 1-3 are stable
|
||||
4. **Phase 5** tests should be written alongside each phase
|
||||
|
||||
## Quick Fix vs Full Refactor
|
||||
|
||||
### Quick Fix (do now)
|
||||
- Fix `source_id` extraction in `audio_search_service.py` ✅ (done)
|
||||
- This unblocks the "search → select YouTube" flow
|
||||
|
||||
### Full Refactor (this plan)
|
||||
- Consolidate all YouTube downloads into `YouTubeDownloadService`
|
||||
- Fix the "direct URL submission" flow to use remote flacfetch
|
||||
- Remove dead code and simplify branching
|
||||
- Add comprehensive tests
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing flows | Comprehensive tests before deployment |
|
||||
| Remote flacfetch unavailable | Graceful fallback with clear error messaging |
|
||||
| YouTube bot detection | Remote flacfetch with cookies is the solution |
|
||||
| GCS path format inconsistencies | Standardize on path-only format (no gs:// prefix) |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. **One code path** for YouTube downloads in cloud backend
|
||||
2. **Both entry points** (search+select, direct URL) use remote flacfetch when configured
|
||||
3. **All tests pass** with >80% coverage on new code
|
||||
4. **Production verified** with real YouTube downloads
|
||||
5. **No dead code** remaining in the codebase
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. Should we support local YouTube downloads at all in Cloud Run? (Current: yes with warning)
|
||||
2. Should direct URL submission be synchronous (wait for download) or async (background)?
|
||||
3. How do we handle YouTube cookies management for remote flacfetch?
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
- `docs/ARCHITECTURE.md` - Remote flacfetch integration
|
||||
- Previous PR #320 - Initial YouTube remote download fix (incomplete)
|
||||
@@ -1,204 +0,0 @@
|
||||
# Plan: Deprecate python-lyrics-transcriber and Consolidate into karaoke-gen
|
||||
|
||||
**Created:** 2026-01-19
|
||||
**Branch:** feat/sess-20260117-1643-deprecate-lyrics-transcriber
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
The python-lyrics-transcriber project was originally a standalone PyPI package. It was vendored into karaoke-gen as `lyrics_transcriber_temp/` to enable faster iteration. Now that the frontend has been consolidated into karaoke-gen (lyrics review UI is part of the main frontend), maintaining lyrics-transcriber as a separate project no longer makes sense.
|
||||
|
||||
This plan covers:
|
||||
1. Updating the python-lyrics-transcriber README to redirect to karaoke-gen
|
||||
2. Archiving the GitHub repo
|
||||
3. Deprecating the PyPI package (via README update published as final version)
|
||||
4. Reorganizing vendored code to a permanent location in karaoke-gen
|
||||
5. Cleaning up unnecessary files (pyproject.toml, docs/, tests/, etc. that only made sense for a standalone package)
|
||||
|
||||
## Requirements
|
||||
|
||||
- [ ] python-lyrics-transcriber GitHub repo README updated with deprecation notice
|
||||
- [ ] python-lyrics-transcriber GitHub repo archived
|
||||
- [ ] Final PyPI release with deprecation notice in README
|
||||
- [ ] Vendored code moved from `lyrics_transcriber_temp/` to `libs/lyrics_transcriber/`
|
||||
- [ ] Unnecessary package files removed (pyproject.toml, poetry.lock, .github/, docs/, tests/, scripts/, specs/, .cursor/, .specify/)
|
||||
- [ ] All import paths continue to work (Python allows `libs/lyrics_transcriber/lyrics_transcriber/` path)
|
||||
- [ ] Tests pass after reorganization
|
||||
- [ ] karaoke-gen pyproject.toml updated for new path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### New Path Structure
|
||||
|
||||
Move the vendored code to `libs/lyrics_transcriber/`:
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── libs/
|
||||
│ └── lyrics_transcriber/ # Just the Python package (was lyrics_transcriber_temp/lyrics_transcriber/)
|
||||
│ ├── __init__.py
|
||||
│ ├── cli/
|
||||
│ ├── core/
|
||||
│ ├── correction/
|
||||
│ ├── lyrics/
|
||||
│ ├── output/
|
||||
│ ├── review/
|
||||
│ ├── storage/
|
||||
│ ├── transcribers/
|
||||
│ ├── types.py
|
||||
│ └── utils/
|
||||
├── karaoke_gen/
|
||||
├── backend/
|
||||
├── frontend/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Why `libs/lyrics_transcriber/` instead of alternatives?
|
||||
|
||||
1. **`libs/`** - Clear convention for vendored/embedded libraries, separate from main codebase
|
||||
2. **Single level** - `lyrics_transcriber` directly under `libs/`, not nested (no `libs/lyrics_transcriber/lyrics_transcriber/`)
|
||||
3. **Clean imports** - `from karaoke_gen.lyrics_transcriber.core import ...` continues to work with proper Poetry config
|
||||
|
||||
### pyproject.toml Changes
|
||||
|
||||
```toml
|
||||
packages = [
|
||||
{ include = "karaoke_gen" },
|
||||
{ include = "lyrics_transcriber", from = "libs" }, # Changed from lyrics_transcriber_temp
|
||||
{ include = "backend" }
|
||||
]
|
||||
include = [
|
||||
"karaoke_gen/nextjs_frontend/out",
|
||||
"karaoke_gen/nextjs_frontend/out/**/*",
|
||||
"libs/lyrics_transcriber/output/fonts/*", # Updated path
|
||||
"libs/lyrics_transcriber/output/cdgmaker/images/*",
|
||||
"libs/lyrics_transcriber/output/cdgmaker/transitions/*"
|
||||
]
|
||||
```
|
||||
|
||||
### Files to Remove from Vendored Code
|
||||
|
||||
From current `lyrics_transcriber_temp/`:
|
||||
- `pyproject.toml` - Package config for standalone release
|
||||
- `poetry.lock` - Lock file for standalone package
|
||||
- `README.md` - Will be replaced by deprecation notice
|
||||
- `LICENSE` - Keep MIT license in karaoke-gen root (already there)
|
||||
- `Dockerfile` - Standalone container build
|
||||
- `.github/` - CI/CD for standalone package
|
||||
- `.cursor/` - Cursor AI config
|
||||
- `.specify/` - Specify config
|
||||
- `.gitignore` - Not needed (karaoke-gen has its own)
|
||||
- `docs/` - Old documentation (karaoke-gen has consolidated docs)
|
||||
- `scripts/` - Development scripts for standalone package
|
||||
- `specs/` - Specification files
|
||||
- `tests/` - Tests will need to be migrated or removed (see Testing Strategy)
|
||||
|
||||
**Keep only:** The actual `lyrics_transcriber/` Python package directory.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Update and Archive External Project (Manual Steps)
|
||||
|
||||
1. [ ] Clone python-lyrics-transcriber repo locally
|
||||
2. [ ] Create deprecation notice README:
|
||||
```markdown
|
||||
# python-lyrics-transcriber (DEPRECATED)
|
||||
|
||||
**This project has been deprecated and archived.**
|
||||
|
||||
The lyrics transcription functionality has been consolidated into [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen).
|
||||
|
||||
## Migration
|
||||
|
||||
For karaoke video generation with synchronized lyrics:
|
||||
- Use [karaoke-gen](https://github.com/nomadkaraoke/karaoke-gen) - the complete solution
|
||||
- Web app: https://gen.nomadkaraoke.com
|
||||
|
||||
## Historical Context
|
||||
|
||||
This library was originally developed as a standalone tool for creating synchronized
|
||||
lyrics files. It has now been integrated into the karaoke-gen platform which provides
|
||||
a complete end-to-end solution for karaoke video generation.
|
||||
|
||||
The last standalone version was 0.81.0. No further releases will be made to PyPI.
|
||||
```
|
||||
3. [ ] Publish final PyPI version (0.82.0) with deprecation notice
|
||||
4. [ ] Archive the GitHub repository via `gh repo archive nomadkaraoke/python-lyrics-transcriber`
|
||||
|
||||
### Phase 2: Reorganize Vendored Code in karaoke-gen
|
||||
|
||||
5. [ ] Create `libs/` directory
|
||||
6. [ ] Move `lyrics_transcriber_temp/lyrics_transcriber/` to `libs/lyrics_transcriber/`
|
||||
7. [ ] Remove `lyrics_transcriber_temp/` directory (with all its extra files)
|
||||
8. [ ] Update `pyproject.toml` packages and include paths
|
||||
9. [ ] Update the one script that references `lyrics_transcriber_temp` directly:
|
||||
- `scripts/benchmark_encoding.py` line 232: change `from lyrics_transcriber_temp.lyrics_transcriber.output.video` to `from karaoke_gen.lyrics_transcriber.output.video`
|
||||
|
||||
### Phase 3: Verify and Test
|
||||
|
||||
10. [ ] Run `poetry install` to verify package configuration
|
||||
11. [ ] Run `make test` to ensure all tests pass
|
||||
12. [ ] Verify imports work:
|
||||
```python
|
||||
from karaoke_gen.lyrics_transcriber.core.controller import LyricsControllerResult
|
||||
from karaoke_gen.lyrics_transcriber.output.generator import OutputGenerator
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
```
|
||||
|
||||
### Phase 4: Handle Tests from lyrics_transcriber
|
||||
|
||||
13. [ ] Evaluate lyrics_transcriber tests:
|
||||
- **74 test files** in `lyrics_transcriber_temp/tests/`
|
||||
- These test the internal lyrics_transcriber library functionality
|
||||
- Decision: Move critical tests to `tests/unit/lyrics_transcriber/` in karaoke-gen
|
||||
- Skip integration/performance/manual tests that were for standalone development
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `libs/lyrics_transcriber/` | Create (move) | New location for lyrics_transcriber package |
|
||||
| `lyrics_transcriber_temp/` | Delete | Remove old vendored location |
|
||||
| `pyproject.toml` | Modify | Update package paths |
|
||||
| `scripts/benchmark_encoding.py` | Modify | Update import path |
|
||||
| `tests/unit/lyrics_transcriber/` | Create | Migrate critical tests (optional) |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Immediate Testing
|
||||
- Run `make test` after reorganization - all existing tests must pass
|
||||
- Verify Poetry can build/install the package correctly
|
||||
|
||||
### Test Migration Decision
|
||||
The vendored `lyrics_transcriber_temp/tests/` contains 74 test files covering:
|
||||
- `unit/correction/` - Correction logic tests (important)
|
||||
- `unit/lyrics/` - Lyrics provider tests
|
||||
- `unit/review/` - Review server tests (less relevant - frontend consolidated)
|
||||
- `integration/` - Integration tests
|
||||
- `performance/` - Performance benchmarks
|
||||
|
||||
**Recommendation:** Migrate a subset of critical unit tests, particularly:
|
||||
- `test_corrector.py` - Core correction logic
|
||||
- `test_anchor_sequence*.py` - Anchor sequence matching
|
||||
- Handler tests in `handlers/`
|
||||
|
||||
This keeps karaoke-gen's test suite focused while preserving coverage of critical functionality.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should lyrics_transcriber tests be migrated? **Recommendation: Migrate critical unit tests only**
|
||||
- [x] What's the best path for vendored code? **Answer: `libs/lyrics_transcriber/`**
|
||||
- [ ] Should we publish a final deprecation release to PyPI before archiving? **Recommended: Yes, with deprecation notice**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues arise:
|
||||
1. Git revert the consolidation commits
|
||||
2. `lyrics_transcriber_temp/` can be restored from git history
|
||||
3. The external python-lyrics-transcriber repo can be unarchived if needed
|
||||
|
||||
## Notes
|
||||
|
||||
- The import paths like `from karaoke_gen.lyrics_transcriber.core import ...` will continue to work unchanged
|
||||
- Only the internal project structure changes, not the API
|
||||
- This is primarily a cleanup/organization task with some external repo admin work
|
||||
@@ -1,110 +0,0 @@
|
||||
# Plan: Disable Auto-Correction (Agentic and Heuristic)
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1914-disable-auto-correction
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Disable all auto-correction during the lyrics transcription & correction phase. Currently, auto-correction (both agentic AI-based and heuristic rule-based) creates more work for human reviewers by introducing errors that need to be manually fixed. The solution is to skip all correction processing and pass the raw transcription directly to human review.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Disable agentic AI correction (LLM-based via Gemini)
|
||||
- [x] Disable heuristic correction (rule-based handlers like ExtendAnchorHandler, WordCountMatchHandler, etc.)
|
||||
- [x] Preserve the transcription → human review → video generation flow
|
||||
- [x] Minimal code changes (configuration-based if possible)
|
||||
- [x] Easy to re-enable in the future if correction quality improves
|
||||
|
||||
## Technical Approach
|
||||
|
||||
The simplest approach is to change the backend configuration default for `use_agentic_ai` from `true` to `false`, AND ensure that when agentic AI is disabled, the heuristic handlers also don't run.
|
||||
|
||||
**Current behavior:**
|
||||
1. `backend/config.py` line 46: `use_agentic_ai` defaults to `true`
|
||||
2. `lyrics_worker.py` sets `USE_AGENTIC_AI` env var based on config
|
||||
3. `corrector.py` line 84/132: Checks `USE_AGENTIC_AI` env var
|
||||
4. When agentic is disabled, rule-based handlers still run (lines 669-733)
|
||||
|
||||
**Proposed changes:**
|
||||
|
||||
### Option A: Add a new `skip_correction` config (Recommended)
|
||||
Add a single boolean `skip_correction` setting that:
|
||||
- When `true`: Skips the entire correction process, returning raw transcription
|
||||
- When `false` (future): Falls back to existing agentic/heuristic behavior
|
||||
|
||||
This is cleaner than disabling both systems separately because:
|
||||
1. Single config controls both
|
||||
2. Clear intent ("skip correction" vs "disable two different systems")
|
||||
3. Easy to understand for future maintainers
|
||||
|
||||
### Option B: Just disable both individually
|
||||
- Set `use_agentic_ai` default to `false`
|
||||
- Disable default heuristic handlers by setting `enabled_handlers=[]`
|
||||
|
||||
Option A is cleaner, but Option B requires fewer code changes.
|
||||
|
||||
**Decision: Option A** - Add `skip_correction` config for clarity and easier future re-enablement.
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Add `skip_correction` config to backend/config.py**
|
||||
- Add `skip_correction: bool` setting, default `true`
|
||||
- Comment explaining this disables both agentic and heuristic correction
|
||||
|
||||
2. [x] **Update lyrics_worker.py to pass skip_correction**
|
||||
- Pass the config value through to LyricsProcessor/LyricsTranscriber
|
||||
|
||||
3. [x] **Update corrector.py to support skip_correction**
|
||||
- When `skip_correction=True`, return immediately with uncorrected segments
|
||||
- No handler processing, no agentic AI
|
||||
- Still need to do anchor/gap finding? No - skip entirely
|
||||
|
||||
4. [x] **Update LyricsProcessor to support skip_correction**
|
||||
- Pass through to LyricsTranscriber
|
||||
|
||||
5. [x] **Update LyricsTranscriber controller to support skip_correction**
|
||||
- Pass through to corrector or skip corrector.run() entirely
|
||||
|
||||
6. [x] **Update docs/README.md**
|
||||
- Note that auto-correction is disabled
|
||||
- Update "Agentic AI correction" status to "Disabled"
|
||||
|
||||
7. [x] **Run tests to verify no regressions**
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/config.py` | Modify | Add `skip_correction` config (default: true) |
|
||||
| `backend/workers/lyrics_worker.py` | Modify | Pass `skip_correction` to transcription |
|
||||
| `karaoke_gen/lyrics_transcriber/correction/corrector.py` | Modify | Return early when skip_correction=True |
|
||||
| `karaoke_gen/lyrics_transcriber/controller.py` | Modify | Pass skip_correction to corrector |
|
||||
| `karaoke_gen/lyrics_processor.py` | Modify | Pass skip_correction through |
|
||||
| `docs/README.md` | Modify | Update status to reflect disabled correction |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Unit tests**: No new tests needed - this is a config change
|
||||
- **Existing tests**: Run `make test` to ensure no regressions
|
||||
- **Manual verification**: The correction metadata in the corrections.json should show no corrections made
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the approach is straightforward.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
To re-enable auto-correction:
|
||||
1. Set `SKIP_CORRECTION=false` environment variable, OR
|
||||
2. Change the default back to `false` in `backend/config.py`
|
||||
|
||||
The code paths for agentic and heuristic correction remain intact and can be re-enabled at any time.
|
||||
|
||||
## Alternative Considered
|
||||
|
||||
Instead of adding a new config, we could:
|
||||
1. Set `USE_AGENTIC_AI=false` (disables agentic)
|
||||
2. Set `enabled_handlers=[]` somewhere (disables heuristic)
|
||||
|
||||
But this is more fragmented and harder to understand. A single `skip_correction` config is clearer.
|
||||
@@ -1,99 +0,0 @@
|
||||
# Plan: Encoding Worker Immutable Deployment Pattern
|
||||
|
||||
## Status: In Progress
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The GCE encoding worker has recurring version mismatch issues because:
|
||||
1. `startup.sh` is baked into the Packer image (requires image rebuild + VM recreation)
|
||||
2. Wheel selection uses `sort -V` which can fail if image is outdated
|
||||
3. CI only restarts service - doesn't verify version strictly
|
||||
4. Multiple actors (CI, Pulumi, manual SSH) can cause drift
|
||||
5. 51 old wheel files accumulate in GCS bucket
|
||||
|
||||
## Solution: Immutable Deployment Pattern
|
||||
|
||||
### Core Principles
|
||||
1. **Single Source of Truth**: All code and config from GCS, nothing baked in image
|
||||
2. **Fixed Wheel Path**: `karaoke_gen-current.whl` eliminates sorting
|
||||
3. **Version Manifest**: CI writes expected version, service MUST match
|
||||
4. **Strict Verification**: CI FAILS if deployed version doesn't match
|
||||
5. **Self-Updating Startup**: Download startup.sh from GCS on every boot
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: GCS Structure & Scripts
|
||||
|
||||
- [x] **Step 1.1**: Create `infrastructure/encoding-worker/` directory
|
||||
- [x] **Step 1.2**: Create `bootstrap.sh` (minimal, baked in image)
|
||||
- [x] **Step 1.3**: Create `startup.sh` (CI-managed, deployed to GCS)
|
||||
- [x] **Step 1.4**: Create README.md with documentation
|
||||
|
||||
### Phase 2: CI Workflow Updates
|
||||
|
||||
- [x] **Step 2.1**: Update CI to upload wheel to fixed path
|
||||
- [x] **Step 2.2**: Update CI to upload startup.sh to GCS
|
||||
- [x] **Step 2.3**: Update CI to write version.txt manifest
|
||||
- [x] **Step 2.4**: Update CI to strictly verify deployed version (fail on mismatch)
|
||||
|
||||
### Phase 3: Packer Image Updates
|
||||
|
||||
- [x] **Step 3.1**: Update provision.sh to use bootstrap.sh
|
||||
- [ ] **Step 3.2**: Rebuild Packer image (manual step after merge)
|
||||
- [ ] **Step 3.3**: Recreate VM to use new image (manual step after merge)
|
||||
|
||||
### Phase 4: Documentation & Cleanup
|
||||
|
||||
- [x] **Step 4.1**: Update LESSONS-LEARNED.md with new pattern
|
||||
- [x] **Step 4.2**: Update infrastructure/packer/README.md
|
||||
- [ ] **Step 4.3**: Add lifecycle rule for old wheels (optional, later)
|
||||
|
||||
## New GCS Structure
|
||||
|
||||
```
|
||||
gs://karaoke-gen-storage-nomadkaraoke/
|
||||
├── wheels/
|
||||
│ ├── karaoke_gen-current.whl # Latest (CI overwrites)
|
||||
│ └── karaoke_gen-{version}.whl # Versioned for rollback
|
||||
│
|
||||
└── encoding-worker/
|
||||
├── bootstrap.sh # Reference copy
|
||||
├── startup.sh # CI-managed, downloaded on boot
|
||||
└── version.txt # Expected version number
|
||||
```
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Purpose |
|
||||
|------|--------|---------|
|
||||
| `infrastructure/encoding-worker/bootstrap.sh` | Create | Minimal loader (baked in image) |
|
||||
| `infrastructure/encoding-worker/startup.sh` | Create | Main logic (deployed to GCS) |
|
||||
| `infrastructure/packer/scripts/provision.sh` | Modify | Use bootstrap.sh |
|
||||
| `.github/workflows/ci.yml` | Modify | New deployment flow |
|
||||
| `docs/LESSONS-LEARNED.md` | Modify | Document pattern |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. **Unit test**: Verify startup.sh logic locally
|
||||
2. **Integration test**: After merge, verify:
|
||||
- CI uploads files to correct GCS paths
|
||||
- Service starts with correct version
|
||||
- Health endpoint returns expected version
|
||||
3. **Manual verification**: Check VM logs after first deployment
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
If issues occur:
|
||||
1. CI can re-upload previous wheel to `karaoke_gen-current.whl`
|
||||
2. Previous versioned wheels remain in GCS for manual rollback
|
||||
3. Old Packer images remain for VM rollback
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - design is complete.
|
||||
|
||||
## Notes
|
||||
|
||||
- Bootstrap.sh is ~10 lines and should never need changes
|
||||
- All logic changes go through startup.sh in GCS
|
||||
- Version verification is strict - CI fails on mismatch
|
||||
@@ -1,166 +0,0 @@
|
||||
# Plan: Fix YouTube Remote Download Missing GCS Path
|
||||
|
||||
**Created:** 2026-01-20
|
||||
**Branch:** feat/sess-20260120-1756-fix-youtube-remote-download
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
When remote flacfetch is enabled and a user selects a YouTube audio source, the backend calls `download_by_id()` without the `gcs_path` parameter. This causes flacfetch to download the file locally on the VM instead of uploading to GCS. The backend then tries to open the VM's local path, which doesn't exist on Cloud Run, causing the job to fail.
|
||||
|
||||
This is the same class of bug that was fixed on 2025-12-22 for RED/OPS torrent sources (see `docs/archive/2025-12-phase1-testing-gaps.md`), but the fix only covered torrent sources, not YouTube.
|
||||
|
||||
## Root Cause
|
||||
|
||||
In `backend/api/routes/audio_search.py` lines 354-363:
|
||||
|
||||
```python
|
||||
# Use download_by_id if we have source_id and remote is enabled
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
logger.info(f"Using download_by_id for local download: {source_name} ID={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=temp_dir, # <-- This is a local temp dir, but flacfetch runs on VM!
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
# <-- MISSING: gcs_path parameter
|
||||
)
|
||||
```
|
||||
|
||||
When `is_remote_enabled=True`, `download_by_id()` routes to the remote flacfetch VM. Without `gcs_path`, the VM downloads locally and returns a local path like `/var/lib/transmission-daemon/downloads/...` which doesn't exist on Cloud Run.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] YouTube downloads with remote flacfetch should upload directly to GCS
|
||||
- [x] Backend should receive a `gs://` path from flacfetch
|
||||
- [x] No changes to flacfetch itself (it already supports `upload_to_gcs` and `gcs_destination`)
|
||||
- [x] Maintain backwards compatibility (local-only deployments should still work)
|
||||
- [x] Add test coverage for this code path
|
||||
|
||||
## Technical Approach
|
||||
|
||||
**Option A (chosen):** Pass `gcs_path` to `download_by_id()` for ALL remote downloads, not just torrent sources.
|
||||
|
||||
The current code structure has two branches:
|
||||
1. `if is_torrent_source and is_remote_enabled:` - handles RED/OPS with GCS upload ✅
|
||||
2. `else:` branch - handles YouTube, but incorrectly assumes "local download" even when remote is enabled
|
||||
|
||||
The fix refactors to check `is_remote_enabled` first, then handle torrent vs YouTube within that:
|
||||
- If remote enabled: always pass `gcs_path` to let flacfetch upload directly
|
||||
- If remote disabled: download locally and upload to GCS from Cloud Run
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] Refactor the `else` branch in `_download_and_start_processing()` to check for remote-enabled YouTube downloads
|
||||
2. [x] Pass `gcs_path` when calling `download_by_id()` for remote YouTube downloads
|
||||
3. [x] Add test case for YouTube with remote enabled to verify `gcs_path` is passed
|
||||
4. [x] Update logging to clarify the download path being used
|
||||
5. [x] Bump version in `pyproject.toml`
|
||||
6. [x] Run tests to verify no regressions
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/api/routes/audio_search.py` | Modify | Add `gcs_path` parameter for remote YouTube downloads |
|
||||
| `backend/tests/test_audio_search.py` | Modify | Add test case for YouTube remote download with GCS path |
|
||||
| `pyproject.toml` | Modify | Bump version |
|
||||
|
||||
## Detailed Code Changes
|
||||
|
||||
### audio_search.py (lines 349-393)
|
||||
|
||||
Current structure:
|
||||
```python
|
||||
else:
|
||||
# Local download (YouTube or fallback) <-- WRONG: not always local!
|
||||
temp_dir = tempfile.mkdtemp(...)
|
||||
|
||||
if source_id and source_name and is_remote_enabled:
|
||||
result = audio_search_service.download_by_id(
|
||||
...
|
||||
output_dir=temp_dir, # <-- BUG: unused for remote
|
||||
# Missing gcs_path!
|
||||
)
|
||||
```
|
||||
|
||||
Fixed structure:
|
||||
```python
|
||||
else:
|
||||
# Non-torrent download (YouTube or fallback)
|
||||
|
||||
if is_remote_enabled and source_id and source_name:
|
||||
# Remote download via flacfetch VM - upload directly to GCS
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Using remote download_by_id for {source_name} ID={source_id} with GCS upload to: {gcs_destination}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="", # Not used for remote
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination, # <-- FIX: Add this!
|
||||
)
|
||||
|
||||
# Handle GCS path response (same logic as torrent branch)
|
||||
if result.filepath.startswith("gs://"):
|
||||
parts = result.filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
audio_gcs_path = parts[1]
|
||||
else:
|
||||
audio_gcs_path = result.filepath
|
||||
filename = os.path.basename(result.filepath)
|
||||
else:
|
||||
# Fallback: upload manually
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
logger.warning(f"Remote download returned local path: {result.filepath}, uploading manually")
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(f, audio_gcs_path, content_type='audio/flac')
|
||||
|
||||
logger.info(f"Remote download complete, GCS path: {audio_gcs_path}")
|
||||
else:
|
||||
# Local download (remote disabled or no source_id)
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
# ... existing local download logic ...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
Add to `test_audio_search.py` in `TestAudioSearchApiRouteDownload`:
|
||||
|
||||
```python
|
||||
def test_youtube_remote_download_passes_gcs_path(self, mock_job_manager, mock_storage_service):
|
||||
"""
|
||||
Test that YouTube downloads via remote flacfetch include GCS path.
|
||||
|
||||
This is the same bug pattern as the torrent download bug fixed in 2025-12:
|
||||
when remote is enabled, we must pass gcs_path so flacfetch uploads to GCS.
|
||||
"""
|
||||
# Setup: YouTube source with remote enabled
|
||||
# Verify: download_by_id called with gcs_path parameter
|
||||
```
|
||||
|
||||
### Manual Testing (post-deploy)
|
||||
1. Create a job and select a YouTube audio source
|
||||
2. Verify job completes successfully
|
||||
3. Check GCS path is properly set in job data
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - the fix is straightforward and follows the pattern established for torrent sources.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
This is a minor code change. If issues occur:
|
||||
1. Revert the PR
|
||||
2. The failure mode returns to the current state (YouTube downloads fail when remote is enabled)
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/archive/2025-12-phase1-testing-gaps.md` - Original bug fix for torrent sources
|
||||
- `docs/LESSONS-LEARNED.md` - "Fix Both Sides of Dual Code Paths" pattern
|
||||
@@ -1,218 +0,0 @@
|
||||
# Plan: Job Email Notifications & Admin Message Tools
|
||||
|
||||
**Created:** 2026-01-06
|
||||
**Branch:** feat/sess-20260106-0116-add-email-button
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add comprehensive email notification system for karaoke-gen that:
|
||||
1. **Auto-sends emails** when jobs complete or need user input (after 5-minute idle threshold)
|
||||
2. **Uses cloud-stored templates** for the job completion message
|
||||
3. **Provides admin tools** to copy/send completion messages manually (for Fiverr, etc.)
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Auto-send job completion email to user (CC: gen@nomadkaraoke.com)
|
||||
- [ ] Auto-send "action needed" reminder emails when user is idle for 5+ minutes at blocking states (lyrics review, instrumental selection)
|
||||
- [ ] Store email templates in GCS (editable without code deploy)
|
||||
- [ ] Admin button: "Copy Message" - copies templated completion message to clipboard
|
||||
- [ ] Admin button: "Send Email" - prompts for email address, then sends completion message
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Email sending must not block job completion (async/background)
|
||||
- [ ] Idle detection via Cloud Tasks scheduled task (not polling)
|
||||
- [ ] Template changes should take effect without restart (no long-term caching)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### 1. Template Storage (GCS)
|
||||
Store templates in `gs://karaoke-gen-storage/templates/`:
|
||||
- `job-completion.txt` - Plain text template for job completion
|
||||
- `job-completion.html` - HTML version (optional, for rich emails)
|
||||
- `action-needed-lyrics.txt` - Reminder for lyrics review
|
||||
- `action-needed-instrumental.txt` - Reminder for instrumental selection
|
||||
|
||||
**Template Variables:**
|
||||
- `{name}` - User's display name or "there" if unknown
|
||||
- `{youtube_url}` - YouTube video URL
|
||||
- `{dropbox_url}` - Dropbox folder URL
|
||||
- `{artist}` - Artist name
|
||||
- `{title}` - Song title
|
||||
- `{job_id}` - Job ID
|
||||
- `{review_url}` - Lyrics review URL (for action-needed emails)
|
||||
- `{instrumental_url}` - Instrumental selection URL
|
||||
- `{feedback_url}` - Feedback form URL (for completion emails)
|
||||
|
||||
### 2. Email Service Extensions
|
||||
|
||||
Extend `backend/services/email_service.py`:
|
||||
- Add `send_job_completion()` method with CC support
|
||||
- Add `send_action_reminder()` method
|
||||
- Add `render_template()` helper to fetch & render GCS templates
|
||||
|
||||
### 3. Auto-Email Triggers
|
||||
|
||||
**Job Completion:**
|
||||
- Trigger point: When job enters `COMPLETE` status in `job_manager.transition_to_state()`
|
||||
- Implementation: Call email service async in background task
|
||||
|
||||
**Action Reminders (5-min idle detection):**
|
||||
- When job enters `AWAITING_REVIEW` or `AWAITING_INSTRUMENTAL_SELECTION`:
|
||||
- Record `blocking_state_entered_at` timestamp in `state_data`
|
||||
- Schedule a Cloud Tasks task for 5 minutes later
|
||||
- The scheduled task checks if job is still in blocking state and user hasn't started review
|
||||
- If idle, send reminder email and set `reminder_sent` flag to prevent duplicates
|
||||
- **Only one reminder per job** - if `reminder_sent` is true, skip sending
|
||||
|
||||
### 4. Admin UI (Frontend)
|
||||
|
||||
Add admin-only buttons to completed jobs in `OutputLinks.tsx`:
|
||||
- "Copy Message" button - fetches rendered template, copies to clipboard
|
||||
- "Send Email" button - opens dialog for email input, calls API to send
|
||||
|
||||
### 5. Backend API Endpoints
|
||||
|
||||
New endpoints in `backend/api/routes/admin.py`:
|
||||
- `GET /api/admin/jobs/{job_id}/completion-message` - Returns rendered completion message
|
||||
- `POST /api/admin/jobs/{job_id}/send-completion-email` - Sends email to specified address
|
||||
|
||||
New internal endpoint for scheduled reminder:
|
||||
- `POST /api/internal/jobs/{job_id}/check-idle-reminder` - Called by Cloud Tasks
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Template Infrastructure
|
||||
1. [ ] Upload initial template to GCS from user's existing template file
|
||||
2. [ ] Create `template_service.py` to fetch/render templates from GCS
|
||||
3. [ ] Add unit tests for template rendering
|
||||
|
||||
### Phase 2: Email Extensions
|
||||
4. [ ] Extend `email_service.py` with CC support in `send_email()`
|
||||
5. [ ] Add `send_job_completion()` method
|
||||
6. [ ] Add `send_action_reminder()` method
|
||||
7. [ ] Add unit tests for new email methods
|
||||
|
||||
### Phase 3: Auto-Email on Completion
|
||||
8. [ ] Modify `job_manager.py` to trigger completion email when entering COMPLETE
|
||||
9. [ ] Add integration test for completion email trigger
|
||||
|
||||
### Phase 4: Idle Reminder System
|
||||
10. [ ] Add `blocking_state_entered_at` field tracking to job state transitions
|
||||
11. [ ] Create Cloud Tasks queue for idle reminders (`idle-reminder-queue`)
|
||||
12. [ ] Schedule 5-minute delayed task when entering blocking states
|
||||
13. [ ] Implement `/api/internal/jobs/{job_id}/check-idle-reminder` endpoint
|
||||
14. [ ] Add tests for idle reminder flow
|
||||
|
||||
### Phase 5: Admin API
|
||||
15. [ ] Add `GET /api/admin/jobs/{job_id}/completion-message` endpoint
|
||||
16. [ ] Add `POST /api/admin/jobs/{job_id}/send-completion-email` endpoint
|
||||
17. [ ] Add tests for admin endpoints
|
||||
|
||||
### Phase 6: Admin UI
|
||||
18. [ ] Add `isAdmin` check to `OutputLinks.tsx` component
|
||||
19. [ ] Add "Copy Message" button (admin-only) with loading state
|
||||
20. [ ] Add "Send Email" button (admin-only) with email input dialog
|
||||
21. [ ] Add E2E test for admin buttons
|
||||
|
||||
### Phase 7: Documentation
|
||||
22. [ ] Document template management in docs/DEVELOPMENT.md
|
||||
23. [ ] Update docs/API.md with new endpoints
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/services/template_service.py` | Create | GCS template fetching and rendering |
|
||||
| `backend/services/email_service.py` | Modify | Add CC support, completion/reminder methods |
|
||||
| `backend/services/job_manager.py` | Modify | Trigger emails on state transitions |
|
||||
| `backend/services/worker_service.py` | Modify | Add idle reminder task scheduling |
|
||||
| `backend/api/routes/admin.py` | Modify | Add completion message/email endpoints |
|
||||
| `backend/api/routes/internal.py` | Modify | Add idle reminder check endpoint |
|
||||
| `backend/models/job.py` | Modify | Add `blocking_state_entered_at`, `reminder_sent_at` to state_data docs |
|
||||
| `infrastructure/__main__.py` | Modify | Add idle-reminder-queue Cloud Tasks queue |
|
||||
| `frontend/components/job/OutputLinks.tsx` | Modify | Add admin-only Copy/Send buttons |
|
||||
| `frontend/lib/api.ts` | Modify | Add API methods for admin endpoints |
|
||||
| `tests/unit/test_template_service.py` | Create | Template rendering tests |
|
||||
| `tests/unit/test_email_service_completion.py` | Create | Completion email tests |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
**Unit Tests:**
|
||||
- Template rendering with all variable combinations
|
||||
- Email service CC support
|
||||
- Completion/reminder email generation
|
||||
- Idle detection logic
|
||||
|
||||
**Integration Tests:**
|
||||
- Full flow: job completion → email sent
|
||||
- Idle reminder: job enters blocking state → 5 min → reminder sent
|
||||
- Admin endpoint authentication
|
||||
|
||||
**Manual Testing:**
|
||||
- Verify emails render correctly in Gmail/Outlook
|
||||
- Test admin buttons on completed job
|
||||
- Test clipboard copy works across browsers
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Where to store templates? → **GCS bucket: `gs://karaoke-gen-storage/templates/`**
|
||||
- [x] Should we rate-limit reminder emails? → **Max 1 reminder per job** (send once after 5 min idle, then stop)
|
||||
- [x] Should completion emails also include a feedback request link? → **Yes, for all users**
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. **Email sending failures** - All email sends are async/fire-and-forget; job completion unaffected
|
||||
2. **Template fetch failures** - Fall back to hardcoded template in code
|
||||
3. **Idle reminder failures** - Jobs continue normally; users can still access review UI manually
|
||||
4. **Feature flag** - Add `ENABLE_AUTO_EMAILS=true/false` env var for quick disable
|
||||
|
||||
## Template Content
|
||||
|
||||
Initial template (from user's file, with feedback link added):
|
||||
|
||||
```
|
||||
Hi {name},
|
||||
|
||||
Thanks for your order!
|
||||
|
||||
Here's the link for the karaoke video published to YouTube:
|
||||
{youtube_url}
|
||||
|
||||
Here's the dropbox folder with all the finished files and source files, including:
|
||||
- "(Final Karaoke Lossless).mkv": combined karaoke video in 4k H264 with lossless FLAC audio
|
||||
- "(Final Karaoke).mp4": combined karaoke video with title/end screen in 4k H264/AAC
|
||||
- "(Final Karaoke 720p).mp4": combined karaoke video in 720p H264/AAC (smaller file for older systems)
|
||||
- "(With Vocals).mp4": sing along video in 4k H264/AAC with original vocals
|
||||
- "(Karaoke).mov": karaoke video output from MidiCo (no title/end screen)
|
||||
- "(Title).mov"/"(End).mov": title card and end screen videos
|
||||
- "(Final Karaoke CDG).zip": CDG+MP3 format for older/commercial karaoke systems
|
||||
- "(Final Karaoke TXT).zip": TXT+MP3 format for Power Karaoke
|
||||
- stems/*.flac: various separated instrumental and vocal audio stems in lossless format
|
||||
- lyrics/*.txt song lyrics from various sources in plain text format
|
||||
|
||||
{dropbox_url}
|
||||
|
||||
Let me know if anything isn't perfect and I'll happily tweak / fix, or if you need it in any other format I can probably convert it for you!
|
||||
|
||||
If you have a moment, I'd really appreciate your feedback (takes 2 minutes):
|
||||
{feedback_url}
|
||||
|
||||
Thanks again and have a great day!
|
||||
-Andrew
|
||||
```
|
||||
|
||||
**Note:** The `{feedback_url}` placeholder will be populated with the feedback form URL. If no feedback URL is configured, this section can be omitted from the rendered output.
|
||||
|
||||
## Updating Templates
|
||||
|
||||
To update email templates after deployment:
|
||||
|
||||
```bash
|
||||
# Upload new template
|
||||
gsutil cp my-template.txt gs://karaoke-gen-storage/templates/job-completion.txt
|
||||
|
||||
# Templates are fetched fresh on each use (no caching in code)
|
||||
# Changes take effect immediately
|
||||
```
|
||||
@@ -1,106 +0,0 @@
|
||||
# Plan: Migrate from MailSlurp to testmail.app
|
||||
|
||||
**Created:** 2026-01-02
|
||||
**Branch:** feat/sess-20260102-2109-migrate-mailslurp-testmail
|
||||
**Status:** Complete
|
||||
|
||||
## Overview
|
||||
|
||||
Replace MailSlurp email testing service with testmail.app for E2E testing of magic link authentication. testmail.app uses a simpler API with namespace-based email addresses that require no inbox creation/deletion - emails just arrive at `{namespace}.{tag}@inbox.testmail.app`.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Remove all code referencing MailSlurp
|
||||
- [x] Remove `mailslurp-client` npm dependency
|
||||
- [x] Implement testmail.app API integration
|
||||
- [x] Update environment variable names (`TESTMAIL_API_KEY`, `TESTMAIL_NAMESPACE`)
|
||||
- [x] Maintain same `EmailHelper` interface for minimal test changes
|
||||
- [x] Update archive docs to note migration (optional - historical accuracy)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### API Differences
|
||||
|
||||
| Feature | MailSlurp | testmail.app |
|
||||
|---------|-----------|--------------|
|
||||
| Email format | Random `xxx@mailslurp.com` | `{namespace}.{tag}@inbox.testmail.app` |
|
||||
| Inbox creation | Required (`createInbox()`) | Not needed - use any tag |
|
||||
| Inbox deletion | Required (`deleteInbox()`) | Not needed |
|
||||
| Get emails | `waitForLatestEmail(inboxId)` | GET `/api/json?apikey=X&namespace=Y&tag=Z&livequery=true` |
|
||||
| Auth | API key in constructor | API key as query param |
|
||||
| Email body | `email.body` | `email.html` or `email.text` |
|
||||
|
||||
### Implementation Strategy
|
||||
|
||||
1. **Keep same `EmailHelper` interface** - `createInbox()`, `waitForEmail()`, `extractMagicLink()`, `deleteInbox()`
|
||||
2. **Use testmail.app's live query feature** - `livequery=true` waits for email to arrive (up to timeout)
|
||||
3. **Generate unique tags** - Use timestamp + random suffix for each test run to avoid collisions
|
||||
4. **Inbox becomes a simple object** - `{ id: tag, emailAddress: '{namespace}.{tag}@inbox.testmail.app' }`
|
||||
5. **Delete becomes no-op** - testmail.app emails auto-expire, no cleanup needed
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Old | New |
|
||||
|-----|-----|
|
||||
| `MAILSLURP_API_KEY` | `TESTMAIL_API_KEY` |
|
||||
| (none) | `TESTMAIL_NAMESPACE` |
|
||||
|
||||
Values: Set via environment variables (see `frontend/.env.local.example`)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. [x] **Update `email-testing.ts`** - Replace MailSlurp implementation with testmail.app
|
||||
- Remove `mailslurp-client` import
|
||||
- Add testmail.app API fetch implementation
|
||||
- Use native `fetch` (no new dependencies)
|
||||
- Generate unique tag for each "inbox"
|
||||
- Implement `livequery=true` for waiting
|
||||
- Adapt email body extraction (`html`/`text` instead of `body`)
|
||||
|
||||
2. [x] **Update `full-user-journey.spec.ts`** - Change references
|
||||
- Update doc comments (env var name)
|
||||
- Update error messages
|
||||
- Update console logs
|
||||
|
||||
3. [x] **Update `playwright.record.config.ts`** - Change env var comment
|
||||
|
||||
4. [x] **Update `package.json`** - Remove `mailslurp-client` dependency
|
||||
|
||||
5. [x] **Run `npm install`** - Update package-lock.json
|
||||
|
||||
6. [x] **Create `.env.local.example`** - Document required env vars for local testing
|
||||
|
||||
7. [x] **Update archive docs** - Skipped intentionally; historical docs preserved for accuracy
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `frontend/e2e/helpers/email-testing.ts` | Modify | Replace MailSlurp with testmail.app |
|
||||
| `frontend/e2e/production/full-user-journey.spec.ts` | Modify | Update env var refs and messages |
|
||||
| `frontend/playwright.record.config.ts` | Modify | Update env var comment |
|
||||
| `frontend/package.json` | Modify | Remove `mailslurp-client` |
|
||||
| `frontend/package-lock.json` | Modify | Auto-updated by npm install |
|
||||
| `frontend/.env.local.example` | Create | Document env vars for local testing |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- **Manual testing**: Run production E2E test with testmail.app credentials
|
||||
```bash
|
||||
cd frontend
|
||||
TESTMAIL_API_KEY=your-api-key \
|
||||
TESTMAIL_NAMESPACE=your-namespace \
|
||||
npm run test:e2e:prod:headed
|
||||
```
|
||||
- **Verify**: Email received, magic link extracted, auth flow completes
|
||||
- **CI testing**: Env vars already configured in GitHub Actions org secrets
|
||||
|
||||
## Open Questions
|
||||
|
||||
None - requirements are clear.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Revert the commit
|
||||
2. Run `npm install` to restore mailslurp-client
|
||||
3. Re-add MAILSLURP_API_KEY to secrets if removed
|
||||
@@ -1,329 +0,0 @@
|
||||
# Plan: Unified Combined Review for Local CLI
|
||||
|
||||
**Created:** 2026-01-24
|
||||
**Branch:** feat/sess-20260124-1444-combined-review-flow
|
||||
**Status:** Draft
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The combined lyrics + instrumental review flow was implemented for the cloud web app, but the local CLI is broken in two ways:
|
||||
|
||||
1. **Bug: Lyrics review is skipped when `SKIP_CORRECTION=true`** (the default)
|
||||
- Review code lives inside `correct_lyrics()` method in controller.py
|
||||
- When auto-correction is disabled, `correct_lyrics()` is never called
|
||||
- Users never get a chance to review lyrics
|
||||
|
||||
2. **Standalone instrumental UI still exists for CLI**
|
||||
- `InstrumentalReviewServer` in `instrumental_review/server.py` is a separate server
|
||||
- CLI launches this standalone server after lyrics review
|
||||
- The combined review flow we built only works for the web app
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Fix the skip bug** - Review should happen even when auto-correction is disabled
|
||||
2. **Unify all review UIs** - One combined review flow for CLI, cloud CLI, and web app
|
||||
3. **Remove dead code** - Delete standalone `InstrumentalReviewServer` and related code
|
||||
4. **Keep it simple** - Minimal changes to achieve the goal
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
### Current State (Broken)
|
||||
|
||||
```
|
||||
CLI Flow:
|
||||
controller.py::process()
|
||||
→ correct_lyrics() [SKIPPED if SKIP_CORRECTION=true]
|
||||
→ ReviewServer starts (lyrics only)
|
||||
→ User reviews
|
||||
→ ReviewServer stops
|
||||
→ gen_cli.py::run_instrumental_review()
|
||||
→ InstrumentalReviewServer starts (instrumental only)
|
||||
→ User selects instrumental
|
||||
→ InstrumentalReviewServer stops
|
||||
|
||||
Cloud Web App Flow:
|
||||
Job enters AWAITING_REVIEW
|
||||
→ User visits /app/jobs/{id}/review
|
||||
→ Combined UI: lyrics + instrumental
|
||||
→ POST /api/jobs/{id}/complete-review (includes instrumental_selection)
|
||||
```
|
||||
|
||||
### Target State
|
||||
|
||||
```
|
||||
CLI Flow (and Cloud CLI):
|
||||
controller.py::process()
|
||||
→ _run_human_review() [NEW - always called when review enabled]
|
||||
→ ReviewServer starts (combined: lyrics + instrumental)
|
||||
→ User reviews lyrics AND selects instrumental
|
||||
→ ReviewServer stops, returns both results
|
||||
→ Continue with selected instrumental
|
||||
|
||||
Cloud Web App Flow: (unchanged)
|
||||
Same as before
|
||||
```
|
||||
|
||||
## Key Insight
|
||||
|
||||
The ReviewServer already serves the bundled Next.js frontend. That frontend already has the combined review UI with `InstrumentalSelectorEmbedded`. We just need to:
|
||||
|
||||
1. Make `ReviewServer` serve instrumental options in its `/correction-data` endpoint
|
||||
2. Make `ReviewServer` accept instrumental selection in its `/complete-review` endpoint
|
||||
3. Call the review flow from the right place (not inside `correct_lyrics()`)
|
||||
4. Delete `InstrumentalReviewServer` entirely
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Fix ReviewServer to Support Instrumental Selection
|
||||
|
||||
**1.1. Add instrumental data to ReviewServer's correction-data endpoint**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The `ReviewServer.__init__()` accepts a `CorrectionResult`. We need to also accept instrumental options:
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None, # NEW
|
||||
backing_vocals_analysis: dict | None = None, # NEW
|
||||
...
|
||||
):
|
||||
```
|
||||
|
||||
Update `_correction_data()` endpoint to include these in response:
|
||||
|
||||
```python
|
||||
@app.get("/api/review/{job_id}/correction-data")
|
||||
def _correction_data(job_id: str):
|
||||
return {
|
||||
"lyrics_data": self.correction_result.to_dict(),
|
||||
"instrumental_options": self.instrumental_options, # NEW
|
||||
"backing_vocals_analysis": self.backing_vocals_analysis # NEW
|
||||
}
|
||||
```
|
||||
|
||||
**1.2. Update ReviewServer's complete-review endpoint to accept instrumental selection**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/review/server.py`
|
||||
|
||||
The existing endpoint accepts `corrected_lyrics`. Add `instrumental_selection`:
|
||||
|
||||
```python
|
||||
@app.post("/api/review/{job_id}/complete-review")
|
||||
def _complete_review(job_id: str, request: ReviewCompleteRequest):
|
||||
# Existing: store corrected lyrics
|
||||
self.corrected_lyrics = request.corrected_lyrics
|
||||
|
||||
# NEW: store instrumental selection
|
||||
self.instrumental_selection = request.instrumental_selection # "clean" | "with_backing"
|
||||
|
||||
self.review_completed = True
|
||||
```
|
||||
|
||||
**1.3. Expose instrumental selection after review completes**
|
||||
|
||||
The calling code needs to get the selection. Update the return value or add an attribute:
|
||||
|
||||
```python
|
||||
# After review loop completes:
|
||||
return {
|
||||
"corrected_lyrics": self.corrected_lyrics,
|
||||
"instrumental_selection": self.instrumental_selection
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 2: Fix controller.py to Call Review at the Right Time
|
||||
|
||||
**2.1. Extract review logic from `correct_lyrics()` into standalone method**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Create new method `_run_human_review()` that can be called independently:
|
||||
|
||||
```python
|
||||
def _run_human_review(
|
||||
self,
|
||||
correction_result: CorrectionResult,
|
||||
instrumental_options: list[dict] | None = None,
|
||||
backing_vocals_analysis: dict | None = None
|
||||
) -> tuple[CorrectionResult, str | None]:
|
||||
"""Run human review server and return corrected result + instrumental selection."""
|
||||
|
||||
if not self.output_config.review_enabled:
|
||||
return correction_result, None
|
||||
|
||||
server = ReviewServer(
|
||||
correction_result=correction_result,
|
||||
instrumental_options=instrumental_options,
|
||||
backing_vocals_analysis=backing_vocals_analysis,
|
||||
...
|
||||
)
|
||||
|
||||
result = server.start() # Blocks until review complete
|
||||
|
||||
return result["corrected_lyrics"], result.get("instrumental_selection")
|
||||
```
|
||||
|
||||
**2.2. Update `process()` to always call review when appropriate**
|
||||
|
||||
File: `karaoke_gen/lyrics_transcriber/core/controller.py`
|
||||
|
||||
Current flow (simplified):
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
if self.output_config.run_correction:
|
||||
self.correct_lyrics() # Review is inside here!
|
||||
# ...
|
||||
```
|
||||
|
||||
New flow:
|
||||
```python
|
||||
def process(self):
|
||||
self.transcribe()
|
||||
self.separate_audio()
|
||||
|
||||
# Build correction result (with or without auto-correction)
|
||||
if self.output_config.run_correction:
|
||||
correction_result = self._run_auto_correction()
|
||||
else:
|
||||
correction_result = self._create_uncorrected_result()
|
||||
|
||||
# Human review is separate from auto-correction
|
||||
if self.output_config.review_enabled:
|
||||
instrumental_options = self._get_instrumental_options()
|
||||
backing_vocals_analysis = self._analyze_backing_vocals()
|
||||
|
||||
corrected, instrumental_selection = self._run_human_review(
|
||||
correction_result,
|
||||
instrumental_options,
|
||||
backing_vocals_analysis
|
||||
)
|
||||
|
||||
self.results.correction_result = corrected
|
||||
self.results.instrumental_selection = instrumental_selection
|
||||
```
|
||||
|
||||
### Phase 3: Update gen_cli.py to Use Combined Review
|
||||
|
||||
**3.1. Remove standalone instrumental review call**
|
||||
|
||||
File: `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
Remove the `run_instrumental_review()` function call from main flow. The instrumental selection now comes from the combined review.
|
||||
|
||||
Before:
|
||||
```python
|
||||
# After lyrics processing
|
||||
if not args.non_interactive:
|
||||
instrumental_selection = run_instrumental_review(...)
|
||||
```
|
||||
|
||||
After:
|
||||
```python
|
||||
# Instrumental selection comes from controller.results.instrumental_selection
|
||||
instrumental_selection = controller.results.instrumental_selection
|
||||
```
|
||||
|
||||
**3.2. Pass instrumental options to controller**
|
||||
|
||||
The controller needs instrumental options to display in review. These come from the separated audio stems:
|
||||
|
||||
```python
|
||||
controller = LyricsTranscriber(
|
||||
# ... existing params ...
|
||||
)
|
||||
|
||||
# Before calling process(), set up instrumental options
|
||||
# These are typically discovered during audio separation
|
||||
```
|
||||
|
||||
### Phase 4: Delete Standalone InstrumentalReviewServer
|
||||
|
||||
**4.1. Delete files**
|
||||
|
||||
- Delete `karaoke_gen/instrumental_review/server.py`
|
||||
- Delete `karaoke_gen/instrumental_review/__init__.py` (if empty after)
|
||||
- Remove `run_instrumental_review()` from `karaoke_gen/utils/gen_cli.py`
|
||||
|
||||
**4.2. Clean up imports**
|
||||
|
||||
Remove any imports of `InstrumentalReviewServer` throughout the codebase.
|
||||
|
||||
### Phase 5: Update Frontend for Local Mode
|
||||
|
||||
**5.1. Verify local mode detection works**
|
||||
|
||||
File: `frontend/lib/api.ts`
|
||||
|
||||
The frontend already has local mode detection based on hostname and port. Verify it works:
|
||||
|
||||
```typescript
|
||||
const isLocalMode = () => {
|
||||
if (typeof window === 'undefined') return false;
|
||||
const hostname = window.location.hostname;
|
||||
const port = parseInt(window.location.port, 10);
|
||||
return hostname === 'localhost' && port >= 8000 && port <= 8770;
|
||||
};
|
||||
```
|
||||
|
||||
**5.2. Verify correction-data endpoint parsing**
|
||||
|
||||
Ensure the frontend correctly parses instrumental_options and backing_vocals_analysis from the local server's response.
|
||||
|
||||
### Phase 6: Testing
|
||||
|
||||
**6.1. Manual testing checklist**
|
||||
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=true` - review should still appear
|
||||
- [ ] Run CLI with `SKIP_CORRECTION=false` - review should appear after correction
|
||||
- [ ] Combined UI shows both lyrics editor and instrumental selector
|
||||
- [ ] Selecting instrumental and completing review works
|
||||
- [ ] No standalone instrumental review page appears
|
||||
- [ ] Cloud web app still works (no regression)
|
||||
|
||||
**6.2. Update unit tests**
|
||||
|
||||
- Remove tests for `InstrumentalReviewServer`
|
||||
- Add tests for `ReviewServer` with instrumental options
|
||||
- Add tests for `controller._run_human_review()`
|
||||
|
||||
## Files to Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `lyrics_transcriber/review/server.py` | Modify | Add instrumental options/selection support |
|
||||
| `lyrics_transcriber/core/controller.py` | Modify | Extract review into standalone method, call at right time |
|
||||
| `utils/gen_cli.py` | Modify | Remove `run_instrumental_review()`, use combined result |
|
||||
| `instrumental_review/server.py` | Delete | Standalone server no longer needed |
|
||||
| `instrumental_review/__init__.py` | Modify/Delete | Clean up imports |
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking cloud flow | Cloud doesn't use ReviewServer, uses backend API - no risk |
|
||||
| Missing instrumental options in local mode | Fail clearly if options not provided |
|
||||
| Review server port conflicts | Keep existing port allocation logic |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Where does backing vocals analysis run for local CLI?**
|
||||
- Currently runs in `render_video_worker.py` for cloud
|
||||
- For local, need to run it after audio separation, before review
|
||||
- May need to call analysis function from controller
|
||||
|
||||
2. **How are instrumental stems exposed to local review?**
|
||||
- Need to serve audio files from ReviewServer
|
||||
- Or use file:// URLs (may have CORS issues)
|
||||
- Existing approach: ReviewServer already serves static files
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. `SKIP_CORRECTION=true` no longer skips human review
|
||||
2. Single combined review UI appears (no separate instrumental step)
|
||||
3. All platforms (CLI, cloud CLI, web) use same review flow
|
||||
4. No dead code remaining for standalone instrumental review
|
||||
@@ -1,401 +0,0 @@
|
||||
# Plan: Video Worker Stage Refactor
|
||||
|
||||
**Created:** 2026-01-03
|
||||
**Branch:** feat/sess-20260103-2250-investigate-youtube-upload
|
||||
**Status:** COMPLETE - Ready for PR
|
||||
**Last Updated:** 2026-01-04
|
||||
|
||||
## Overview
|
||||
|
||||
Refactor `video_worker.py` and `KaraokeFinalise` into separate, composable stages following SOLID principles. This eliminates the GCE vs KaraokeFinalise code path divergence that causes features like YouTube upload to be missing from the GCE encoding path.
|
||||
|
||||
## Three Execution Modes
|
||||
|
||||
Understanding the three ways this system runs is critical to the refactor:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
- Runs entirely on user's machine
|
||||
- KaraokeFinalise.process() orchestrates everything
|
||||
- Writes directly to local Dropbox sync folder (e.g., `~/Dropbox/Karaoke/Tracks-Organized/`)
|
||||
- **Produces full output structure** with stems/, lyrics/, visualizations/, etc.
|
||||
- **This is the source of truth for output format**
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
- User's CLI talks to cloud backend (api.nomadkaraoke.com)
|
||||
- Cloud does ALL processing (audio, lyrics, video encoding)
|
||||
- CLI just monitors job status and downloads final outputs
|
||||
- With Dropbox configured: cloud uploads full structure to user's Dropbox
|
||||
- Without Dropbox: user downloads individual files
|
||||
|
||||
### 3. Web Frontend (`gen.nomadkaraoke.com`)
|
||||
- Same cloud backend as remote CLI
|
||||
- User sees:
|
||||
- YouTube video link
|
||||
- Dropbox folder link
|
||||
- Direct download: final karaoke video (720p)
|
||||
- Direct download: 4K video
|
||||
- Direct download: "with vocals" video
|
||||
- Direct download: CDG zip (if enabled)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Currently, `video_worker.py` has two completely divergent code paths:
|
||||
|
||||
```python
|
||||
if use_gce_encoding:
|
||||
# GCE PATH: Only encoding, missing YouTube/Discord/CDG/TXT
|
||||
result = await _encode_via_gce(...)
|
||||
# result DOES NOT include youtube_url!
|
||||
else:
|
||||
# STANDARD PATH: KaraokeFinalise does everything
|
||||
finalise = KaraokeFinalise(...)
|
||||
result = finalise.process()
|
||||
# result includes youtube_url from upload
|
||||
```
|
||||
|
||||
### What GCE Encoding Broke
|
||||
|
||||
| Feature | KaraokeFinalise Path | GCE Path |
|
||||
|---------|---------------------|----------|
|
||||
| Video encoding | ✅ 4 formats | ✅ 4 formats |
|
||||
| YouTube upload | ✅ | ❌ **BROKEN** |
|
||||
| Discord notification | ✅ | ❌ **BROKEN** |
|
||||
| CDG/TXT packaging | ✅ | ❌ **BROKEN** |
|
||||
| Brand code generation | ✅ | ✅ (via Dropbox service) |
|
||||
| Dropbox upload | ✅ (rclone) | ✅ (native SDK) |
|
||||
| Google Drive upload | ❌ | ✅ (native SDK) |
|
||||
|
||||
### Root Cause
|
||||
|
||||
`KaraokeFinalise.process()` is a monolith that handles:
|
||||
1. CDG/TXT packaging
|
||||
2. Video encoding (the CPU-intensive part)
|
||||
3. YouTube upload
|
||||
4. Discord notifications
|
||||
5. Folder organization
|
||||
|
||||
The GCE encoding was added as a bypass around the entire class, rather than replacing just the encoding step.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] YouTube upload works regardless of encoding backend (GCE or local)
|
||||
- [ ] Discord notifications work regardless of encoding backend
|
||||
- [ ] CDG/TXT packaging works regardless of encoding backend
|
||||
- [ ] Brand code generation works consistently
|
||||
- [ ] Dropbox/Google Drive upload works (already works in both paths)
|
||||
- [ ] All existing CLI functionality continues to work
|
||||
- [ ] Output structure matches local CLI exactly
|
||||
|
||||
### Output Structure Parity
|
||||
|
||||
Cloud backend must produce identical output to local CLI:
|
||||
|
||||
```
|
||||
NOMAD-XXXX - Artist - Title/
|
||||
├── Artist - Title (Title).mov
|
||||
├── Artist - Title (Title).jpg
|
||||
├── Artist - Title (End).mov
|
||||
├── Artist - Title (End).jpg
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4
|
||||
├── Artist - Title (Final Karaoke CDG).zip
|
||||
├── Artist - Title (Final Karaoke TXT).zip
|
||||
├── Artist - Title (Instrumental {model}).flac
|
||||
├── Artist - Title (Instrumental +BV {model}).flac
|
||||
├── Artist - Title (Karaoke).mp4
|
||||
├── Artist - Title (Karaoke).cdg
|
||||
├── Artist - Title (Karaoke).mp3
|
||||
├── Artist - Title (Karaoke).lrc
|
||||
├── Artist - Title (With Vocals).mp4
|
||||
├── stems/
|
||||
│ ├── Artist - Title (Instrumental {model}).flac
|
||||
│ ├── Artist - Title (Vocals {model}).flac
|
||||
│ ├── Artist - Title (Lead Vocals {model}).flac
|
||||
│ ├── Artist - Title (Backing Vocals {model}).flac
|
||||
│ └── ... (other stems)
|
||||
├── lyrics/
|
||||
│ ├── Artist - Title (Karaoke).lrc
|
||||
│ ├── Artist - Title (Karaoke).ass
|
||||
│ ├── Artist - Title (Karaoke).srt
|
||||
│ └── ... (corrections JSONs)
|
||||
├── previews/
|
||||
├── visualizations/
|
||||
└── styles_assets/
|
||||
```
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Single Responsibility: Each stage has one job
|
||||
- [ ] Open/Closed: Easy to add new encoding backends without modifying stages
|
||||
- [ ] Testable: Each stage can be unit tested in isolation
|
||||
- [ ] No duplicate code between paths
|
||||
- [ ] Backwards compatible with existing KaraokeFinalise CLI usage
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture: Stage-Based Pipeline
|
||||
|
||||
```
|
||||
video_worker.py (VideoWorkerOrchestrator)
|
||||
│
|
||||
├── 1. EncodingStage (interface)
|
||||
│ ├── LocalEncodingService (wraps KaraokeFinalise encoding methods)
|
||||
│ └── GCEEncodingService (existing encoding_service.py)
|
||||
│
|
||||
├── 2. PackagingStage
|
||||
│ ├── CDGPackager (extracts from KaraokeFinalise)
|
||||
│ └── TXTPackager (extracts from KaraokeFinalise)
|
||||
│
|
||||
├── 3. DistributionStage
|
||||
│ ├── YouTubeUploadService (extracts from KaraokeFinalise)
|
||||
│ ├── DropboxService (existing)
|
||||
│ ├── GoogleDriveService (existing)
|
||||
│ └── DiscordService (extracts from KaraokeFinalise)
|
||||
│
|
||||
└── 4. OrganizationStage
|
||||
└── BrandCodeService (extracts from KaraokeFinalise + DropboxService)
|
||||
```
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Extract, Don't Rewrite**: Pull methods out of KaraokeFinalise into services, don't rewrite from scratch
|
||||
2. **Keep KaraokeFinalise for CLI**: Local CLI still uses KaraokeFinalise.process() - we extract services that it can also use
|
||||
3. **Interface-Based Encoding**: EncodingStage interface allows swapping GCE/local without changing orchestration
|
||||
4. **Existing Service Pattern**: Follow singleton + factory pattern already used in codebase
|
||||
5. **Local CLI stays sync**: Local karaoke-gen CLI remains synchronous as it is today
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
JobConfig (from Firestore)
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ VideoWorkerOrchestrator │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 1. Setup: Download files, prepare directories ││
|
||||
│ │ - Download title.mov, end.mov, with_vocals.mkv ││
|
||||
│ │ - Download instrumental audio ││
|
||||
│ │ - Prepare temp directory ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 2. Packaging: CDG/TXT (if enabled) - runs BEFORE encoding ││
|
||||
│ │ Reason: CDG depends on LRC file which is available ││
|
||||
│ │ Input: LRC file, instrumental audio ││
|
||||
│ │ Output: CDG.zip, TXT.zip, .cdg, .mp3 ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 3. Encoding: GCE or Local (via interface) ││
|
||||
│ │ Input: title.mov, karaoke.mov, end.mov ││
|
||||
│ │ Output: 4K lossless, 4K lossy, 720p, MKV ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 4. Organization: Brand code, folder structure ││
|
||||
│ │ - Generate brand code (NOMAD-XXXX) ││
|
||||
│ │ - Prepare full output directory structure ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 5. Distribution: Upload to destinations ││
|
||||
│ │ - YouTube upload (if enabled) ││
|
||||
│ │ - Dropbox upload (if configured) ││
|
||||
│ │ - Google Drive upload (if configured) ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────┐│
|
||||
│ │ 6. Notification: Discord, Email ││
|
||||
│ │ Input: YouTube URL, brand code, Dropbox link ││
|
||||
│ └─────────────────────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Result (youtube_url, brand_code, dropbox_link, file_paths, etc.)
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Extract Services from KaraokeFinalise (Non-Breaking) ✅ COMPLETE
|
||||
|
||||
These services can be used by both the cloud orchestrator AND KaraokeFinalise.
|
||||
|
||||
1. [x] **Create `backend/services/youtube_upload_service.py`** ✅
|
||||
- Extract YouTube upload logic from KaraokeFinalise
|
||||
- Methods: `upload_video()`, `check_duplicate()`, `delete_video()`, `truncate_title()`
|
||||
- Supports pre-loaded credentials (server-side) or client secrets file (CLI)
|
||||
- 23 unit tests with mocked YouTube API
|
||||
|
||||
2. [x] **Create `backend/services/discord_service.py`** ✅
|
||||
- Extract Discord notification from KaraokeFinalise
|
||||
- Methods: `post_message()`, `post_video_notification()`, `is_enabled()`
|
||||
- Simple HTTP POST to webhook with validation
|
||||
- 20 unit tests with mocked requests
|
||||
|
||||
3. [x] **Create `backend/services/packaging_service.py`** ✅
|
||||
- Extract CDG/TXT generation from KaraokeFinalise
|
||||
- Methods: `create_cdg_package()`, `create_txt_package()`
|
||||
- Uses lyrics_converter and CDGGenerator (existing dependencies)
|
||||
- 18 unit tests with temp files
|
||||
|
||||
4. [x] **Create `backend/services/local_encoding_service.py`** ✅
|
||||
- Extract encoding methods from KaraokeFinalise
|
||||
- Methods: `encode_all_formats()`, individual encoding methods
|
||||
- Hardware acceleration detection with software fallback
|
||||
- 28 unit tests with mocked subprocess
|
||||
|
||||
### Phase 2: Create Encoding Interface ✅ COMPLETE
|
||||
|
||||
5. [x] **Create `backend/services/encoding_interface.py`** ✅
|
||||
- Define ABC `EncodingBackend` with methods: `encode()`, `is_available()`, `get_status()`
|
||||
- `EncodingInput` and `EncodingOutput` dataclasses for config/results
|
||||
- Factory function `get_encoding_backend()` with "local", "gce", "auto" options
|
||||
|
||||
6. [x] **`LocalEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps `LocalEncodingService` with `asyncio.to_thread()`
|
||||
- 17 unit tests
|
||||
|
||||
7. [x] **`GCEEncodingBackend`** ✅
|
||||
- Implements `EncodingBackend` interface
|
||||
- Wraps existing `EncodingService` (encoding_service.py)
|
||||
- Requires GCS paths in options for cloud execution
|
||||
- 6 additional unit tests
|
||||
|
||||
### Phase 3: Create Orchestrator ✅ COMPLETE
|
||||
|
||||
8. [x] **Create `backend/workers/video_worker_orchestrator.py`** ✅
|
||||
- New class that coordinates all stages
|
||||
- Replaces the branching logic in `generate_video()`
|
||||
- Methods:
|
||||
- `__init__(config, job_manager, storage, job_logger)`
|
||||
- `async run() -> OrchestratorResult`
|
||||
- `_run_packaging()` - CDG/TXT
|
||||
- `_run_encoding()` - GCE or local via EncodingBackend interface
|
||||
- `_run_organization()` - Brand code, folder structure
|
||||
- `_run_distribution()` - YouTube, Dropbox, GDrive
|
||||
- `_run_notifications()` - Discord
|
||||
- 27 unit tests (test_video_worker_orchestrator.py)
|
||||
|
||||
9. [x] **Update `backend/workers/video_worker.py`** ✅
|
||||
- Added `USE_NEW_ORCHESTRATOR` feature flag (default: true)
|
||||
- `generate_video()` routes to orchestrator or legacy path
|
||||
- `generate_video_orchestrated()` - new unified pipeline
|
||||
- `generate_video_legacy()` - original code for rollback
|
||||
- Legacy GCE and KaraokeFinalise paths preserved for safety
|
||||
|
||||
### Phase 4: Update KaraokeFinalise to Use Services (OPTIONAL)
|
||||
|
||||
10. [ ] **Update `karaoke_gen/karaoke_finalise/karaoke_finalise.py`**
|
||||
- Import and use the new services
|
||||
- Replace inline YouTube/Discord/CDG code with service calls
|
||||
- Keep process() working for CLI backwards compatibility
|
||||
- Services become the single source of truth
|
||||
- **NOTE**: This is optional for MVP - the orchestrator already uses the extracted services directly
|
||||
|
||||
### Phase 5: Testing & Validation ✅ COMPLETE
|
||||
|
||||
11. [x] **Add unit tests for each service** ✅
|
||||
- test_youtube_upload_service.py (23 tests)
|
||||
- test_discord_service.py (20 tests)
|
||||
- test_packaging_service.py (18 tests)
|
||||
- test_local_encoding_service.py (28 tests)
|
||||
- test_encoding_interface.py (23 tests)
|
||||
- test_video_worker_orchestrator.py (27 tests)
|
||||
- **Total: 139 new tests**
|
||||
|
||||
12. [x] **Full test suite passes** ✅
|
||||
- 976 tests passed (86 skipped - emulator/integration tests)
|
||||
- No regressions from changes
|
||||
|
||||
13. [ ] **Manual E2E testing** (After deployment)
|
||||
- [ ] GCE encoding job with YouTube upload enabled
|
||||
- [ ] Local encoding job with YouTube upload enabled
|
||||
- [ ] CLI karaoke-gen with all features
|
||||
- [ ] Job with CDG/TXT enabled
|
||||
- [ ] Verify output structure matches local CLI
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description | Status |
|
||||
|------|--------|-------------|--------|
|
||||
| `backend/services/youtube_upload_service.py` | Create | YouTube upload logic (~446 lines) | ✅ |
|
||||
| `backend/services/discord_service.py` | Create | Discord notification service (~167 lines) | ✅ |
|
||||
| `backend/services/packaging_service.py` | Create | CDG/TXT package generation (~266 lines) | ✅ |
|
||||
| `backend/services/local_encoding_service.py` | Create | Local FFmpeg encoding service (~512 lines) | ✅ |
|
||||
| `backend/services/encoding_interface.py` | Create | Abstract interface + backends (~418 lines) | ✅ |
|
||||
| `backend/workers/video_worker_orchestrator.py` | Create | Stage orchestration logic (~550 lines) | ✅ |
|
||||
| `backend/workers/video_worker.py` | Modify | Added orchestrator path + feature flag | ✅ |
|
||||
| `backend/tests/test_youtube_upload_service.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_discord_service.py` | Create | 20 unit tests | ✅ |
|
||||
| `backend/tests/test_packaging_service.py` | Create | 18 unit tests | ✅ |
|
||||
| `backend/tests/test_local_encoding_service.py` | Create | 28 unit tests | ✅ |
|
||||
| `backend/tests/test_encoding_interface.py` | Create | 23 unit tests | ✅ |
|
||||
| `backend/tests/test_video_worker_orchestrator.py` | Create | 27 unit tests | ✅ |
|
||||
| `karaoke_gen/karaoke_finalise/karaoke_finalise.py` | Modify | Use extracted services (optional) | Deferred |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- **youtube_upload_service**: Mock YouTube API, test upload/delete/duplicate check
|
||||
- **discord_service**: Mock requests, test webhook POST
|
||||
- **packaging_service**: Use temp files, test CDG/TXT generation
|
||||
- **local_encoding_service**: Mock subprocess, test FFmpeg command construction
|
||||
|
||||
### Integration Tests
|
||||
- **video_worker_orchestrator**: Mock all services, test stage sequencing
|
||||
- **video_worker**: Test with mocked orchestrator
|
||||
|
||||
### E2E Verification
|
||||
- [ ] GCE encoding job with YouTube upload enabled → YouTube link appears
|
||||
- [ ] Local encoding job with YouTube upload enabled → Same behavior
|
||||
- [ ] CLI `karaoke-gen` with all features → No regression
|
||||
- [ ] Job with CDG/TXT enabled → Packages generated
|
||||
- [ ] Dropbox output structure → Matches local CLI exactly
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] Should we keep KaraokeFinalise.process() or deprecate it?
|
||||
- **Decision**: Keep it for CLI backwards compatibility. It will use the new services internally.
|
||||
|
||||
- [x] Should local_encoding_service run FFmpeg sync or async?
|
||||
- **Decision**: Sync implementation, async wrapper via `asyncio.to_thread()` for interface compatibility.
|
||||
|
||||
- [x] Order of packaging vs encoding?
|
||||
- **Decision**: CDG/TXT runs BEFORE encoding (depends on LRC file, not videos). KaraokeFinalise does it this way too.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. The refactor is additive - new services don't replace old code until orchestrator is wired up
|
||||
2. Feature flag: `USE_NEW_ORCHESTRATOR=true` to enable new code path
|
||||
3. If issues found in production:
|
||||
- Set `USE_NEW_ORCHESTRATOR=false` to revert to old behavior
|
||||
- Old `generate_video()` code preserved behind feature flag
|
||||
4. After 1 week stable, remove feature flag and old code
|
||||
|
||||
## Success Criteria
|
||||
|
||||
1. Job `5b78ad46` scenario (GCE + YouTube) would upload to YouTube
|
||||
2. All 803+ existing tests pass
|
||||
3. E2E happy path test passes
|
||||
4. No regression in local CLI functionality
|
||||
5. Output structure matches local CLI exactly
|
||||
6. Code coverage maintained or improved
|
||||
|
||||
## Estimated Scope
|
||||
|
||||
- **Phase 1** (Extract Services): 4 new service files + tests (~400-600 lines each)
|
||||
- **Phase 2** (Interface): 1 new file, 2 modifications (~100 lines)
|
||||
- **Phase 3** (Orchestrator): 2 files (~300-400 lines)
|
||||
- **Phase 4** (KaraokeFinalise): 1 modification (~100 lines of changes)
|
||||
- **Phase 5** (Testing): Integration tests + manual verification
|
||||
|
||||
This is a medium-sized refactor that can be done incrementally. Each phase produces working, tested code.
|
||||
@@ -1,317 +0,0 @@
|
||||
# Plan: White Label B2B Karaoke Portals
|
||||
|
||||
**Created:** 2026-01-08
|
||||
**Branch:** feat/sess-20260108-0358-white-label-portal
|
||||
**Status:** Draft
|
||||
|
||||
## Overview
|
||||
|
||||
Add multi-tenant white label portal capability to karaoke-gen, enabling B2B customers like Vocal Star to access a customized version of the generator app on their own subdomain with their branding, theme, and restricted feature set.
|
||||
|
||||
**First customer: Vocal Star**
|
||||
- Subdomain: `vocalstar.nomadkaraoke.com`
|
||||
- Workflow: Upload original + instrumental audio (no audio search)
|
||||
- Theme: Custom Vocal Star karaoke theme (already exists as JSON)
|
||||
- Distribution: Download only (no Dropbox/GDrive/YouTube)
|
||||
- Auth: Magic link for `@vocal-star.com` / `@vocalstarmusic.com` domains, plus fixed API token option
|
||||
|
||||
## Requirements
|
||||
|
||||
### Functional Requirements
|
||||
- [ ] Tenant identification via subdomain (e.g., `vocalstar.nomadkaraoke.com`)
|
||||
- [ ] Per-tenant frontend branding (logo, colors, metadata)
|
||||
- [ ] Per-tenant feature flags (hide audio search, hide distribution options)
|
||||
- [ ] Per-tenant default theme (always use Vocal Star theme)
|
||||
- [ ] Per-tenant allowed email domains for magic link auth
|
||||
- [ ] Per-tenant fixed API token support (for automation)
|
||||
- [ ] Jobs scoped to tenant (Vocal Star users only see Vocal Star jobs)
|
||||
- [ ] Users scoped to tenant
|
||||
|
||||
### Non-Functional Requirements
|
||||
- [ ] Minimal code duplication - single codebase serves all tenants
|
||||
- [ ] Easy to add new tenants via config, not code changes
|
||||
- [ ] Graceful fallback - default to Nomad Karaoke if no tenant match
|
||||
- [ ] No performance impact on existing single-tenant flow
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Architecture Decision: Config-Driven Multi-Tenancy
|
||||
|
||||
Rather than building a full tenant management system (overkill for ~1 customer/year), we'll use a **config-driven approach**:
|
||||
|
||||
1. **Tenant configs stored in GCS** at `tenants/{tenant_id}/config.json`
|
||||
2. **Frontend detects subdomain** and fetches tenant config from API
|
||||
3. **Backend middleware** extracts tenant from request and applies config
|
||||
4. **Firestore collections** get `tenant_id` field for isolation
|
||||
|
||||
This approach:
|
||||
- Requires no database migrations for new tenants
|
||||
- Configs can be managed via simple JSON files
|
||||
- Easy to test locally with query param override (`?tenant=vocalstar`)
|
||||
- Can evolve to admin UI later if needed
|
||||
|
||||
### Tenant Config Schema
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "vocalstar",
|
||||
"name": "Vocal Star",
|
||||
"subdomain": "vocalstar.nomadkaraoke.com",
|
||||
"branding": {
|
||||
"logo_url": "gs://karaoke-gen-storage/tenants/vocalstar/logo.png",
|
||||
"primary_color": "#ffff00",
|
||||
"secondary_color": "#006CF9",
|
||||
"favicon_url": "gs://karaoke-gen-storage/tenants/vocalstar/favicon.ico",
|
||||
"site_title": "Vocal Star Karaoke Generator"
|
||||
},
|
||||
"features": {
|
||||
"audio_search": false,
|
||||
"file_upload": true,
|
||||
"youtube_upload": false,
|
||||
"dropbox_upload": false,
|
||||
"gdrive_upload": false,
|
||||
"theme_selection": false
|
||||
},
|
||||
"defaults": {
|
||||
"theme_id": "vocalstar",
|
||||
"distribution_mode": "download_only"
|
||||
},
|
||||
"auth": {
|
||||
"allowed_email_domains": ["vocal-star.com", "vocalstarmusic.com"],
|
||||
"fixed_tokens": ["<will be in Secret Manager>"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Key Implementation Details
|
||||
|
||||
#### 1. Subdomain Detection (Frontend)
|
||||
- Next.js middleware checks `request.headers.host`
|
||||
- If subdomain matches tenant, store tenant ID in cookie/context
|
||||
- Fetch `/api/tenant/config` to get branding/features
|
||||
- Apply CSS variables dynamically for colors
|
||||
- Swap logo component based on config
|
||||
|
||||
#### 2. Tenant Middleware (Backend)
|
||||
- FastAPI middleware extracts tenant from:
|
||||
- `X-Tenant-ID` header (set by frontend)
|
||||
- Or subdomain from `Host` header
|
||||
- Or query param `?tenant=` (dev only)
|
||||
- Attaches `tenant_id` to request state
|
||||
- Tenant config loaded from GCS (cached 5 min like themes)
|
||||
|
||||
#### 3. Auth Enhancement
|
||||
- `AuthService.validate_token_full()` already returns `user_email`
|
||||
- Add tenant validation: check if user's email domain is in tenant's allowed list
|
||||
- Support fixed tokens per tenant (stored in `auth_tokens` with `tenant_id`)
|
||||
- Magic links include tenant context for proper redirect
|
||||
|
||||
#### 4. Data Isolation
|
||||
- Add `tenant_id: Optional[str]` to User and Job models
|
||||
- Default to `None` for existing Nomad Karaoke users (backwards compatible)
|
||||
- Query filters: `where('tenant_id', '==', tenant_id or None)`
|
||||
- No schema migration needed - new field, optional
|
||||
|
||||
#### 5. Theme Integration
|
||||
- Upload Vocal Star theme to `themes/vocalstar/` in GCS
|
||||
- Upload assets (backgrounds, fonts) to `themes/vocalstar/assets/`
|
||||
- Tenant config specifies `default_theme_id: "vocalstar"`
|
||||
- When `theme_selection: false`, UI hides theme picker and uses default
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Tenant Foundation (Day 1-2)
|
||||
|
||||
1. [ ] **Create tenant model and config loader**
|
||||
- Create `backend/models/tenant.py` with `TenantConfig`, `TenantBranding`, `TenantFeatures`
|
||||
- Create `backend/services/tenant_service.py` with GCS config loading and caching
|
||||
- Add `GET /api/tenant/config` endpoint (unauthenticated, returns public config)
|
||||
- Add `GET /api/tenant/config/{tenant_id}` for explicit tenant fetch
|
||||
|
||||
2. [ ] **Add tenant middleware**
|
||||
- Create `backend/api/middleware/tenant.py`
|
||||
- Extract tenant from subdomain or header
|
||||
- Attach to `request.state.tenant_id` and `request.state.tenant_config`
|
||||
- Add `get_tenant()` dependency for routes
|
||||
|
||||
3. [ ] **Update data models**
|
||||
- Add `tenant_id: Optional[str] = None` to `User` model
|
||||
- Add `tenant_id: Optional[str] = None` to `Job` model
|
||||
- Update Firestore indexes in Pulumi for tenant-scoped queries
|
||||
|
||||
4. [ ] **Update auth for tenant awareness**
|
||||
- Modify `AuthService` to validate email domain against tenant config
|
||||
- Add tenant context to magic link generation
|
||||
- Support per-tenant fixed tokens in `auth_tokens` collection
|
||||
|
||||
### Phase 2: Upload Vocal Star Theme (Day 2)
|
||||
|
||||
5. [ ] **Prepare Vocal Star theme for GCS**
|
||||
- Convert local paths in `karaoke-prep-styles-vocalstar.json` to GCS paths
|
||||
- Upload background images to `themes/vocalstar/assets/`
|
||||
- Upload Oswald font to `themes/vocalstar/assets/`
|
||||
- Upload `style_params.json` to `themes/vocalstar/`
|
||||
- Create preview image and upload to `themes/vocalstar/preview.png`
|
||||
- Add Vocal Star to `themes/_metadata.json`
|
||||
|
||||
6. [ ] **Create Vocal Star tenant config**
|
||||
- Create `tenants/vocalstar/config.json` in GCS
|
||||
- Upload Vocal Star logo to `tenants/vocalstar/logo.png`
|
||||
- Upload favicon to `tenants/vocalstar/favicon.ico`
|
||||
|
||||
### Phase 3: Frontend Tenant Support (Day 3-4)
|
||||
|
||||
7. [ ] **Add tenant context provider**
|
||||
- Create `frontend/lib/tenant.ts` with tenant config types
|
||||
- Create `frontend/hooks/useTenant.ts` for tenant state
|
||||
- Create `TenantProvider` component that fetches config on mount
|
||||
- Detect subdomain in `_app.tsx` or middleware
|
||||
|
||||
8. [ ] **Dynamic branding**
|
||||
- Create `TenantLogo` component that renders tenant or default logo
|
||||
- Update `globals.css` to support CSS variable overrides
|
||||
- Add `applyTenantColors()` function to set CSS variables from config
|
||||
- Update page metadata (title, favicon) from tenant config
|
||||
|
||||
9. [ ] **Feature-flagged UI**
|
||||
- Wrap audio search in `{tenant.features.audio_search && ...}`
|
||||
- Wrap theme selector in `{tenant.features.theme_selection && ...}`
|
||||
- Wrap distribution options based on tenant features
|
||||
- Show simplified form for Vocal Star (just file upload fields)
|
||||
|
||||
10. [ ] **Update magic link flow**
|
||||
- Pass tenant context when requesting magic link
|
||||
- Magic link redirect goes to tenant subdomain
|
||||
- Verify endpoint extracts tenant from link context
|
||||
|
||||
### Phase 4: Backend Feature Enforcement (Day 4-5)
|
||||
|
||||
11. [ ] **Enforce tenant features in API**
|
||||
- Audio search endpoint: return 403 if `audio_search: false`
|
||||
- File upload: check `file_upload: true`
|
||||
- Distribution settings: ignore/error if disabled features requested
|
||||
- Theme selection: use default if `theme_selection: false`
|
||||
|
||||
12. [ ] **Job creation with tenant context**
|
||||
- `JobCreate` handler sets `tenant_id` from request state
|
||||
- Distribution config respects tenant defaults
|
||||
- Theme defaults to tenant's default theme
|
||||
|
||||
13. [ ] **Query scoping**
|
||||
- Update `JobManager.get_jobs()` to filter by tenant_id
|
||||
- Update `UserService` queries to include tenant_id
|
||||
- Admin endpoints bypass tenant filter (see all tenants)
|
||||
|
||||
### Phase 5: Infrastructure & Deployment (Day 5-6)
|
||||
|
||||
14. [ ] **DNS setup**
|
||||
- Add `vocalstar.nomadkaraoke.com` CNAME to Cloudflare
|
||||
- Configure Cloudflare Pages to accept wildcard subdomain
|
||||
- Add SSL certificate for subdomain
|
||||
|
||||
15. [ ] **Backend CORS update**
|
||||
- Add `vocalstar.nomadkaraoke.com` to allowed origins
|
||||
- Or use pattern matching for `*.nomadkaraoke.com`
|
||||
|
||||
16. [ ] **Pulumi updates**
|
||||
- Add Firestore indexes for tenant_id queries
|
||||
- Add any new secrets (tenant fixed tokens)
|
||||
|
||||
### Phase 6: Testing & Documentation (Day 6-7)
|
||||
|
||||
17. [ ] **Testing**
|
||||
- Unit tests for TenantService config loading
|
||||
- Unit tests for tenant middleware
|
||||
- E2E test for Vocal Star flow (upload audio, generate, download)
|
||||
- Test magic link works for @vocal-star.com email
|
||||
|
||||
18. [ ] **Documentation**
|
||||
- Update CLAUDE.md with tenant info
|
||||
- Create `docs/WHITE-LABEL.md` with tenant setup guide
|
||||
- Document how to add new tenants
|
||||
|
||||
## Files to Create/Modify
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/tenant.py` | Create | Tenant config models |
|
||||
| `backend/services/tenant_service.py` | Create | Tenant config loading/caching |
|
||||
| `backend/api/routes/tenant.py` | Create | Tenant config API endpoint |
|
||||
| `backend/api/middleware/tenant.py` | Create | Tenant extraction middleware |
|
||||
| `backend/api/dependencies.py` | Modify | Add `get_tenant()` dependency |
|
||||
| `backend/models/user.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/models/job.py` | Modify | Add `tenant_id` field |
|
||||
| `backend/services/auth_service.py` | Modify | Tenant email domain validation |
|
||||
| `backend/services/user_service.py` | Modify | Tenant-aware user queries |
|
||||
| `backend/services/job_manager.py` | Modify | Tenant-aware job queries |
|
||||
| `backend/api/routes/file_upload.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/api/routes/audio_search.py` | Modify | Tenant feature enforcement |
|
||||
| `backend/main.py` | Modify | Add tenant middleware |
|
||||
| `frontend/lib/tenant.ts` | Create | Tenant types and API |
|
||||
| `frontend/hooks/useTenant.ts` | Create | Tenant context hook |
|
||||
| `frontend/components/TenantProvider.tsx` | Create | Tenant context provider |
|
||||
| `frontend/components/TenantLogo.tsx` | Create | Dynamic logo component |
|
||||
| `frontend/app/layout.tsx` | Modify | Add TenantProvider |
|
||||
| `frontend/app/globals.css` | Modify | CSS variable support for tenant colors |
|
||||
| `frontend/app/app/page.tsx` | Modify | Feature-flag UI sections |
|
||||
| `infrastructure/__main__.py` | Modify | Add tenant Firestore indexes |
|
||||
| `docs/WHITE-LABEL.md` | Create | Tenant setup documentation |
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `TenantService`: Config loading, caching, validation
|
||||
- `TenantMiddleware`: Subdomain extraction, header extraction
|
||||
- `AuthService`: Tenant email domain validation
|
||||
- `JobManager`: Tenant-scoped queries
|
||||
|
||||
### Integration Tests (Emulator)
|
||||
- Create job with tenant context
|
||||
- Query jobs filtered by tenant
|
||||
- Auth with tenant-allowed email domain
|
||||
- Auth rejection for wrong email domain
|
||||
|
||||
### E2E Tests
|
||||
- Vocal Star happy path: Upload audio files, review lyrics, download video
|
||||
- Auth flow: Magic link for @vocal-star.com, verify redirect to subdomain
|
||||
- Feature hiding: Verify audio search not shown on Vocal Star portal
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [x] **Q: Should tenant configs be in GCS or Firestore?**
|
||||
A: GCS - simpler, matches theme pattern, easy to edit JSON directly
|
||||
|
||||
- [x] **Q: How to handle localhost development for specific tenant?**
|
||||
A: Query param override `?tenant=vocalstar` for local testing
|
||||
|
||||
- [x] **Q: Should admin portal be tenant-aware?**
|
||||
A: No - admins see all tenants. Admin routes bypass tenant filter.
|
||||
|
||||
- [x] **Q: Do we need tenant-specific email sender identity?**
|
||||
A: Yes - use `{tenant}@nomadkaraoke.com` (e.g., `vocalstar@nomadkaraoke.com`). Catchall is configured.
|
||||
|
||||
- [x] **Q: Vocal Star logo and favicon - do you have these assets?**
|
||||
A: Logo provided as `vocalstar-logo.jpg` - yellow star, white text on black. Will convert to PNG with transparency for web use.
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Since this is additive (new optional `tenant_id` field), rollback is straightforward:
|
||||
|
||||
1. Revert frontend to not check tenant
|
||||
2. Revert backend middleware to not extract tenant
|
||||
3. Existing data unaffected (tenant_id = null = Nomad Karaoke)
|
||||
4. No schema migration to reverse
|
||||
|
||||
For DNS: Simply remove CNAME record if needed.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Vocal Star user can access `vocalstar.nomadkaraoke.com`
|
||||
- [ ] Portal shows Vocal Star branding (logo, colors)
|
||||
- [ ] Audio search is hidden; only file upload available
|
||||
- [ ] Theme is locked to Vocal Star style
|
||||
- [ ] Distribution options hidden; download only
|
||||
- [ ] Magic link works for @vocal-star.com emails
|
||||
- [ ] Fixed API token works for automation
|
||||
- [ ] Generated videos use Vocal Star theme
|
||||
- [ ] Nomad Karaoke portal unchanged at gen.nomadkaraoke.com
|
||||
@@ -1,222 +0,0 @@
|
||||
# Plan: Worker Logs Rearchitecture
|
||||
|
||||
**Created:** 2026-01-04
|
||||
**Branch:** `feat/sess-20260104-1931-logs-research`
|
||||
**Status:** Implementation Complete - Pending PR Merge & Deployment
|
||||
|
||||
## Overview
|
||||
|
||||
Move `worker_logs` from embedded array in job documents to a Firestore subcollection to avoid the 1MB document size limit. The current approach caused job 501258e1 to fail when logs reached 1.26 MB (98.6% of document size).
|
||||
|
||||
## Problem Statement
|
||||
|
||||
- Firestore documents have a 1 MB size limit
|
||||
- `worker_logs` is stored as an array field within job documents
|
||||
- Long-running jobs (especially video encoding) can generate thousands of log entries
|
||||
- When the limit is exceeded, job updates fail with: `Document cannot be written because its size exceeds the maximum allowed size`
|
||||
|
||||
## Requirements
|
||||
|
||||
- [x] Logs must remain associated with their job
|
||||
- [x] Logs should be queryable by job_id, worker type, timestamp
|
||||
- [x] Old logs should auto-cleanup (TTL: 30 days)
|
||||
- [x] Existing API endpoint (`GET /api/jobs/{id}/logs`) must continue to work
|
||||
- [x] Frontend must continue to display logs without changes
|
||||
- [x] Migration path for existing jobs (optional - can just start fresh)
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Option A: Subcollection (Recommended)
|
||||
Store logs in `jobs/{job_id}/logs/{log_id}` subcollection.
|
||||
|
||||
**Pros:**
|
||||
- Natural association with job (delete job = delete logs with cascading)
|
||||
- Each log entry is its own document (no size limit concerns)
|
||||
- Can add TTL via Firestore TTL policies
|
||||
- Efficient queries with composite indexes
|
||||
|
||||
**Cons:**
|
||||
- More Firestore read operations for paginated log fetching
|
||||
- Slightly more complex write logic
|
||||
|
||||
### Option B: Separate Collection
|
||||
Store logs in `job_logs` collection with `job_id` field.
|
||||
|
||||
**Pros:**
|
||||
- Can query across all jobs
|
||||
- Simpler TTL management
|
||||
|
||||
**Cons:**
|
||||
- Orphaned logs if job deleted (need cleanup logic)
|
||||
- Extra index needed for job_id queries
|
||||
|
||||
### Decision: Option A (Subcollection)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Backend Changes
|
||||
|
||||
1. [x] **Create log entry model** - `backend/models/worker_log.py`
|
||||
- `WorkerLogEntry` dataclass with: id, job_id, timestamp, level, worker, message, metadata, ttl_expiry
|
||||
- Factory method `create()` for easy instantiation
|
||||
- `to_dict()` for Firestore storage
|
||||
- `to_legacy_dict()` for API compatibility
|
||||
- `from_dict()` for Firestore retrieval
|
||||
|
||||
2. [x] **Update FirestoreService** - `backend/services/firestore_service.py`
|
||||
- Added `append_log_to_subcollection(job_id, log_entry)` method
|
||||
- Added `get_logs_from_subcollection(job_id, limit, since_timestamp, worker, offset)` method
|
||||
- Added `get_logs_count_from_subcollection(job_id)` method
|
||||
- Added `delete_logs_subcollection(job_id, batch_size)` method
|
||||
- Old `append_worker_log` marked as deprecated
|
||||
|
||||
3. [x] **Update JobManager** - `backend/services/job_manager.py`
|
||||
- Modified `append_worker_log()` to use subcollection when `USE_LOG_SUBCOLLECTION=true`
|
||||
- Modified `get_worker_logs()` to read from subcollection with fallback to embedded array
|
||||
- Added `get_worker_logs_count()` for total count
|
||||
- Updated `delete_job()` to clean up logs subcollection
|
||||
|
||||
4. [x] **Update jobs API** - `backend/api/routes/jobs.py`
|
||||
- `GET /api/jobs/{id}/logs` - Updated to use new methods
|
||||
- Response format stays the same (backward compatible)
|
||||
- Fixed `next_index` calculation
|
||||
|
||||
5. [x] **Add feature flag** - `backend/config.py`
|
||||
- Added `USE_LOG_SUBCOLLECTION` environment variable (default: true)
|
||||
|
||||
6. [x] **Add TTL configuration** - `infrastructure/__main__.py`
|
||||
- Added Firestore TTL policy for logs subcollection (30 days)
|
||||
- Added composite index for worker + timestamp queries
|
||||
|
||||
### Phase 2: Testing
|
||||
|
||||
7. [x] **Unit tests** - `backend/tests/test_worker_log_subcollection.py`
|
||||
- 24 tests covering:
|
||||
- WorkerLogEntry model creation and conversion
|
||||
- FirestoreService subcollection methods
|
||||
- JobManager logging with feature flag
|
||||
- Edge cases (unicode, long messages, empty messages)
|
||||
|
||||
8. [x] **Integration tests** - `backend/tests/emulator/test_worker_logs_subcollection.py`
|
||||
- 11 tests using real Firestore emulator:
|
||||
- Single and sequential writes
|
||||
- TTL field verification
|
||||
- Concurrent writes (no race conditions)
|
||||
- Large volume (500 logs)
|
||||
- Worker filtering
|
||||
- Timestamp ordering
|
||||
- Subcollection deletion
|
||||
- Concurrent workers interleaved
|
||||
- Job deletion cleanup
|
||||
- Comparison with embedded array
|
||||
|
||||
### Phase 3: Deployment & Migration
|
||||
|
||||
9. [ ] **Deploy with feature flag**
|
||||
- Add `USE_LOG_SUBCOLLECTION=true` env var (default is true)
|
||||
- Deploy to production
|
||||
|
||||
10. [ ] **Monitor**
|
||||
- Watch for errors in Cloud Logging
|
||||
- Verify logs appear in frontend
|
||||
|
||||
11. [x] **Backward compatibility**
|
||||
- Old jobs with embedded arrays still work (fallback in get_worker_logs)
|
||||
- No migration needed - new jobs use subcollection, old jobs use array
|
||||
|
||||
### Phase 4: Cleanup (Future)
|
||||
|
||||
12. [ ] **Remove old code path** (after confidence period)
|
||||
- Remove `worker_logs` field from job model
|
||||
- Remove old `append_worker_log` method
|
||||
- Remove feature flag
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
| File | Action | Description |
|
||||
|------|--------|-------------|
|
||||
| `backend/models/worker_log.py` | Created | WorkerLogEntry dataclass |
|
||||
| `backend/services/firestore_service.py` | Modified | Added subcollection methods |
|
||||
| `backend/services/job_manager.py` | Modified | Switch to subcollection with fallback |
|
||||
| `backend/api/routes/jobs.py` | Modified | Updated logs endpoint |
|
||||
| `backend/config.py` | Modified | Added USE_LOG_SUBCOLLECTION setting |
|
||||
| `backend/tests/test_worker_log_subcollection.py` | Created | Unit tests (24 tests) |
|
||||
| `backend/tests/emulator/test_worker_logs_subcollection.py` | Created | Integration tests (11 tests) |
|
||||
| `infrastructure/__main__.py` | Modified | Added TTL policy and index |
|
||||
|
||||
## Log Entry Schema
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "video", "render", "screens", "distribution"
|
||||
message: str
|
||||
id: str # Auto-generated UUID
|
||||
job_id: str
|
||||
ttl_expiry: datetime # timestamp + 30 days
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
```
|
||||
|
||||
## Firestore Structure
|
||||
|
||||
```
|
||||
jobs/
|
||||
{job_id}/
|
||||
... (job fields, worker_logs still present for old jobs)
|
||||
logs/ # subcollection for new jobs
|
||||
{log_id}/
|
||||
id: string
|
||||
job_id: string
|
||||
timestamp: datetime
|
||||
level: string
|
||||
worker: string
|
||||
message: string
|
||||
metadata: map (optional)
|
||||
ttl_expiry: datetime
|
||||
```
|
||||
|
||||
## API Response (Unchanged)
|
||||
|
||||
```json
|
||||
{
|
||||
"logs": [
|
||||
{
|
||||
"timestamp": "2026-01-04T12:00:00Z",
|
||||
"level": "INFO",
|
||||
"worker": "video",
|
||||
"message": "Starting video generation"
|
||||
}
|
||||
],
|
||||
"next_index": 42,
|
||||
"total_logs": 150
|
||||
}
|
||||
```
|
||||
|
||||
## Testing Results
|
||||
|
||||
All tests pass:
|
||||
- 24 unit tests in `test_worker_log_subcollection.py`
|
||||
- 11 emulator integration tests in `test_worker_logs_subcollection.py`
|
||||
- 68 total tests in emulator suite pass
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
1. Set `USE_LOG_SUBCOLLECTION=false` env var
|
||||
2. Redeploy backend
|
||||
3. New logs will go back to embedded array
|
||||
4. Subcollection logs will remain but won't be read
|
||||
|
||||
## Answers to Open Questions
|
||||
|
||||
- [x] **Should we paginate logs in the API?** - Yes, added `offset` parameter support
|
||||
- [x] **What's the optimal TTL period?** - 30 days (configurable via DEFAULT_LOG_TTL_DAYS)
|
||||
- [x] **Should we index by timestamp for efficient range queries?** - Yes, added composite index (worker + timestamp)
|
||||
|
||||
## References
|
||||
|
||||
- [Firestore TTL](https://cloud.google.com/firestore/docs/ttl)
|
||||
- [Firestore Subcollections](https://firebase.google.com/docs/firestore/data-model#subcollections)
|
||||
- [Firestore Limits](https://firebase.google.com/docs/firestore/quotas#limits)
|
||||
- [Pulumi Firestore Field](https://www.pulumi.com/registry/packages/gcp/api-docs/firestore/field/)
|
||||
@@ -1,101 +0,0 @@
|
||||
---
|
||||
description: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Goal: Identify inconsistencies, duplications, ambiguities, and underspecified items across the three core artifacts (`spec.md`, `plan.md`, `tasks.md`) before implementation. This command MUST run only after `/tasks` has successfully produced a complete `tasks.md`.
|
||||
|
||||
STRICTLY READ-ONLY: Do **not** modify any files. Output a structured analysis report. Offer an optional remediation plan (user must explicitly approve before any follow-up editing commands would be invoked manually).
|
||||
|
||||
Constitution Authority: The project constitution (`.specify/memory/constitution.md`) is **non-negotiable** within this analysis scope. Constitution conflicts are automatically CRITICAL and require adjustment of the spec, plan, or tasks—not dilution, reinterpretation, or silent ignoring of the principle. If a principle itself needs to change, that must occur in a separate, explicit constitution update outside `/analyze`.
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` once from repo root and parse JSON for FEATURE_DIR and AVAILABLE_DOCS. Derive absolute paths:
|
||||
- SPEC = FEATURE_DIR/spec.md
|
||||
- PLAN = FEATURE_DIR/plan.md
|
||||
- TASKS = FEATURE_DIR/tasks.md
|
||||
Abort with an error message if any required file is missing (instruct the user to run missing prerequisite command).
|
||||
|
||||
2. Load artifacts:
|
||||
- Parse spec.md sections: Overview/Context, Functional Requirements, Non-Functional Requirements, User Stories, Edge Cases (if present).
|
||||
- Parse plan.md: Architecture/stack choices, Data Model references, Phases, Technical constraints.
|
||||
- Parse tasks.md: Task IDs, descriptions, phase grouping, parallel markers [P], referenced file paths.
|
||||
- Load constitution `.specify/memory/constitution.md` for principle validation.
|
||||
|
||||
3. Build internal semantic models:
|
||||
- Requirements inventory: Each functional + non-functional requirement with a stable key (derive slug based on imperative phrase; e.g., "User can upload file" -> `user-can-upload-file`).
|
||||
- User story/action inventory.
|
||||
- Task coverage mapping: Map each task to one or more requirements or stories (inference by keyword / explicit reference patterns like IDs or key phrases).
|
||||
- Constitution rule set: Extract principle names and any MUST/SHOULD normative statements.
|
||||
|
||||
4. Detection passes:
|
||||
A. Duplication detection:
|
||||
- Identify near-duplicate requirements. Mark lower-quality phrasing for consolidation.
|
||||
B. Ambiguity detection:
|
||||
- Flag vague adjectives (fast, scalable, secure, intuitive, robust) lacking measurable criteria.
|
||||
- Flag unresolved placeholders (TODO, TKTK, ???, <placeholder>, etc.).
|
||||
C. Underspecification:
|
||||
- Requirements with verbs but missing object or measurable outcome.
|
||||
- User stories missing acceptance criteria alignment.
|
||||
- Tasks referencing files or components not defined in spec/plan.
|
||||
D. Constitution alignment:
|
||||
- Any requirement or plan element conflicting with a MUST principle.
|
||||
- Missing mandated sections or quality gates from constitution.
|
||||
E. Coverage gaps:
|
||||
- Requirements with zero associated tasks.
|
||||
- Tasks with no mapped requirement/story.
|
||||
- Non-functional requirements not reflected in tasks (e.g., performance, security).
|
||||
F. Inconsistency:
|
||||
- Terminology drift (same concept named differently across files).
|
||||
- Data entities referenced in plan but absent in spec (or vice versa).
|
||||
- Task ordering contradictions (e.g., integration tasks before foundational setup tasks without dependency note).
|
||||
- Conflicting requirements (e.g., one requires to use Next.js while other says to use Vue as the framework).
|
||||
|
||||
5. Severity assignment heuristic:
|
||||
- CRITICAL: Violates constitution MUST, missing core spec artifact, or requirement with zero coverage that blocks baseline functionality.
|
||||
- HIGH: Duplicate or conflicting requirement, ambiguous security/performance attribute, untestable acceptance criterion.
|
||||
- MEDIUM: Terminology drift, missing non-functional task coverage, underspecified edge case.
|
||||
- LOW: Style/wording improvements, minor redundancy not affecting execution order.
|
||||
|
||||
6. Produce a Markdown report (no file writes) with sections:
|
||||
|
||||
### Specification Analysis Report
|
||||
| ID | Category | Severity | Location(s) | Summary | Recommendation |
|
||||
|----|----------|----------|-------------|---------|----------------|
|
||||
| A1 | Duplication | HIGH | spec.md:L120-134 | Two similar requirements ... | Merge phrasing; keep clearer version |
|
||||
(Add one row per finding; generate stable IDs prefixed by category initial.)
|
||||
|
||||
Additional subsections:
|
||||
- Coverage Summary Table:
|
||||
| Requirement Key | Has Task? | Task IDs | Notes |
|
||||
- Constitution Alignment Issues (if any)
|
||||
- Unmapped Tasks (if any)
|
||||
- Metrics:
|
||||
* Total Requirements
|
||||
* Total Tasks
|
||||
* Coverage % (requirements with >=1 task)
|
||||
* Ambiguity Count
|
||||
* Duplication Count
|
||||
* Critical Issues Count
|
||||
|
||||
7. At end of report, output a concise Next Actions block:
|
||||
- If CRITICAL issues exist: Recommend resolving before `/implement`.
|
||||
- If only LOW/MEDIUM: User may proceed, but provide improvement suggestions.
|
||||
- Provide explicit command suggestions: e.g., "Run /specify with refinement", "Run /plan to adjust architecture", "Manually edit tasks.md to add coverage for 'performance-metrics'".
|
||||
|
||||
8. Ask the user: "Would you like me to suggest concrete remediation edits for the top N issues?" (Do NOT apply them automatically.)
|
||||
|
||||
Behavior rules:
|
||||
- NEVER modify files.
|
||||
- NEVER hallucinate missing sections—if absent, report them.
|
||||
- KEEP findings deterministic: if rerun without changes, produce consistent IDs and counts.
|
||||
- LIMIT total findings in the main table to 50; aggregate remainder in a summarized overflow note.
|
||||
- If zero issues found, emit a success report with coverage statistics and proceed recommendation.
|
||||
|
||||
Context: $ARGUMENTS
|
||||
@@ -1,158 +0,0 @@
|
||||
---
|
||||
description: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Goal: Detect and reduce ambiguity or missing decision points in the active feature specification and record the clarifications directly in the spec file.
|
||||
|
||||
Note: This clarification workflow is expected to run (and be completed) BEFORE invoking `/plan`. If the user explicitly states they are skipping clarification (e.g., exploratory spike), you may proceed, but must warn that downstream rework risk increases.
|
||||
|
||||
Execution steps:
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --paths-only` from repo root **once** (combined `--json --paths-only` mode / `-Json -PathsOnly`). Parse minimal JSON payload fields:
|
||||
- `FEATURE_DIR`
|
||||
- `FEATURE_SPEC`
|
||||
- (Optionally capture `IMPL_PLAN`, `TASKS` for future chained flows.)
|
||||
- If JSON parsing fails, abort and instruct user to re-run `/specify` or verify feature branch environment.
|
||||
|
||||
2. Load the current spec file. Perform a structured ambiguity & coverage scan using this taxonomy. For each category, mark status: Clear / Partial / Missing. Produce an internal coverage map used for prioritization (do not output raw map unless no questions will be asked).
|
||||
|
||||
Functional Scope & Behavior:
|
||||
- Core user goals & success criteria
|
||||
- Explicit out-of-scope declarations
|
||||
- User roles / personas differentiation
|
||||
|
||||
Domain & Data Model:
|
||||
- Entities, attributes, relationships
|
||||
- Identity & uniqueness rules
|
||||
- Lifecycle/state transitions
|
||||
- Data volume / scale assumptions
|
||||
|
||||
Interaction & UX Flow:
|
||||
- Critical user journeys / sequences
|
||||
- Error/empty/loading states
|
||||
- Accessibility or localization notes
|
||||
|
||||
Non-Functional Quality Attributes:
|
||||
- Performance (latency, throughput targets)
|
||||
- Scalability (horizontal/vertical, limits)
|
||||
- Reliability & availability (uptime, recovery expectations)
|
||||
- Observability (logging, metrics, tracing signals)
|
||||
- Security & privacy (authN/Z, data protection, threat assumptions)
|
||||
- Compliance / regulatory constraints (if any)
|
||||
|
||||
Integration & External Dependencies:
|
||||
- External services/APIs and failure modes
|
||||
- Data import/export formats
|
||||
- Protocol/versioning assumptions
|
||||
|
||||
Edge Cases & Failure Handling:
|
||||
- Negative scenarios
|
||||
- Rate limiting / throttling
|
||||
- Conflict resolution (e.g., concurrent edits)
|
||||
|
||||
Constraints & Tradeoffs:
|
||||
- Technical constraints (language, storage, hosting)
|
||||
- Explicit tradeoffs or rejected alternatives
|
||||
|
||||
Terminology & Consistency:
|
||||
- Canonical glossary terms
|
||||
- Avoided synonyms / deprecated terms
|
||||
|
||||
Completion Signals:
|
||||
- Acceptance criteria testability
|
||||
- Measurable Definition of Done style indicators
|
||||
|
||||
Misc / Placeholders:
|
||||
- TODO markers / unresolved decisions
|
||||
- Ambiguous adjectives ("robust", "intuitive") lacking quantification
|
||||
|
||||
For each category with Partial or Missing status, add a candidate question opportunity unless:
|
||||
- Clarification would not materially change implementation or validation strategy
|
||||
- Information is better deferred to planning phase (note internally)
|
||||
|
||||
3. Generate (internally) a prioritized queue of candidate clarification questions (maximum 5). Do NOT output them all at once. Apply these constraints:
|
||||
- Maximum of 5 total questions across the whole session.
|
||||
- Each question must be answerable with EITHER:
|
||||
* A short multiple‑choice selection (2–5 distinct, mutually exclusive options), OR
|
||||
* A one-word / short‑phrase answer (explicitly constrain: "Answer in <=5 words").
|
||||
- Only include questions whose answers materially impact architecture, data modeling, task decomposition, test design, UX behavior, operational readiness, or compliance validation.
|
||||
- Ensure category coverage balance: attempt to cover the highest impact unresolved categories first; avoid asking two low-impact questions when a single high-impact area (e.g., security posture) is unresolved.
|
||||
- Exclude questions already answered, trivial stylistic preferences, or plan-level execution details (unless blocking correctness).
|
||||
- Favor clarifications that reduce downstream rework risk or prevent misaligned acceptance tests.
|
||||
- If more than 5 categories remain unresolved, select the top 5 by (Impact * Uncertainty) heuristic.
|
||||
|
||||
4. Sequential questioning loop (interactive):
|
||||
- Present EXACTLY ONE question at a time.
|
||||
- For multiple‑choice questions render options as a Markdown table:
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| A | <Option A description> |
|
||||
| B | <Option B description> |
|
||||
| C | <Option C description> | (add D/E as needed up to 5)
|
||||
| Short | Provide a different short answer (<=5 words) | (Include only if free-form alternative is appropriate)
|
||||
|
||||
- For short‑answer style (no meaningful discrete options), output a single line after the question: `Format: Short answer (<=5 words)`.
|
||||
- After the user answers:
|
||||
* Validate the answer maps to one option or fits the <=5 word constraint.
|
||||
* If ambiguous, ask for a quick disambiguation (count still belongs to same question; do not advance).
|
||||
* Once satisfactory, record it in working memory (do not yet write to disk) and move to the next queued question.
|
||||
- Stop asking further questions when:
|
||||
* All critical ambiguities resolved early (remaining queued items become unnecessary), OR
|
||||
* User signals completion ("done", "good", "no more"), OR
|
||||
* You reach 5 asked questions.
|
||||
- Never reveal future queued questions in advance.
|
||||
- If no valid questions exist at start, immediately report no critical ambiguities.
|
||||
|
||||
5. Integration after EACH accepted answer (incremental update approach):
|
||||
- Maintain in-memory representation of the spec (loaded once at start) plus the raw file contents.
|
||||
- For the first integrated answer in this session:
|
||||
* Ensure a `## Clarifications` section exists (create it just after the highest-level contextual/overview section per the spec template if missing).
|
||||
* Under it, create (if not present) a `### Session YYYY-MM-DD` subheading for today.
|
||||
- Append a bullet line immediately after acceptance: `- Q: <question> → A: <final answer>`.
|
||||
- Then immediately apply the clarification to the most appropriate section(s):
|
||||
* Functional ambiguity → Update or add a bullet in Functional Requirements.
|
||||
* User interaction / actor distinction → Update User Stories or Actors subsection (if present) with clarified role, constraint, or scenario.
|
||||
* Data shape / entities → Update Data Model (add fields, types, relationships) preserving ordering; note added constraints succinctly.
|
||||
* Non-functional constraint → Add/modify measurable criteria in Non-Functional / Quality Attributes section (convert vague adjective to metric or explicit target).
|
||||
* Edge case / negative flow → Add a new bullet under Edge Cases / Error Handling (or create such subsection if template provides placeholder for it).
|
||||
* Terminology conflict → Normalize term across spec; retain original only if necessary by adding `(formerly referred to as "X")` once.
|
||||
- If the clarification invalidates an earlier ambiguous statement, replace that statement instead of duplicating; leave no obsolete contradictory text.
|
||||
- Save the spec file AFTER each integration to minimize risk of context loss (atomic overwrite).
|
||||
- Preserve formatting: do not reorder unrelated sections; keep heading hierarchy intact.
|
||||
- Keep each inserted clarification minimal and testable (avoid narrative drift).
|
||||
|
||||
6. Validation (performed after EACH write plus final pass):
|
||||
- Clarifications session contains exactly one bullet per accepted answer (no duplicates).
|
||||
- Total asked (accepted) questions ≤ 5.
|
||||
- Updated sections contain no lingering vague placeholders the new answer was meant to resolve.
|
||||
- No contradictory earlier statement remains (scan for now-invalid alternative choices removed).
|
||||
- Markdown structure valid; only allowed new headings: `## Clarifications`, `### Session YYYY-MM-DD`.
|
||||
- Terminology consistency: same canonical term used across all updated sections.
|
||||
|
||||
7. Write the updated spec back to `FEATURE_SPEC`.
|
||||
|
||||
8. Report completion (after questioning loop ends or early termination):
|
||||
- Number of questions asked & answered.
|
||||
- Path to updated spec.
|
||||
- Sections touched (list names).
|
||||
- Coverage summary table listing each taxonomy category with Status: Resolved (was Partial/Missing and addressed), Deferred (exceeds question quota or better suited for planning), Clear (already sufficient), Outstanding (still Partial/Missing but low impact).
|
||||
- If any Outstanding or Deferred remain, recommend whether to proceed to `/plan` or run `/clarify` again later post-plan.
|
||||
- Suggested next command.
|
||||
|
||||
Behavior rules:
|
||||
- If no meaningful ambiguities found (or all potential questions would be low-impact), respond: "No critical ambiguities detected worth formal clarification." and suggest proceeding.
|
||||
- If spec file missing, instruct user to run `/specify` first (do not create a new spec here).
|
||||
- Never exceed 5 total asked questions (clarification retries for a single question do not count as new questions).
|
||||
- Avoid speculative tech stack questions unless the absence blocks functional clarity.
|
||||
- Respect user early termination signals ("stop", "done", "proceed").
|
||||
- If no questions asked due to full coverage, output a compact coverage summary (all categories Clear) then suggest advancing.
|
||||
- If quota reached with unresolved high-impact categories remaining, explicitly flag them under Deferred with rationale.
|
||||
|
||||
Context for prioritization: $ARGUMENTS
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
description: Create or update the project constitution from interactive or provided principle inputs, ensuring all dependent templates stay in sync.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
You are updating the project constitution at `.specify/memory/constitution.md`. This file is a TEMPLATE containing placeholder tokens in square brackets (e.g. `[PROJECT_NAME]`, `[PRINCIPLE_1_NAME]`). Your job is to (a) collect/derive concrete values, (b) fill the template precisely, and (c) propagate any amendments across dependent artifacts.
|
||||
|
||||
Follow this execution flow:
|
||||
|
||||
1. Load the existing constitution template at `.specify/memory/constitution.md`.
|
||||
- Identify every placeholder token of the form `[ALL_CAPS_IDENTIFIER]`.
|
||||
**IMPORTANT**: The user might require less or more principles than the ones used in the template. If a number is specified, respect that - follow the general template. You will update the doc accordingly.
|
||||
|
||||
2. Collect/derive values for placeholders:
|
||||
- If user input (conversation) supplies a value, use it.
|
||||
- Otherwise infer from existing repo context (README, docs, prior constitution versions if embedded).
|
||||
- For governance dates: `RATIFICATION_DATE` is the original adoption date (if unknown ask or mark TODO), `LAST_AMENDED_DATE` is today if changes are made, otherwise keep previous.
|
||||
- `CONSTITUTION_VERSION` must increment according to semantic versioning rules:
|
||||
* MAJOR: Backward incompatible governance/principle removals or redefinitions.
|
||||
* MINOR: New principle/section added or materially expanded guidance.
|
||||
* PATCH: Clarifications, wording, typo fixes, non-semantic refinements.
|
||||
- If version bump type ambiguous, propose reasoning before finalizing.
|
||||
|
||||
3. Draft the updated constitution content:
|
||||
- Replace every placeholder with concrete text (no bracketed tokens left except intentionally retained template slots that the project has chosen not to define yet—explicitly justify any left).
|
||||
- Preserve heading hierarchy and comments can be removed once replaced unless they still add clarifying guidance.
|
||||
- Ensure each Principle section: succinct name line, paragraph (or bullet list) capturing non‑negotiable rules, explicit rationale if not obvious.
|
||||
- Ensure Governance section lists amendment procedure, versioning policy, and compliance review expectations.
|
||||
|
||||
4. Consistency propagation checklist (convert prior checklist into active validations):
|
||||
- Read `.specify/templates/plan-template.md` and ensure any "Constitution Check" or rules align with updated principles.
|
||||
- Read `.specify/templates/spec-template.md` for scope/requirements alignment—update if constitution adds/removes mandatory sections or constraints.
|
||||
- Read `.specify/templates/tasks-template.md` and ensure task categorization reflects new or removed principle-driven task types (e.g., observability, versioning, testing discipline).
|
||||
- Read each command file in `.specify/templates/commands/*.md` (including this one) to verify no outdated references (agent-specific names like CLAUDE only) remain when generic guidance is required.
|
||||
- Read any runtime guidance docs (e.g., `README.md`, `docs/quickstart.md`, or agent-specific guidance files if present). Update references to principles changed.
|
||||
|
||||
5. Produce a Sync Impact Report (prepend as an HTML comment at top of the constitution file after update):
|
||||
- Version change: old → new
|
||||
- List of modified principles (old title → new title if renamed)
|
||||
- Added sections
|
||||
- Removed sections
|
||||
- Templates requiring updates (✅ updated / ⚠ pending) with file paths
|
||||
- Follow-up TODOs if any placeholders intentionally deferred.
|
||||
|
||||
6. Validation before final output:
|
||||
- No remaining unexplained bracket tokens.
|
||||
- Version line matches report.
|
||||
- Dates ISO format YYYY-MM-DD.
|
||||
- Principles are declarative, testable, and free of vague language ("should" → replace with MUST/SHOULD rationale where appropriate).
|
||||
|
||||
7. Write the completed constitution back to `.specify/memory/constitution.md` (overwrite).
|
||||
|
||||
8. Output a final summary to the user with:
|
||||
- New version and bump rationale.
|
||||
- Any files flagged for manual follow-up.
|
||||
- Suggested commit message (e.g., `docs: amend constitution to vX.Y.Z (principle additions + governance update)`).
|
||||
|
||||
Formatting & Style Requirements:
|
||||
- Use Markdown headings exactly as in the template (do not demote/promote levels).
|
||||
- Wrap long rationale lines to keep readability (<100 chars ideally) but do not hard enforce with awkward breaks.
|
||||
- Keep a single blank line between sections.
|
||||
- Avoid trailing whitespace.
|
||||
|
||||
If the user supplies partial updates (e.g., only one principle revision), still perform validation and version decision steps.
|
||||
|
||||
If critical info missing (e.g., ratification date truly unknown), insert `TODO(<FIELD_NAME>): explanation` and include in the Sync Impact Report under deferred items.
|
||||
|
||||
Do not create a new template; always operate on the existing `.specify/memory/constitution.md` file.
|
||||
@@ -1,56 +0,0 @@
|
||||
---
|
||||
description: Execute the implementation plan by processing and executing all tasks defined in tasks.md
|
||||
---
|
||||
|
||||
The user input can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute.
|
||||
|
||||
2. Load and analyze the implementation context:
|
||||
- **REQUIRED**: Read tasks.md for the complete task list and execution plan
|
||||
- **REQUIRED**: Read plan.md for tech stack, architecture, and file structure
|
||||
- **IF EXISTS**: Read data-model.md for entities and relationships
|
||||
- **IF EXISTS**: Read contracts/ for API specifications and test requirements
|
||||
- **IF EXISTS**: Read research.md for technical decisions and constraints
|
||||
- **IF EXISTS**: Read quickstart.md for integration scenarios
|
||||
|
||||
3. Parse tasks.md structure and extract:
|
||||
- **Task phases**: Setup, Tests, Core, Integration, Polish
|
||||
- **Task dependencies**: Sequential vs parallel execution rules
|
||||
- **Task details**: ID, description, file paths, parallel markers [P]
|
||||
- **Execution flow**: Order and dependency requirements
|
||||
|
||||
4. Execute implementation following the task plan:
|
||||
- **Phase-by-phase execution**: Complete each phase before moving to the next
|
||||
- **Respect dependencies**: Run sequential tasks in order, parallel tasks [P] can run together
|
||||
- **Follow TDD approach**: Execute test tasks before their corresponding implementation tasks
|
||||
- **File-based coordination**: Tasks affecting the same files must run sequentially
|
||||
- **Validation checkpoints**: Verify each phase completion before proceeding
|
||||
|
||||
5. Implementation execution rules:
|
||||
- **Setup first**: Initialize project structure, dependencies, configuration
|
||||
- **Tests before code**: If you need to write tests for contracts, entities, and integration scenarios
|
||||
- **Core development**: Implement models, services, CLI commands, endpoints
|
||||
- **Integration work**: Database connections, middleware, logging, external services
|
||||
- **Polish and validation**: Unit tests, performance optimization, documentation
|
||||
|
||||
6. Progress tracking and error handling:
|
||||
- Report progress after each completed task
|
||||
- Halt execution if any non-parallel task fails
|
||||
- For parallel tasks [P], continue with successful tasks, report failed ones
|
||||
- Provide clear error messages with context for debugging
|
||||
- Suggest next steps if implementation cannot proceed
|
||||
- **IMPORTANT** For completed tasks, make sure to mark the task off as [X] in the tasks file.
|
||||
|
||||
7. Completion validation:
|
||||
- Verify all required tasks are completed
|
||||
- Check that implemented features match the original specification
|
||||
- Validate that tests pass and coverage meets requirements
|
||||
- Confirm the implementation follows the technical plan
|
||||
- Report final status with summary of completed work
|
||||
|
||||
Note: This command assumes a complete task breakdown exists in tasks.md. If tasks are incomplete or missing, suggest running `/tasks` first to regenerate the task list.
|
||||
@@ -1,43 +0,0 @@
|
||||
---
|
||||
description: Execute the implementation planning workflow using the plan template to generate design artifacts.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
Given the implementation details provided as an argument, do this:
|
||||
|
||||
1. Run `.specify/scripts/bash/setup-plan.sh --json` from the repo root and parse JSON for FEATURE_SPEC, IMPL_PLAN, SPECS_DIR, BRANCH. All future file paths must be absolute.
|
||||
- BEFORE proceeding, inspect FEATURE_SPEC for a `## Clarifications` section with at least one `Session` subheading. If missing or clearly ambiguous areas remain (vague adjectives, unresolved critical choices), PAUSE and instruct the user to run `/clarify` first to reduce rework. Only continue if: (a) Clarifications exist OR (b) an explicit user override is provided (e.g., "proceed without clarification"). Do not attempt to fabricate clarifications yourself.
|
||||
2. Read and analyze the feature specification to understand:
|
||||
- The feature requirements and user stories
|
||||
- Functional and non-functional requirements
|
||||
- Success criteria and acceptance criteria
|
||||
- Any technical constraints or dependencies mentioned
|
||||
|
||||
3. Read the constitution at `.specify/memory/constitution.md` to understand constitutional requirements.
|
||||
|
||||
4. Execute the implementation plan template:
|
||||
- Load `.specify/templates/plan-template.md` (already copied to IMPL_PLAN path)
|
||||
- Set Input path to FEATURE_SPEC
|
||||
- Run the Execution Flow (main) function steps 1-9
|
||||
- The template is self-contained and executable
|
||||
- Follow error handling and gate checks as specified
|
||||
- Let the template guide artifact generation in $SPECS_DIR:
|
||||
* Phase 0 generates research.md
|
||||
* Phase 1 generates data-model.md, contracts/, quickstart.md
|
||||
* Phase 2 generates tasks.md
|
||||
- Incorporate user-provided details from arguments into Technical Context: $ARGUMENTS
|
||||
- Update Progress Tracking as you complete each phase
|
||||
|
||||
5. Verify execution completed:
|
||||
- Check Progress Tracking shows all phases complete
|
||||
- Ensure all required artifacts were generated
|
||||
- Confirm no ERROR states in execution
|
||||
|
||||
6. Report results with branch name, file paths, and generated artifacts.
|
||||
|
||||
Use absolute paths with the repository root for all file operations to avoid path issues.
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
description: Create or update the feature specification from a natural language feature description.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
The text the user typed after `/specify` in the triggering message **is** the feature description. Assume you always have it available in this conversation even if `$ARGUMENTS` appears literally below. Do not ask the user to repeat it unless they provided an empty command.
|
||||
|
||||
Given that feature description, do this:
|
||||
|
||||
1. Run the script `.specify/scripts/bash/create-new-feature.sh --json "$ARGUMENTS"` from repo root and parse its JSON output for BRANCH_NAME and SPEC_FILE. All file paths must be absolute.
|
||||
**IMPORTANT** You must only ever run this script once. The JSON is provided in the terminal as output - always refer to it to get the actual content you're looking for.
|
||||
2. Load `.specify/templates/spec-template.md` to understand required sections.
|
||||
3. Write the specification to SPEC_FILE using the template structure, replacing placeholders with concrete details derived from the feature description (arguments) while preserving section order and headings.
|
||||
4. Report completion with branch name, spec file path, and readiness for the next phase.
|
||||
|
||||
Note: The script creates and checks out the new branch and initializes the spec file before writing.
|
||||
@@ -1,62 +0,0 @@
|
||||
---
|
||||
description: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
|
||||
---
|
||||
|
||||
The user input to you can be provided directly by the agent or as a command argument - you **MUST** consider it before proceeding with the prompt (if not empty).
|
||||
|
||||
User input:
|
||||
|
||||
$ARGUMENTS
|
||||
|
||||
1. Run `.specify/scripts/bash/check-prerequisites.sh --json` from repo root and parse FEATURE_DIR and AVAILABLE_DOCS list. All paths must be absolute.
|
||||
2. Load and analyze available design documents:
|
||||
- Always read plan.md for tech stack and libraries
|
||||
- IF EXISTS: Read data-model.md for entities
|
||||
- IF EXISTS: Read contracts/ for API endpoints
|
||||
- IF EXISTS: Read research.md for technical decisions
|
||||
- IF EXISTS: Read quickstart.md for test scenarios
|
||||
|
||||
Note: Not all projects have all documents. For example:
|
||||
- CLI tools might not have contracts/
|
||||
- Simple libraries might not need data-model.md
|
||||
- Generate tasks based on what's available
|
||||
|
||||
3. Generate tasks following the template:
|
||||
- Use `.specify/templates/tasks-template.md` as the base
|
||||
- Replace example tasks with actual tasks based on:
|
||||
* **Setup tasks**: Project init, dependencies, linting
|
||||
* **Test tasks [P]**: One per contract, one per integration scenario
|
||||
* **Core tasks**: One per entity, service, CLI command, endpoint
|
||||
* **Integration tasks**: DB connections, middleware, logging
|
||||
* **Polish tasks [P]**: Unit tests, performance, docs
|
||||
|
||||
4. Task generation rules:
|
||||
- Each contract file → contract test task marked [P]
|
||||
- Each entity in data-model → model creation task marked [P]
|
||||
- Each endpoint → implementation task (not parallel if shared files)
|
||||
- Each user story → integration test marked [P]
|
||||
- Different files = can be parallel [P]
|
||||
- Same file = sequential (no [P])
|
||||
|
||||
5. Order tasks by dependencies:
|
||||
- Setup before everything
|
||||
- Tests before implementation (TDD)
|
||||
- Models before services
|
||||
- Services before endpoints
|
||||
- Core before integration
|
||||
- Everything before polish
|
||||
|
||||
6. Include parallel execution examples:
|
||||
- Group [P] tasks that can run together
|
||||
- Show actual Task agent commands
|
||||
|
||||
7. Create FEATURE_DIR/tasks.md with:
|
||||
- Correct feature name from implementation plan
|
||||
- Numbered tasks (T001, T002, etc.)
|
||||
- Clear file paths for each task
|
||||
- Dependency notes
|
||||
- Parallel execution guidance
|
||||
|
||||
Context for task generation: $ARGUMENTS
|
||||
|
||||
The tasks.md should be immediately executable - each task must be specific enough that an LLM can complete it without additional context.
|
||||
@@ -1,368 +0,0 @@
|
||||
---
|
||||
name: Anchor Sequence Optimization
|
||||
overview: Replace the current multiprocessing-based anchor sequence algorithm with an efficient single-threaded implementation using hash-based n-gram indexing, greedy longest-first matching, and simplified scoring to achieve sub-30-second performance in both local and cloud environments.
|
||||
todos:
|
||||
- id: create-golden-fixtures
|
||||
content: Create script to generate 10 golden test fixtures from representative song data (varying lengths/complexity), run current algorithm to capture expected anchor outputs
|
||||
status: completed
|
||||
- id: create-golden-test
|
||||
content: Create test_anchor_sequence_golden.py with regression tests that verify anchor finder output matches fixtures, ensure tests pass with current implementation before any changes
|
||||
status: completed
|
||||
- id: remove-multiprocessing
|
||||
content: Remove multiprocessing.Pool from find_anchors() and _remove_overlapping_sequences(), replace with single-threaded loops
|
||||
status: completed
|
||||
- id: build-ngram-index
|
||||
content: Add _build_ngram_index() method to create hash-based lookup table for O(1) n-gram matching
|
||||
status: completed
|
||||
- id: greedy-skip
|
||||
content: Add covered_positions set tracking and skip logic to avoid processing overlapping n-grams
|
||||
status: completed
|
||||
- id: simple-scoring
|
||||
content: Add _simple_score_anchor() with rule-based heuristics, make spaCy scoring optional via use_nlp_scoring flag
|
||||
status: completed
|
||||
- id: optimize-overlap
|
||||
content: Replace O(n^2) overlap removal with O(n log n) sorted single-pass algorithm
|
||||
status: completed
|
||||
- id: add-perf-tests
|
||||
content: Add performance benchmark test asserting <30 second completion for all fixture sizes
|
||||
status: pending
|
||||
---
|
||||
|
||||
# Anchor Sequence Algorithm Optimization
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
The current `AnchorSequenceFinder` has critical performance issues:
|
||||
|
||||
### Root Causes of Hanging
|
||||
|
||||
1. **Multiprocessing Pool with fork()**: Python's `Pool()` uses `fork()` on Linux, which causes deadlocks in Cloud Run containers
|
||||
2. **spaCy model loading in forked processes**: The `PhraseAnalyzer` loads spaCy's `en_core_web_sm` model, which can hang when forked
|
||||
3. **Pool creation blocking**: The hang occurs before job submission, so timeout checks never trigger
|
||||
|
||||
### Algorithmic Inefficiencies
|
||||
|
||||
1. **O(n^2) n-gram matching**: Linear search through references for each n-gram
|
||||
2. **Redundant work**: Processes all n-gram lengths (3 to ~30) even though longer matches are preferred
|
||||
3. **O(n^2) overlap removal**: Each anchor checked against all previous anchors
|
||||
4. **Expensive phrase scoring**: spaCy NLP called twice per anchor (phrase + context)
|
||||
|
||||
## Proposed Solution: Hash-Based Greedy Algorithm
|
||||
|
||||
### Core Algorithm Changes
|
||||
|
||||
**1. Hash-Based N-gram Index (O(1) lookup instead of O(n))**
|
||||
|
||||
- Pre-build a dictionary mapping n-grams to their positions in each reference
|
||||
- Use tuple of cleaned words as key for O(1) lookup
|
||||
- Build index once during preprocessing
|
||||
|
||||
**2. Greedy Longest-First Matching**
|
||||
|
||||
- Process n-gram lengths from longest to shortest (current behavior)
|
||||
- BUT: Skip positions already covered by longer anchors immediately
|
||||
- Use a "covered positions" set to avoid redundant work
|
||||
|
||||
**3. Single-Threaded Processing**
|
||||
|
||||
- Remove all `multiprocessing.Pool` usage
|
||||
- Use efficient algorithms instead of parallelism
|
||||
- Works identically in local and cloud environments
|
||||
|
||||
**4. Simplified Phrase Scoring (Optional)**
|
||||
|
||||
- Make spaCy-based scoring optional via configuration flag
|
||||
- Default to rule-based heuristic scoring:
|
||||
- Prefer sequences starting/ending at word boundaries
|
||||
- Prefer sequences matching more reference sources
|
||||
- Prefer longer sequences
|
||||
- Option to enable full NLP scoring for higher-quality results offline
|
||||
|
||||
**5. O(n log n) Overlap Removal**
|
||||
|
||||
- Sort anchors by position
|
||||
- Single pass with interval tracking using sorted list
|
||||
- Skip anchors that overlap with any already-selected anchor
|
||||
|
||||
### New Data Structures
|
||||
|
||||
```python
|
||||
# N-gram index: maps (word1, word2, ...) -> {source: [positions]}
|
||||
ngram_index: Dict[Tuple[str, ...], Dict[str, List[int]]]
|
||||
|
||||
# Covered positions in transcription (for greedy skip)
|
||||
covered_trans_positions: Set[int]
|
||||
```
|
||||
|
||||
### Performance Targets
|
||||
|
||||
- **Target**: Complete in under 30 seconds for any song
|
||||
- **Typical song**: 200-400 words, 3 reference sources
|
||||
- **Expected performance**: 1-5 seconds for typical inputs
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Execution Order (Critical)
|
||||
|
||||
1. **Phase 0**: Create golden test fixtures and verify tests pass with current code
|
||||
2. **Phase 1-5**: Implement optimizations incrementally, running golden tests after each phase
|
||||
3. **Final**: Add performance benchmarks
|
||||
|
||||
### Files to Create/Modify
|
||||
|
||||
- `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py` (new)
|
||||
- `lyrics_transcriber_temp/tests/fixtures/anchor_golden/*.json` (new)
|
||||
- `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py` (new)
|
||||
- `lyrics_transcriber_temp/lyrics_transcriber/correction/anchor_sequence.py` (modify)
|
||||
|
||||
### Key Changes
|
||||
|
||||
**Phase 0: Golden Test Setup** (MUST complete and pass before any algorithm changes)
|
||||
|
||||
- Create fixture generation script with 10 representative test cases
|
||||
- Run script to generate fixtures using current implementation
|
||||
- Create golden test file that loads fixtures and verifies outputs
|
||||
- Verify all golden tests pass with current code
|
||||
|
||||
**Phase 1: Remove Multiprocessing**
|
||||
|
||||
- Replace `Pool().apply_async()` with direct function calls
|
||||
- Remove all pool creation, cleanup, and timeout handling
|
||||
- Single-threaded loop through n-gram lengths
|
||||
|
||||
**Phase 2: Add Hash-Based Index**
|
||||
|
||||
- New method `_build_ngram_index()` that builds lookup table during preprocessing
|
||||
- Modify `_find_matching_sources()` to use O(1) index lookup
|
||||
- Index built once, shared across all n-gram length iterations
|
||||
|
||||
**Phase 3: Greedy Position Skipping**
|
||||
|
||||
- Track covered transcription positions in a set
|
||||
- Skip n-grams that start in already-covered positions
|
||||
- Reduces redundant work significantly for songs with good matches
|
||||
|
||||
**Phase 4: Simplify Scoring**
|
||||
|
||||
- Add `use_nlp_scoring: bool = False` parameter to constructor
|
||||
- New method `_simple_score_anchor()` using rule-based heuristics
|
||||
- Keep spaCy scoring available for offline/high-quality mode
|
||||
|
||||
**Phase 5: Optimize Overlap Removal**
|
||||
|
||||
- Sort anchors by (start_position, -length)
|
||||
- Single pass: keep anchor if no overlap with last kept anchor
|
||||
- O(n log n) total instead of O(n^2)
|
||||
|
||||
## API Compatibility
|
||||
|
||||
The public interface remains unchanged:
|
||||
|
||||
- `find_anchors(transcribed, references, transcription_result) -> List[ScoredAnchor]`
|
||||
- `find_gaps(transcribed, anchors, references, transcription_result) -> List[GapSequence]`
|
||||
|
||||
New optional constructor parameters:
|
||||
|
||||
- `use_nlp_scoring: bool = False` - Enable spaCy-based phrase scoring (slower but higher quality)
|
||||
- Timeout parameter retained but less critical with efficient algorithm
|
||||
|
||||
## Fallback Strategy
|
||||
|
||||
If performance is still insufficient for edge cases:
|
||||
|
||||
1. **Early termination**: Stop after finding sufficient coverage (e.g., 80% of words anchored)
|
||||
2. **Adaptive n-gram range**: Start with medium lengths (8-15), expand only if needed
|
||||
3. **Sampling**: For very long songs, sample n-gram positions instead of exhaustive search
|
||||
|
||||
## Phase 0: Golden Test Setup (BEFORE any algorithm changes)
|
||||
|
||||
### Goal
|
||||
|
||||
Create regression tests using REAL song data from local cache to verify the optimized algorithm produces equivalent results.
|
||||
|
||||
### Cache File Structure (Understanding)
|
||||
|
||||
The local cache at `~/lyrics-transcriber-cache/` contains:
|
||||
|
||||
- **Reference lyrics**: `{provider}_{hash}_converted.json` where provider is spotify/genius/lrclib/musixmatch
|
||||
- Hash is based on (artist, title) lookup - same song shares same hash across providers
|
||||
- `_raw.json` contains metadata (track name, artist)
|
||||
- `_converted.json` contains LyricsData structure with segments/words
|
||||
- **Transcriptions**: `audioshake_{hash}_converted.json`
|
||||
- Hash is based on audio file content (different from reference hash)
|
||||
- Contains TranscriptionData structure
|
||||
- **Anchor results**: `anchors_{hash}.json`
|
||||
- Hash is computed from combined inputs (transcription + all references)
|
||||
|
||||
### Manual Song Selection
|
||||
|
||||
Identify 10 songs from cache with good coverage. Search raw files for track names:
|
||||
|
||||
### Finding Song Mappings
|
||||
|
||||
To link audioshake transcriptions to reference lyrics, search cache files:
|
||||
|
||||
```bash
|
||||
# Find songs by searching raw files for track names
|
||||
grep -h "trackName" ~/lyrics-transcriber-cache/spotify_*_raw.json | head -20
|
||||
|
||||
# Find audioshake files for a known song (e.g., search for filename patterns)
|
||||
ls ~/lyrics-transcriber-cache/audioshake_*_raw.json | head -20
|
||||
|
||||
# Search converted files for specific lyrics text
|
||||
grep -l "Napoleon did surrender" ~/lyrics-transcriber-cache/*_converted.json
|
||||
```
|
||||
|
||||
**Target song list** (reference_hash to be paired with audioshake_hash):
|
||||
|
||||
1. ABBA - Waterloo (refs: 3365f6a1e42388d84bcbfc0ac6f8aa6c) - short, repetitive chorus
|
||||
2. The Format - Time Bomb (refs: 3a3802ec28c4ab6689909b7ef784bb6f) - medium length
|
||||
3. Rancid - Time Bomb (refs: d02d20b854cbc6b890f274bd6c278752) - fast punk lyrics
|
||||
4. Minutemen - Storm in My House (refs: bc625704b26b307d8680ae5235b74994) - short
|
||||
5. + 6 more songs to be identified during fixture generation by exploring cache
|
||||
|
||||
### Fixture Generation Script
|
||||
|
||||
Create `lyrics_transcriber_temp/scripts/generate_anchor_fixtures.py`:
|
||||
|
||||
```python
|
||||
# Input: List of (name, audioshake_hash, reference_hash) tuples
|
||||
# For each:
|
||||
# 1. Load audioshake_{hash}_converted.json as transcription
|
||||
# 2. Load {spotify,genius,lrclib}_{hash}_converted.json as references
|
||||
# 3. Run current AnchorSequenceFinder
|
||||
# 4. Save complete fixture with inputs + outputs
|
||||
```
|
||||
|
||||
The script will:
|
||||
|
||||
1. Accept a config file listing the 10 song mappings
|
||||
2. Load real LyricsData from cache files
|
||||
3. Run the CURRENT AnchorSequenceFinder implementation
|
||||
4. Save fixtures that capture the real behavior
|
||||
|
||||
### Fixture Format
|
||||
|
||||
```
|
||||
lyrics_transcriber_temp/tests/fixtures/anchor_golden/
|
||||
- waterloo_abba.json
|
||||
- time_bomb_format.json
|
||||
- time_bomb_rancid.json
|
||||
- ...
|
||||
```
|
||||
|
||||
Each fixture contains REAL data:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "waterloo_abba",
|
||||
"description": "ABBA - Waterloo, short song with repetitive chorus",
|
||||
"source_hashes": {
|
||||
"audioshake": "1654b7c0755fad7f23bc888308f274de",
|
||||
"references": "3365f6a1e42388d84bcbfc0ac6f8aa6c"
|
||||
},
|
||||
"transcription": {
|
||||
"segments": [...], // Full LyricsData from audioshake cache
|
||||
"word_count": 150
|
||||
},
|
||||
"references": {
|
||||
"spotify": { "segments": [...] },
|
||||
"genius": { "segments": [...] },
|
||||
"lrclib": { "segments": [...] }
|
||||
},
|
||||
"expected_anchors": [
|
||||
{
|
||||
"text": "My my at Waterloo Napoleon did surrender",
|
||||
"transcription_position": 0,
|
||||
"length": 7,
|
||||
"reference_positions": {"spotify": 0, "genius": 0, "lrclib": 0},
|
||||
"confidence": 1.0
|
||||
},
|
||||
// ... all anchors from current implementation
|
||||
],
|
||||
"expected_anchor_count": 25,
|
||||
"expected_word_coverage": 142,
|
||||
"expected_coverage_percent": 94.6
|
||||
}
|
||||
```
|
||||
|
||||
### Golden Test
|
||||
|
||||
Create `lyrics_transcriber_temp/tests/unit/correction/test_anchor_sequence_golden.py`:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
FIXTURES_DIR = Path(__file__).parent.parent.parent / "fixtures" / "anchor_golden"
|
||||
|
||||
@pytest.fixture(params=list(FIXTURES_DIR.glob("*.json")))
|
||||
def golden_fixture(request):
|
||||
with open(request.param) as f:
|
||||
return json.load(f)
|
||||
|
||||
def test_anchor_count_matches(golden_fixture, anchor_finder):
|
||||
"""Anchor count should be within 10% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchor_count"]
|
||||
assert abs(len(anchors) - expected) <= expected * 0.1
|
||||
|
||||
def test_key_anchors_match(golden_fixture, anchor_finder):
|
||||
"""First, last, and longest anchors should match exactly."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
expected = golden_fixture["expected_anchors"]
|
||||
# Compare first anchor
|
||||
assert anchors[0].text == expected[0]["text"]
|
||||
# Compare positions match
|
||||
...
|
||||
|
||||
def test_word_coverage(golden_fixture, anchor_finder):
|
||||
"""Total word coverage should be within 5% of expected."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
coverage = calculate_coverage(anchors, golden_fixture)
|
||||
expected = golden_fixture["expected_coverage_percent"]
|
||||
assert abs(coverage - expected) <= 5.0
|
||||
|
||||
def test_no_invalid_positions(golden_fixture, anchor_finder):
|
||||
"""All anchor positions should be valid."""
|
||||
anchors = run_anchor_finder(golden_fixture)
|
||||
word_count = golden_fixture["transcription"]["word_count"]
|
||||
for anchor in anchors:
|
||||
assert 0 <= anchor.transcription_position < word_count
|
||||
```
|
||||
|
||||
### Validation Steps
|
||||
|
||||
1. **Generate fixtures from current implementation**:
|
||||
```bash
|
||||
cd lyrics_transcriber_temp
|
||||
python scripts/generate_anchor_fixtures.py
|
||||
```
|
||||
|
||||
2. **Verify fixtures were created**:
|
||||
```bash
|
||||
ls tests/fixtures/anchor_golden/
|
||||
# Should show 10 .json files
|
||||
```
|
||||
|
||||
3. **Run golden tests with CURRENT implementation**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
```
|
||||
|
||||
4. **All tests MUST pass before any algorithm changes**
|
||||
|
||||
5. **After each optimization phase, re-run golden tests**:
|
||||
```bash
|
||||
pytest tests/unit/correction/test_anchor_sequence_golden.py -v
|
||||
# Must still pass!
|
||||
```
|
||||
|
||||
|
||||
## Testing (Post-Optimization)
|
||||
|
||||
- Golden tests verify correctness (same outputs as before)
|
||||
- Existing unit tests verify API compatibility
|
||||
- Performance benchmark test with 30-second timeout assertion
|
||||
- Test with various song lengths via golden fixtures
|
||||
@@ -1,717 +0,0 @@
|
||||
---
|
||||
name: Instrumental Review UI
|
||||
overview: Add an intelligent instrumental selection workflow with audio analysis, waveform visualization, playback controls, and the ability to mute sections of backing vocals to create custom instrumentals. This replaces the current text-only CLI prompt with a web UI that works for both local and remote workflows, with shared core logic for DRY/maintainable code.
|
||||
todos:
|
||||
- id: shared-core-module
|
||||
content: Create karaoke_gen/instrumental_review/ shared module with AudioAnalyzer and AudioEditor classes (used by both local and remote)
|
||||
status: completed
|
||||
- id: shared-core-tests
|
||||
content: Add comprehensive unit tests for shared module with >80% coverage (tests/unit/test_instrumental_review.py)
|
||||
status: completed
|
||||
- id: backend-services
|
||||
content: Create backend service wrappers that use shared core module (audio_analysis_service.py, audio_editing_service.py)
|
||||
status: completed
|
||||
- id: backend-services-tests
|
||||
content: Add unit tests for backend services with mocked GCS interactions (backend/tests/test_audio_services.py)
|
||||
status: completed
|
||||
- id: api-endpoints
|
||||
content: Add new API endpoints for analysis, audio streaming, and custom instrumental creation
|
||||
status: completed
|
||||
- id: api-endpoints-tests
|
||||
content: Add API endpoint tests with >80% coverage (backend/tests/test_instrumental_api.py)
|
||||
status: completed
|
||||
- id: render-worker-integration
|
||||
content: Integrate audio analysis into render_video_worker before transitioning to awaiting_instrumental_selection
|
||||
status: completed
|
||||
- id: render-worker-tests
|
||||
content: Add/update render_video_worker tests to cover new analysis integration
|
||||
status: completed
|
||||
- id: frontend-components
|
||||
content: Create React components (WaveformDisplay, AudioPlayer, RegionSelector, InstrumentalReviewPage)
|
||||
status: completed
|
||||
- id: local-cli-integration
|
||||
content: Update local gen_cli.py to use shared core module with local file paths (feature parity with remote)
|
||||
status: completed
|
||||
- id: local-cli-review-server
|
||||
content: Add lightweight review server to local CLI for browser-based review (similar to lyrics review pattern)
|
||||
status: completed
|
||||
- id: remote-cli-update
|
||||
content: Update remote_cli.py to use new analysis API and open browser for review
|
||||
status: completed
|
||||
- id: cli-tests
|
||||
content: Add CLI tests verifying feature parity between local and remote modes
|
||||
status: completed
|
||||
- id: integration-tests
|
||||
content: Add end-to-end integration tests for the full instrumental review workflow
|
||||
status: completed
|
||||
---
|
||||
|
||||
# Intelligent Instrumental Selection Plan
|
||||
|
||||
## Current State
|
||||
|
||||
The instrumental selection workflow currently:
|
||||
|
||||
- Presents a simple 1/2 choice in the CLI (clean vs. with_backing)
|
||||
- Requires manual inspection in Audacity to evaluate backing vocals quality
|
||||
- Cannot handle the common "Scenario 3" where backing vocals need partial muting
|
||||
- Has no audio preview capability in remote mode
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────────────────────┐
|
||||
│ INSTRUMENTAL REVIEW WORKFLOW │
|
||||
├────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. AUDIO ANALYSIS (automatic after render_video completes) │
|
||||
│ │ │
|
||||
│ ├── Analyze backing vocals stem for amplitude │
|
||||
│ ├── Detect audible segments (> -40dB threshold) │
|
||||
│ ├── Generate waveform visualization image │
|
||||
│ └── Store analysis results in job state_data │
|
||||
│ │
|
||||
│ 2. AUTO-DECISION CHECK │
|
||||
│ │ │
|
||||
│ └── IF no audible segments detected: │
|
||||
│ ├── Set recommended_selection = "clean" │
|
||||
│ └── Skip to step 4 (or show minimal UI with suggestion) │
|
||||
│ │
|
||||
│ 3. HUMAN REVIEW (Web UI) │
|
||||
│ │ │
|
||||
│ ├── Display waveform with time axis │
|
||||
│ ├── Show audible segment markers │
|
||||
│ ├── Audio playback with waveform-click seeking │
|
||||
│ ├── Toggle between: clean / with_backing / preview-custom │
|
||||
│ ├── Select regions to mute (click/drag on waveform) │
|
||||
│ └── Create custom instrumental if regions selected │
|
||||
│ │
|
||||
│ 4. SELECTION SUBMISSION │
|
||||
│ │ │
|
||||
│ └── POST /api/jobs/{job_id}/select-instrumental │
|
||||
│ with selection = "clean" | "with_backing" | "custom" │
|
||||
│ │
|
||||
└────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Architecture: Shared Core for DRY/SOLID
|
||||
|
||||
To ensure feature parity between local and remote CLIs while keeping code maintainable, we use a **shared core module** pattern:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ SHARED CORE ARCHITECTURE │
|
||||
├─────────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ karaoke_gen/instrumental_review/ <── SHARED CORE MODULE │
|
||||
│ ├── __init__.py │
|
||||
│ ├── analyzer.py # AudioAnalyzer class (pure Python, no GCS) │
|
||||
│ ├── editor.py # AudioEditor class (pure Python, no GCS) │
|
||||
│ ├── waveform.py # WaveformGenerator class (matplotlib) │
|
||||
│ └── models.py # Pydantic models: AnalysisResult, MuteRegion │
|
||||
│ │
|
||||
│ ┌─────────────────────┐ ┌──────────────────────┐ │
|
||||
│ │ LOCAL CLI │ │ REMOTE BACKEND │ │
|
||||
│ │ (karaoke-gen) │ │ (Cloud Run) │ │
|
||||
│ ├─────────────────────┤ ├──────────────────────┤ │
|
||||
│ │ │ │ │ │
|
||||
│ │ gen_cli.py │ │ audio_analysis_ │ │
|
||||
│ │ └── uses shared ───┼──────────────┼──► service.py │ │
|
||||
│ │ core directly │ │ └── wraps shared│ │
|
||||
│ │ with local │ │ core with │ │
|
||||
│ │ file paths │ │ GCS I/O │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ local_review_ │ │ audio_editing_ │ │
|
||||
│ │ server.py │ │ service.py │ │
|
||||
│ │ └── serves same ───┼──────────────┼──► (same pattern) │ │
|
||||
│ │ React UI │ │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ └─────────────────────┘ └──────────────────────┘ │
|
||||
│ │
|
||||
│ BENEFITS: │
|
||||
│ ✓ Single source of truth for analysis logic │
|
||||
│ ✓ Bug fixes apply to both local and remote │
|
||||
│ ✓ Easy to test core logic without GCS mocks │
|
||||
│ ✓ Backend services are thin wrappers (Single Responsibility) │
|
||||
│ ✓ Same React UI served by both local server and Cloud Run │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Shared Core Module Design
|
||||
|
||||
```python
|
||||
# karaoke_gen/instrumental_review/analyzer.py
|
||||
class AudioAnalyzer:
|
||||
"""
|
||||
Analyzes audio files for backing vocals content.
|
||||
Pure Python - no cloud dependencies. Works with local file paths.
|
||||
"""
|
||||
def __init__(self, silence_threshold_db: float = -40.0,
|
||||
min_segment_duration_ms: int = 100):
|
||||
self.silence_threshold_db = silence_threshold_db
|
||||
self.min_segment_duration_ms = min_segment_duration_ms
|
||||
|
||||
def analyze(self, audio_path: str) -> AnalysisResult:
|
||||
"""Analyze audio file and return structured result."""
|
||||
...
|
||||
|
||||
def get_amplitude_envelope(self, audio_path: str,
|
||||
window_ms: int = 100) -> List[float]:
|
||||
"""Get amplitude envelope for waveform rendering."""
|
||||
...
|
||||
|
||||
# karaoke_gen/instrumental_review/editor.py
|
||||
class AudioEditor:
|
||||
"""
|
||||
Creates custom instrumentals by muting regions of backing vocals.
|
||||
Pure Python - no cloud dependencies.
|
||||
"""
|
||||
def create_custom_instrumental(
|
||||
self,
|
||||
clean_instrumental_path: str,
|
||||
backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
output_path: str
|
||||
) -> str:
|
||||
"""Create custom instrumental with muted regions."""
|
||||
...
|
||||
|
||||
# karaoke_gen/instrumental_review/waveform.py
|
||||
class WaveformGenerator:
|
||||
"""Generates waveform visualization images."""
|
||||
def generate(self, audio_path: str, output_path: str,
|
||||
segments: List[AudibleSegment] = None,
|
||||
width: int = 1200, height: int = 200) -> str:
|
||||
"""Generate waveform PNG with optional segment highlighting."""
|
||||
...
|
||||
```
|
||||
|
||||
## Implementation Components
|
||||
|
||||
### 1. Shared Core Module (karaoke_gen/instrumental_review/)
|
||||
|
||||
Create the shared module first - this is the foundation:
|
||||
|
||||
```python
|
||||
# karaoke_gen/instrumental_review/models.py
|
||||
from pydantic import BaseModel
|
||||
from typing import List, Optional
|
||||
from enum import Enum
|
||||
|
||||
class RecommendedSelection(str, Enum):
|
||||
CLEAN = "clean"
|
||||
WITH_BACKING = "with_backing"
|
||||
REVIEW_NEEDED = "review_needed"
|
||||
|
||||
class AudibleSegment(BaseModel):
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
duration_seconds: float
|
||||
avg_amplitude_db: float
|
||||
|
||||
class MuteRegion(BaseModel):
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
|
||||
class AnalysisResult(BaseModel):
|
||||
has_audible_content: bool
|
||||
total_duration_seconds: float
|
||||
audible_segments: List[AudibleSegment]
|
||||
recommended_selection: RecommendedSelection
|
||||
waveform_data: Optional[List[float]] = None # For JSON API response
|
||||
```
|
||||
|
||||
### 2. Backend Service Wrappers
|
||||
|
||||
Create `backend/services/audio_analysis_service.py`:
|
||||
|
||||
```python
|
||||
# Key functionality:
|
||||
- analyze_backing_vocals(audio_path) -> AnalysisResult
|
||||
- Uses pydub for audio loading
|
||||
- Calculates RMS amplitude in dB across time windows
|
||||
- Detects audible segments (contiguous regions > threshold)
|
||||
- Returns: {
|
||||
"has_audible_content": bool,
|
||||
"total_duration_seconds": float,
|
||||
"audible_segments": [
|
||||
{"start_seconds": float, "end_seconds": float,
|
||||
"duration_seconds": float, "avg_amplitude_db": float}
|
||||
],
|
||||
"recommended_selection": "clean" | "with_backing" | "review_needed"
|
||||
}
|
||||
|
||||
- generate_waveform_image(audio_path, output_path, segments=None) -> str
|
||||
- Uses matplotlib to render amplitude envelope over time
|
||||
- Highlights audible segments with different colors
|
||||
- Returns path to generated PNG image
|
||||
```
|
||||
|
||||
### 2. Audio Editing Service
|
||||
|
||||
Create `backend/services/audio_editing_service.py`:
|
||||
|
||||
```python
|
||||
# Key functionality:
|
||||
- create_custom_instrumental(
|
||||
clean_instrumental_path: str,
|
||||
backing_vocals_path: str,
|
||||
mute_regions: List[Tuple[float, float]], # [(start_s, end_s), ...]
|
||||
output_path: str
|
||||
) -> str
|
||||
- Loads backing vocals
|
||||
- Applies silence to mute_regions
|
||||
- Mixes with clean instrumental
|
||||
- Saves as new FLAC file
|
||||
```
|
||||
|
||||
### 3. Backend API Endpoints
|
||||
|
||||
Add to `backend/api/routes/jobs.py`:
|
||||
|
||||
```python
|
||||
GET /api/jobs/{job_id}/instrumental-analysis
|
||||
# Returns analysis data + waveform URL
|
||||
|
||||
GET /api/jobs/{job_id}/audio-stream/{stem_type}
|
||||
# Stream audio for playback (clean, backing_vocals, combined)
|
||||
|
||||
POST /api/jobs/{job_id}/create-custom-instrumental
|
||||
# Body: {"mute_regions": [[0.5, 2.3], [45.0, 48.5]]}
|
||||
# Creates custom instrumental, updates job state
|
||||
|
||||
POST /api/jobs/{job_id}/select-instrumental
|
||||
# Extended to accept: "clean" | "with_backing" | "custom"
|
||||
```
|
||||
|
||||
### 4. Frontend Web UI
|
||||
|
||||
Create `frontend/app/jobs/[id]/instrumental-review/page.tsx`:
|
||||
|
||||
```
|
||||
Components needed:
|
||||
- WaveformDisplay: SVG/Canvas waveform with time axis
|
||||
- AudioPlayer: HTML5 audio with controls + waveform sync
|
||||
- RegionSelector: Click/drag to select mute regions
|
||||
- InstrumentalOptions: Radio buttons for clean/backing/custom
|
||||
- SegmentList: Table showing detected audible segments
|
||||
```
|
||||
|
||||
Key interactions:
|
||||
|
||||
- Click on waveform to seek playback position
|
||||
- Drag on waveform to select mute region
|
||||
- Toggle audio source (clean/backing/preview)
|
||||
- Submit selection
|
||||
|
||||
### 5. Backend Worker Updates
|
||||
|
||||
Modify `backend/workers/render_video_worker.py`:
|
||||
|
||||
```python
|
||||
# After video render completes, before transitioning to AWAITING_INSTRUMENTAL_SELECTION:
|
||||
1. Download backing_vocals stem
|
||||
2. Run audio analysis
|
||||
3. Generate waveform image
|
||||
4. Upload waveform to GCS
|
||||
5. Store analysis results in job.state_data
|
||||
6. If no audible content: set recommended_selection = "clean"
|
||||
7. Transition to AWAITING_INSTRUMENTAL_SELECTION
|
||||
```
|
||||
|
||||
### 6. CLI Updates
|
||||
|
||||
Modify `karaoke_gen/utils/remote_cli.py`:
|
||||
|
||||
```python
|
||||
def handle_instrumental_selection(self, job_id: str):
|
||||
# Get analysis data
|
||||
analysis = self.client.get_instrumental_analysis(job_id)
|
||||
|
||||
# If no backing vocals detected, suggest auto-select
|
||||
if analysis['recommended_selection'] == 'clean':
|
||||
if self.config.non_interactive:
|
||||
# Auto-select clean
|
||||
self.client.select_instrumental(job_id, 'clean')
|
||||
return
|
||||
else:
|
||||
# Show suggestion, offer to open browser for verification
|
||||
self.logger.info("No backing vocals detected - recommending clean instrumental")
|
||||
# Open browser anyway so user can verify if desired
|
||||
|
||||
# Open browser to instrumental review UI
|
||||
self.open_instrumental_review_ui(job_id)
|
||||
|
||||
# Poll until selection made
|
||||
while job.status == 'awaiting_instrumental_selection':
|
||||
time.sleep(self.config.poll_interval)
|
||||
```
|
||||
|
||||
### 7. Local CLI Integration (Feature Parity)
|
||||
|
||||
The local CLI (`karaoke-gen`) uses the **same shared core module** as the backend:
|
||||
|
||||
```python
|
||||
# karaoke_gen/utils/gen_cli.py - Updated instrumental selection flow
|
||||
|
||||
def handle_instrumental_selection(self, output_dir: str, base_name: str):
|
||||
"""Handle instrumental selection using shared core module."""
|
||||
from karaoke_gen.instrumental_review import AudioAnalyzer, WaveformGenerator
|
||||
|
||||
# Paths to local files (already exist from audio separation)
|
||||
backing_vocals_path = os.path.join(output_dir, "stems",
|
||||
f"{base_name} (Backing Vocals ...).flac")
|
||||
clean_instrumental_path = os.path.join(output_dir, "stems",
|
||||
f"{base_name} (Instrumental ...).flac")
|
||||
|
||||
# Use SAME analyzer as backend
|
||||
analyzer = AudioAnalyzer(silence_threshold_db=-40.0)
|
||||
analysis = analyzer.analyze(backing_vocals_path)
|
||||
|
||||
# Generate waveform locally
|
||||
waveform_path = os.path.join(output_dir, "backing_vocals_waveform.png")
|
||||
WaveformGenerator().generate(backing_vocals_path, waveform_path,
|
||||
analysis.audible_segments)
|
||||
|
||||
if not analysis.has_audible_content:
|
||||
self.logger.info("No backing vocals detected - using clean instrumental")
|
||||
if not self.non_interactive:
|
||||
# Offer to open browser for verification
|
||||
...
|
||||
return "clean"
|
||||
|
||||
# Start local review server (serves same React UI)
|
||||
from karaoke_gen.instrumental_review.server import InstrumentalReviewServer
|
||||
server = InstrumentalReviewServer(
|
||||
output_dir=output_dir,
|
||||
base_name=base_name,
|
||||
analysis=analysis,
|
||||
waveform_path=waveform_path,
|
||||
)
|
||||
server.start_and_open_browser() # Blocking until user submits selection
|
||||
|
||||
return server.get_selection()
|
||||
```
|
||||
|
||||
### 8. Local Review Server
|
||||
|
||||
Create `karaoke_gen/instrumental_review/server.py`:
|
||||
|
||||
```python
|
||||
"""
|
||||
Lightweight local server for instrumental review.
|
||||
Serves the same React frontend used by the cloud backend.
|
||||
Similar pattern to LyricsTranscriber's ReviewServer.
|
||||
"""
|
||||
from http.server import HTTPServer, SimpleHTTPRequestHandler
|
||||
import json
|
||||
import threading
|
||||
|
||||
class InstrumentalReviewServer:
|
||||
"""
|
||||
Local HTTP server for instrumental review UI.
|
||||
|
||||
Endpoints:
|
||||
- GET /api/analysis - Returns analysis data
|
||||
- GET /api/audio/{type} - Streams audio files
|
||||
- GET /api/waveform - Returns waveform image
|
||||
- POST /api/create-custom - Creates custom instrumental
|
||||
- POST /api/select - Submits selection
|
||||
- GET /* - Serves React frontend static files
|
||||
"""
|
||||
def __init__(self, output_dir: str, base_name: str,
|
||||
analysis: AnalysisResult, waveform_path: str):
|
||||
...
|
||||
|
||||
def start_and_open_browser(self, port: int = 8765):
|
||||
"""Start server, open browser, block until selection submitted."""
|
||||
...
|
||||
|
||||
def get_selection(self) -> str:
|
||||
"""Get the user's selection after browser submission."""
|
||||
...
|
||||
```
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Test Coverage Requirements
|
||||
|
||||
All new code must meet the project's **70% minimum coverage** requirement, with critical paths targeting **>80%**:
|
||||
|
||||
| Module | Target Coverage | Test File |
|
||||
|
||||
|--------|----------------|-----------|
|
||||
|
||||
| `karaoke_gen/instrumental_review/analyzer.py` | >90% | `tests/unit/test_instrumental_analyzer.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/editor.py` | >90% | `tests/unit/test_instrumental_editor.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/waveform.py` | >80% | `tests/unit/test_instrumental_waveform.py` |
|
||||
|
||||
| `karaoke_gen/instrumental_review/server.py` | >70% | `tests/unit/test_instrumental_server.py` |
|
||||
|
||||
| `backend/services/audio_analysis_service.py` | >80% | `backend/tests/test_audio_analysis_service.py` |
|
||||
|
||||
| `backend/services/audio_editing_service.py` | >80% | `backend/tests/test_audio_editing_service.py` |
|
||||
|
||||
| `backend/api/routes/jobs.py` (new endpoints) | >80% | `backend/tests/test_instrumental_api.py` |
|
||||
|
||||
### Test Categories
|
||||
|
||||
**1. Unit Tests (Shared Core) - tests/unit/test_instrumental_*.py**
|
||||
|
||||
```python
|
||||
# tests/unit/test_instrumental_analyzer.py
|
||||
class TestAudioAnalyzer:
|
||||
def test_analyze_silent_audio_returns_no_audible_content(self):
|
||||
"""Silent audio should return has_audible_content=False."""
|
||||
|
||||
def test_analyze_audio_with_content_detects_segments(self):
|
||||
"""Audio with content should detect audible segments."""
|
||||
|
||||
def test_segment_detection_respects_threshold(self):
|
||||
"""Segments below threshold should not be detected."""
|
||||
|
||||
def test_segment_merging_for_close_segments(self):
|
||||
"""Close segments should be merged into one."""
|
||||
|
||||
def test_recommended_selection_clean_for_silent(self):
|
||||
"""Silent audio should recommend 'clean'."""
|
||||
|
||||
def test_recommended_selection_review_for_content(self):
|
||||
"""Audio with content should recommend 'review_needed'."""
|
||||
|
||||
# tests/unit/test_instrumental_editor.py
|
||||
class TestAudioEditor:
|
||||
def test_mute_single_region(self):
|
||||
"""Single mute region should be applied correctly."""
|
||||
|
||||
def test_mute_multiple_regions(self):
|
||||
"""Multiple mute regions should all be applied."""
|
||||
|
||||
def test_mute_region_boundaries(self):
|
||||
"""Mute region at start/end of audio should work."""
|
||||
|
||||
def test_output_format_is_flac(self):
|
||||
"""Output should be FLAC format."""
|
||||
|
||||
def test_combines_with_clean_instrumental(self):
|
||||
"""Result should be backing vocals mixed with clean instrumental."""
|
||||
```
|
||||
|
||||
**2. Unit Tests (Backend Services) - backend/tests/test_audio_*_service.py**
|
||||
|
||||
```python
|
||||
# backend/tests/test_audio_analysis_service.py
|
||||
class TestAudioAnalysisService:
|
||||
@pytest.fixture
|
||||
def mock_storage(self):
|
||||
"""Mock GCS storage service."""
|
||||
|
||||
def test_analyze_downloads_from_gcs(self, mock_storage):
|
||||
"""Service should download audio from GCS before analysis."""
|
||||
|
||||
def test_analyze_uploads_waveform_to_gcs(self, mock_storage):
|
||||
"""Service should upload waveform image to GCS."""
|
||||
|
||||
def test_analyze_updates_job_state_data(self, mock_storage):
|
||||
"""Service should store analysis in job.state_data."""
|
||||
```
|
||||
|
||||
**3. API Endpoint Tests - backend/tests/test_instrumental_api.py**
|
||||
|
||||
```python
|
||||
# backend/tests/test_instrumental_api.py
|
||||
class TestInstrumentalAnalysisEndpoint:
|
||||
def test_get_analysis_returns_data(self, client, mock_job):
|
||||
"""GET /instrumental-analysis should return analysis data."""
|
||||
|
||||
def test_get_analysis_requires_correct_status(self, client, mock_job):
|
||||
"""Should return 400 if job not in correct status."""
|
||||
|
||||
class TestAudioStreamEndpoint:
|
||||
def test_stream_backing_vocals(self, client, mock_job):
|
||||
"""Should stream backing vocals audio."""
|
||||
|
||||
def test_stream_clean_instrumental(self, client, mock_job):
|
||||
"""Should stream clean instrumental audio."""
|
||||
|
||||
class TestCreateCustomInstrumental:
|
||||
def test_create_with_mute_regions(self, client, mock_job):
|
||||
"""Should create custom instrumental with mute regions."""
|
||||
|
||||
def test_validates_mute_regions(self, client, mock_job):
|
||||
"""Should validate mute region boundaries."""
|
||||
```
|
||||
|
||||
**4. Integration Tests - tests/integration/test_instrumental_review.py**
|
||||
|
||||
```python
|
||||
# tests/integration/test_instrumental_review_flow.py
|
||||
class TestInstrumentalReviewFlow:
|
||||
"""End-to-end tests for instrumental review workflow."""
|
||||
|
||||
def test_full_flow_clean_selection(self):
|
||||
"""Test complete flow: analysis -> review -> select clean."""
|
||||
|
||||
def test_full_flow_with_backing_selection(self):
|
||||
"""Test complete flow: analysis -> review -> select with_backing."""
|
||||
|
||||
def test_full_flow_custom_creation(self):
|
||||
"""Test complete flow: analysis -> mute regions -> custom."""
|
||||
|
||||
def test_auto_recommend_clean_for_silent_backing(self):
|
||||
"""When no backing vocals, should recommend clean."""
|
||||
```
|
||||
|
||||
**5. CLI Feature Parity Tests - tests/unit/test_cli_feature_parity.py**
|
||||
|
||||
```python
|
||||
# tests/unit/test_cli_feature_parity.py
|
||||
class TestCLIFeatureParity:
|
||||
"""Verify local and remote CLIs have equivalent functionality."""
|
||||
|
||||
def test_both_clis_use_same_analyzer(self):
|
||||
"""Both CLIs should use AudioAnalyzer from shared module."""
|
||||
|
||||
def test_both_clis_support_custom_instrumental(self):
|
||||
"""Both CLIs should support creating custom instrumentals."""
|
||||
|
||||
def test_analysis_results_identical(self):
|
||||
"""Same audio should produce identical analysis from both CLIs."""
|
||||
```
|
||||
|
||||
### Test Data
|
||||
|
||||
Create test fixtures in `tests/data/instrumental_review/`:
|
||||
|
||||
- `silent_backing_vocals.flac` - Silent/near-silent audio (~10s)
|
||||
- `backing_vocals_with_content.flac` - Audio with clear backing vocals (~30s)
|
||||
- `backing_vocals_mixed_quality.flac` - Audio with some good/some bad sections (~30s)
|
||||
|
||||
Use pydub to programmatically generate these in `tests/conftest.py` fixtures.
|
||||
|
||||
## File Changes Summary
|
||||
|
||||
| Category | Files to Create/Modify |
|
||||
|
||||
|----------|------------------------|
|
||||
|
||||
| **Shared Core** | `karaoke_gen/instrumental_review/__init__.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/models.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/analyzer.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/editor.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/waveform.py` (new) |
|
||||
|
||||
| Shared Core | `karaoke_gen/instrumental_review/server.py` (new) |
|
||||
|
||||
| **Backend Services** | `backend/services/audio_analysis_service.py` (new) |
|
||||
|
||||
| Backend Services | `backend/services/audio_editing_service.py` (new) |
|
||||
|
||||
| **API** | `backend/api/routes/jobs.py` (modify - add endpoints) |
|
||||
|
||||
| API | `backend/api/routes/instrumental.py` (new - if endpoints get large) |
|
||||
|
||||
| **Workers** | `backend/workers/render_video_worker.py` (modify) |
|
||||
|
||||
| **Models** | `backend/models/job.py` (add analysis fields to state_data) |
|
||||
|
||||
| **Frontend** | `frontend/app/jobs/[id]/instrumental-review/page.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/waveform-display.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/audio-player-with-waveform.tsx` (new) |
|
||||
|
||||
| Frontend | `frontend/components/region-selector.tsx` (new) |
|
||||
|
||||
| **Local CLI** | `karaoke_gen/utils/gen_cli.py` (modify) |
|
||||
|
||||
| **Remote CLI** | `karaoke_gen/utils/remote_cli.py` (modify) |
|
||||
|
||||
| **Tests - Shared** | `tests/unit/test_instrumental_analyzer.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_editor.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_waveform.py` (new) |
|
||||
|
||||
| Tests - Shared | `tests/unit/test_instrumental_server.py` (new) |
|
||||
|
||||
| **Tests - Backend** | `backend/tests/test_audio_analysis_service.py` (new) |
|
||||
|
||||
| Tests - Backend | `backend/tests/test_audio_editing_service.py` (new) |
|
||||
|
||||
| Tests - Backend | `backend/tests/test_instrumental_api.py` (new) |
|
||||
|
||||
| **Tests - Parity** | `tests/unit/test_cli_feature_parity.py` (new) |
|
||||
|
||||
| **Tests - Integration** | `tests/integration/test_instrumental_review.py` (new) |
|
||||
|
||||
| **Test Data** | `tests/conftest.py` (modify - add fixtures) |
|
||||
|
||||
## Dependencies
|
||||
|
||||
Already available in `pyproject.toml`:
|
||||
|
||||
- `pydub>=0.25.1` - Audio loading and manipulation
|
||||
- `matplotlib>=3` - Waveform visualization
|
||||
- `numpy>=2` - Audio array processing
|
||||
|
||||
Frontend:
|
||||
|
||||
- React/Next.js (existing)
|
||||
- HTML5 Audio API (native)
|
||||
- Canvas or SVG for waveform (native)
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Execute in this order to enable incremental testing:
|
||||
|
||||
### Phase 1: Shared Core Module (Foundation)
|
||||
|
||||
1. Create `karaoke_gen/instrumental_review/` package with models, analyzer, editor, waveform
|
||||
2. Add comprehensive unit tests for shared core (>90% coverage)
|
||||
3. Create test audio fixtures in `tests/conftest.py`
|
||||
|
||||
### Phase 2: Backend Integration
|
||||
|
||||
4. Create backend service wrappers with GCS integration
|
||||
5. Add backend service unit tests with mocked GCS
|
||||
6. Add new API endpoints to `backend/api/routes/jobs.py`
|
||||
7. Add API endpoint tests
|
||||
8. Integrate analysis into `render_video_worker.py`
|
||||
9. Update render worker tests
|
||||
|
||||
### Phase 3: Frontend
|
||||
|
||||
10. Create React components (WaveformDisplay, AudioPlayer, RegionSelector)
|
||||
11. Create instrumental review page
|
||||
12. Test frontend manually with backend
|
||||
|
||||
### Phase 4: CLI Integration
|
||||
|
||||
13. Update `remote_cli.py` to use new analysis API and open browser
|
||||
14. Create local review server `karaoke_gen/instrumental_review/server.py`
|
||||
15. Update `gen_cli.py` to use shared core and local server
|
||||
16. Add CLI feature parity tests
|
||||
|
||||
### Phase 5: Integration Testing
|
||||
|
||||
17. Add end-to-end integration tests
|
||||
18. Manual testing of full workflow (local and remote)
|
||||
|
||||
### Verification Checklist
|
||||
|
||||
- [ ] All unit tests pass: `pytest tests/unit/ -v`
|
||||
- [ ] All backend tests pass: `pytest backend/tests/ -v`
|
||||
- [ ] Coverage meets 70% minimum: `pytest --cov=karaoke_gen --cov-fail-under=70`
|
||||
- [ ] Local CLI works: `karaoke-gen "Artist" "Title"` (reaches instrumental selection)
|
||||
- [ ] Remote CLI works: `karaoke-gen-remote ./audio.flac "Artist" "Title"`
|
||||
- [ ] Both CLIs produce identical analysis for same audio
|
||||
- [ ] Custom instrumental creation works in both modes
|
||||
- [ ] Version bumped in `pyproject.toml`
|
||||
@@ -1,217 +0,0 @@
|
||||
---
|
||||
name: MidiCo-Style Synchronizer
|
||||
overview: Build a new MidiCo-style Synchronizer component for efficient re-syncing of lyrics timing, particularly for fixing progressive drift in the second half of songs. This involves refactoring the "Edit All" entry point and creating a high-performance timeline-based sync interface.
|
||||
todos:
|
||||
- id: mode-selection-modal
|
||||
content: Create ModeSelectionModal for 'Replace All' vs 'Re-sync Existing' choice
|
||||
status: completed
|
||||
- id: timeline-canvas
|
||||
content: Build canvas-based TimelineCanvas component for high-performance word block rendering
|
||||
status: completed
|
||||
- id: upcoming-words-bar
|
||||
content: Create UpcomingWordsBar component showing next unsynced words in fixed position
|
||||
status: completed
|
||||
- id: sync-controls
|
||||
content: Build SyncControls with Start Sync, Clear Sync, Unsync from Cursor, Delete Selected buttons
|
||||
status: completed
|
||||
- id: lyrics-synchronizer
|
||||
content: Create main LyricsSynchronizer component integrating all sub-components
|
||||
status: completed
|
||||
- id: unsync-from-cursor
|
||||
content: Add 'unsync from cursor' functionality to useManualSync hook
|
||||
status: completed
|
||||
- id: integrate-modal
|
||||
content: Integrate new synchronizer into ReplaceAllLyricsModal with mode selection
|
||||
status: completed
|
||||
- id: word-selection
|
||||
content: Implement single-click and drag-select for word blocks
|
||||
status: completed
|
||||
- id: zoom-slider
|
||||
content: Add smooth zoom slider with 4.5s-24s range
|
||||
status: completed
|
||||
- id: performance-test
|
||||
content: Test performance with long songs and optimize as needed
|
||||
status: completed
|
||||
---
|
||||
|
||||
# MidiCo-Style Synchronizer Implementation
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
The existing implementation in [`ReplaceAllLyricsModal.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/ReplaceAllLyricsModal.tsx) has two phases:
|
||||
|
||||
1. **Paste Phase** - User pastes new lyrics text
|
||||
2. **Sync Phase** - Manual sync using spacebar with timeline view
|
||||
|
||||
Key existing components:
|
||||
|
||||
- [`useManualSync.ts`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/hooks/useManualSync.ts) - Spacebar handling, tap/hold detection
|
||||
- [`EditTimelineSection.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/EditTimelineSection.tsx) - Timeline controls with zoom/scroll
|
||||
- [`TimelineEditor.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/TimelineEditor.tsx) - Timeline rendering
|
||||
|
||||
**Problems with current implementation:**
|
||||
|
||||
1. Forces full lyrics replacement - no option to keep existing sync data
|
||||
2. Performance issues (laggy, slow rendering)
|
||||
3. Poor UX for seeing upcoming words while syncing
|
||||
4. No "unsync from cursor" feature for fixing drift
|
||||
5. No two-level word block layout for overlapping segments
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Entry [Entry Point]
|
||||
EditAll[Edit All Button]
|
||||
EditAll --> ChoiceModal[Mode Selection Modal]
|
||||
ChoiceModal -->|Replace All| PastePhase[Paste New Lyrics]
|
||||
ChoiceModal -->|Re-sync Existing| SyncView[Synchronizer View]
|
||||
PastePhase --> SyncView
|
||||
end
|
||||
|
||||
subgraph Sync [Synchronizer Component]
|
||||
SyncView --> Timeline[Timeline Canvas]
|
||||
SyncView --> Controls[Control Buttons]
|
||||
SyncView --> UpcomingWords[Upcoming Words Display]
|
||||
Timeline --> WordBlocks[Word Blocks Renderer]
|
||||
Timeline --> Playhead[Playhead Marker]
|
||||
Timeline --> TimeBar[Time Bar]
|
||||
end
|
||||
|
||||
subgraph Audio [Audio Integration]
|
||||
AudioPlayer[Audio Player]
|
||||
AudioPlayer <--> SyncView
|
||||
end
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Entry Point Refactor
|
||||
|
||||
Modify the "Edit All" button flow to show a choice modal:
|
||||
|
||||
```typescript
|
||||
// New ModeSelectionModal component
|
||||
interface ModeSelectionModalProps {
|
||||
open: boolean
|
||||
onClose: () => void
|
||||
onSelectReplace: () => void // Goes to paste phase
|
||||
onSelectResync: () => void // Goes directly to sync view
|
||||
}
|
||||
```
|
||||
|
||||
- **Replace All Lyrics**: Current flow (paste box -> sync)
|
||||
- **Re-sync Existing Lyrics**: Load current segments directly into sync view
|
||||
|
||||
### Phase 2: New Synchronizer Component
|
||||
|
||||
Create a new high-performance `LyricsSynchronizer.tsx` component inspired by MidiCo:
|
||||
|
||||
#### Core Features
|
||||
|
||||
1. **Timeline Canvas** (HTML5 Canvas for performance)
|
||||
|
||||
- Time bar at top (MM:SS format with decisecond ticks)
|
||||
- Playhead with arrow indicator + vertical line
|
||||
- Red word blocks with text labels
|
||||
- Two-level word layout for overlapping segments
|
||||
- Zoom slider (4.5s to 24s visible range)
|
||||
- Horizontal scroll with smooth panning
|
||||
|
||||
2. **Upcoming Words Display** (dual location)
|
||||
|
||||
- Fixed position above timeline: next words as red/white blocks
|
||||
- On timeline: words appear to right of playhead during sync
|
||||
|
||||
3. **Control Buttons**
|
||||
|
||||
- **Start Sync** - Begin sync mode from first unsynced word
|
||||
- **Clear Sync** - Reset all word timings to null
|
||||
- **Edit Lyrics** - Modal to edit lyrics text (warns about timing loss)
|
||||
- **Unsync from Cursor** - Critical: reset all words after playhead position
|
||||
- **Split/Edit Word** - Edit single selected word
|
||||
- **Delete Selected** - Remove selected word blocks
|
||||
|
||||
4. **Word Selection**
|
||||
|
||||
- Single click: select one word (white border)
|
||||
- Click-drag on background: selection rectangle for multi-select
|
||||
- Click on time bar: set playhead position
|
||||
|
||||
5. **Spacebar Sync Logic** (reuse/enhance `useManualSync`)
|
||||
|
||||
- Tap: set start time, default 500ms duration
|
||||
- Hold: set both start and end times
|
||||
- Handle previous word end time calculation
|
||||
|
||||
### Phase 3: Performance Optimization
|
||||
|
||||
The current `TimelineEditor` uses DOM elements which causes lag. New approach:
|
||||
|
||||
1. **Canvas-based rendering** for word blocks and timeline
|
||||
|
||||
- Single canvas element, no DOM nodes per word
|
||||
- RequestAnimationFrame for smooth playhead updates
|
||||
- Only re-render changed regions
|
||||
|
||||
2. **Virtual scrolling** for very long songs
|
||||
|
||||
- Only render visible word blocks
|
||||
- Cache off-screen block positions
|
||||
|
||||
3. **Debounced state updates**
|
||||
|
||||
- Batch word timing updates
|
||||
- Throttle zoom/scroll handlers
|
||||
|
||||
### Phase 4: Apply/Cancel Flow
|
||||
|
||||
Changes are held in local state until user clicks Apply:
|
||||
|
||||
```typescript
|
||||
interface SynchronizerState {
|
||||
words: Word[] // Working copy of words
|
||||
segments: LyricsSegment[] // Working copy of segments
|
||||
isDirty: boolean // Has unsaved changes
|
||||
}
|
||||
```
|
||||
|
||||
- **Apply**: Commit changes to parent component
|
||||
- **Cancel**: Discard changes, close modal
|
||||
|
||||
## File Changes
|
||||
|
||||
### New Files
|
||||
|
||||
- `src/components/LyricsSynchronizer.tsx` - Main synchronizer component
|
||||
- `src/components/LyricsSynchronizer/TimelineCanvas.tsx` - Canvas-based timeline
|
||||
- `src/components/LyricsSynchronizer/UpcomingWordsBar.tsx` - Fixed upcoming words display
|
||||
- `src/components/LyricsSynchronizer/SyncControls.tsx` - Control buttons
|
||||
- `src/components/ModeSelectionModal.tsx` - Entry point choice modal
|
||||
|
||||
### Modified Files
|
||||
|
||||
- [`ReplaceAllLyricsModal.tsx`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/components/ReplaceAllLyricsModal.tsx) - Add mode selection, integrate new synchronizer
|
||||
- [`useManualSync.ts`](lyrics_transcriber_temp/lyrics_transcriber/frontend/src/hooks/useManualSync.ts) - Add "unsync from cursor" functionality
|
||||
|
||||
## Key UX Details from MidiCo Reference
|
||||
|
||||
Per the documentation at [`MIDICO_SYNCHRONIZER_REFERENCE.md`](lyrics_transcriber_temp/docs/MIDICO_SYNCHRONIZER_REFERENCE.md):
|
||||
|
||||
1. **Zoom range**: 4.5s (max zoom) to 24s (min zoom) visible
|
||||
2. **Timeline click behavior**:
|
||||
|
||||
- Time bar click = set playhead (no playback change)
|
||||
- Background click = nothing (prevent accidental deselection)
|
||||
- Word block click = select
|
||||
|
||||
3. **Two-level word blocks**: Adjacent/overlapping words display on alternating vertical levels
|
||||
4. **Playhead**: Small arrow in time bar + white vertical line spanning timeline height
|
||||
5. **Selection rectangle**: Click-drag on background creates selection box
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
1. Manual testing with the provided URL: `http://localhost:8000/?baseApiUrl=...`
|
||||
2. Test sync accuracy with songs that have known drift issues
|
||||
3. Performance testing with 400+ word songs
|
||||
4. Test "unsync from cursor" workflow for fixing drift
|
||||
@@ -1,83 +0,0 @@
|
||||
# .dockerignore - Reduce Docker build context size
|
||||
# This is critical for fast builds as it reduces upload time and cache invalidation
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend (not needed for backend image)
|
||||
node_modules/
|
||||
frontend/
|
||||
frontend-react/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python virtual environments (use Docker's Python instead)
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
backend/venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing (not needed in production image)
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
tests/
|
||||
backend/tests/
|
||||
|
||||
# Documentation (not needed in Docker image)
|
||||
docs/
|
||||
*.md
|
||||
!README.md
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# CI/CD configs
|
||||
.github/
|
||||
|
||||
# Cloud Build configs (keep for reference but not in image)
|
||||
cloudbuild*.yaml
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
|
||||
# Large test/sample files
|
||||
tests/data/
|
||||
*.mp3
|
||||
*.wav
|
||||
*.mp4
|
||||
*.mkv
|
||||
*.flac
|
||||
@@ -1,73 +0,0 @@
|
||||
# .gcloudignore - Exclude files from Cloud Build uploads
|
||||
# This dramatically reduces upload time (from 218MB to ~5-10MB)
|
||||
|
||||
# Git
|
||||
.git/
|
||||
.gitignore
|
||||
.gitmodules
|
||||
|
||||
# Node.js / Frontend
|
||||
node_modules/
|
||||
frontend-react/node_modules/
|
||||
frontend-react/dist/
|
||||
frontend-react/.vite/
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
.venv/
|
||||
venv/
|
||||
ENV/
|
||||
env/
|
||||
*.egg-info/
|
||||
.eggs/
|
||||
dist/
|
||||
build/
|
||||
*.egg
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
.DS_Store
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
coverage.xml
|
||||
*.cover
|
||||
|
||||
# Documentation (not needed for Docker build)
|
||||
docs/
|
||||
|
||||
# Local development files
|
||||
lyrics_transcriber_local/
|
||||
*.local
|
||||
.env
|
||||
.env.*
|
||||
|
||||
# Large test files
|
||||
tests/data/*.mp3
|
||||
tests/data/*.wav
|
||||
tests/data/*.mp4
|
||||
tests/data/*.mkv
|
||||
|
||||
# Infrastructure (not needed for Docker build)
|
||||
infrastructure/
|
||||
|
||||
# Misc
|
||||
*.bak
|
||||
*.tmp
|
||||
Thumbs.db
|
||||
@@ -1,128 +0,0 @@
|
||||
# GitHub Actions CI/CD
|
||||
|
||||
This directory contains GitHub Actions workflows for automated testing and deployment.
|
||||
|
||||
## Workflows
|
||||
|
||||
### `test.yml` - Automated Testing & Deployment (CI/CD)
|
||||
|
||||
**Triggers:**
|
||||
- Push to `main` or `replace-modal-with-google-cloud` branches
|
||||
- Pull requests to `main`
|
||||
|
||||
**Jobs:**
|
||||
|
||||
#### Testing Phase (runs in parallel)
|
||||
|
||||
1. **Unit Tests** (~30s)
|
||||
- Runs 62 unit tests with mocked dependencies
|
||||
- Fast validation of business logic
|
||||
- Uses Python 3.12 with pip caching
|
||||
|
||||
2. **Emulator Integration Tests** (~2min)
|
||||
- Runs 11 integration tests against local GCP emulators
|
||||
- Tests real Firestore and GCS interactions
|
||||
- Requires:
|
||||
- Java 21 (for Firestore emulator)
|
||||
- Google Cloud SDK (for Firestore emulator)
|
||||
- Docker (for GCS emulator via fake-gcs-server)
|
||||
- Uses the same `scripts/run-emulator-tests.sh` as local development
|
||||
|
||||
3. **Code Quality** (~20s)
|
||||
- Basic Python syntax validation
|
||||
- Import checks
|
||||
- Optional: flake8, black, isort
|
||||
|
||||
#### Deployment Phase (runs after all tests pass)
|
||||
|
||||
4. **Deploy to Cloud Run** (~3-4min)
|
||||
- **Only runs on push to `replace-modal-with-google-cloud` branch** (not on PRs)
|
||||
- Builds Docker image and pushes to Artifact Registry
|
||||
- Deploys to Cloud Run with proper configuration
|
||||
- Verifies deployment health
|
||||
- **Requires GitHub secrets:**
|
||||
- `GCP_SA_KEY` - Service account key JSON
|
||||
- `ADMIN_TOKENS` - Admin authentication tokens
|
||||
|
||||
**Total Runtime:**
|
||||
- **Testing only** (PRs): ~3 minutes
|
||||
- **Testing + Deployment** (push to branch): ~7 minutes
|
||||
|
||||
## Local vs CI
|
||||
|
||||
The CI runs the exact same tests as local development:
|
||||
|
||||
```bash
|
||||
# Local development
|
||||
./scripts/run-tests.sh # Unit tests (same as CI job 1)
|
||||
./scripts/run-emulator-tests.sh # Emulator tests (same as CI job 2)
|
||||
python backend/validate.py # Validation (same as CI job 3)
|
||||
|
||||
# CI automatically runs all three on every push
|
||||
```
|
||||
|
||||
## Viewing Results
|
||||
|
||||
1. Go to the [Actions tab](../../actions) in GitHub
|
||||
2. Click on the latest workflow run
|
||||
3. View job logs and test results
|
||||
4. Download test artifacts (coverage reports) if needed
|
||||
|
||||
## Adding New Tests
|
||||
|
||||
Tests are automatically discovered by pytest. To add new tests:
|
||||
|
||||
1. **Unit tests:** Add to `backend/tests/test_*.py`
|
||||
2. **Emulator tests:** Add to `backend/tests/emulator/test_*.py`
|
||||
3. **Cloud tests:** Add to `backend/tests/test_api_integration.py` (runs manually)
|
||||
|
||||
The CI will automatically pick them up!
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Emulator Tests Fail in CI
|
||||
|
||||
**Common issues:**
|
||||
- Emulator startup timeout → Increase wait time in `scripts/start-emulators.sh`
|
||||
- Docker not available → Check GitHub Actions runner compatibility
|
||||
- Port conflicts → Emulators use ports 8080 (Firestore) and 4443 (GCS)
|
||||
|
||||
### CI is Slow
|
||||
|
||||
**Optimizations:**
|
||||
- ✅ Python pip caching enabled
|
||||
- ✅ Jobs run in parallel
|
||||
- ✅ Docker image pre-pulled
|
||||
- Future: Cache gcloud SDK installation
|
||||
|
||||
### Tests Pass Locally But Fail in CI
|
||||
|
||||
**Debug steps:**
|
||||
1. Check Python version matches (3.12)
|
||||
2. Check environment variables are set correctly
|
||||
3. Review CI logs for missing dependencies
|
||||
4. Run tests in a clean venv locally
|
||||
|
||||
## Setting Up Continuous Deployment
|
||||
|
||||
The CD pipeline is configured but requires GitHub secrets to be set. See the [CD Setup Guide](../docs/03-deployment/CD-SETUP.md) for detailed instructions.
|
||||
|
||||
**Quick setup:**
|
||||
|
||||
1. Create GCP service account with deployment permissions
|
||||
2. Download service account key JSON
|
||||
3. Add GitHub secrets:
|
||||
- `GCP_SA_KEY` - Full service account key JSON content
|
||||
- `ADMIN_TOKENS` - Comma-separated admin tokens
|
||||
|
||||
Once configured, every push to `replace-modal-with-google-cloud` will automatically deploy to Cloud Run after all tests pass.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Add test coverage reporting (coveralls, codecov)
|
||||
- [x] ✅ Add deployment workflow (deploy to Cloud Run after tests pass)
|
||||
- [ ] Add frontend tests (when frontend is built)
|
||||
- [ ] Add performance benchmarking
|
||||
- [ ] Add security scanning (dependabot, snyk)
|
||||
- [ ] Migrate to Workload Identity Federation (more secure than service account keys)
|
||||
|
||||
@@ -1,28 +0,0 @@
|
||||
2384d2383
|
||||
< triton = ">=2.0.0,<3"
|
||||
4185d4183
|
||||
< triton = {version = "2.3.1", markers = "platform_system == \"Linux\" and platform_machine == \"x86_64\" and python_version < \"3.12\""}
|
||||
4318,4340d4315
|
||||
< name = "triton"
|
||||
< version = "2.3.1"
|
||||
< description = "A language and compiler for custom Deep Learning operations"
|
||||
< optional = false
|
||||
< python-versions = "*"
|
||||
< files = [
|
||||
< {file = "triton-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3c84595cbe5e546b1b290d2a58b1494df5a2ef066dd890655e5b8a8a92205c33"},
|
||||
< {file = "triton-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c9d64ae33bcb3a7a18081e3a746e8cf87ca8623ca13d2c362413ce7a486f893e"},
|
||||
< {file = "triton-2.3.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:eaf80e8761a9e3498aa92e7bf83a085b31959c61f5e8ac14eedd018df6fccd10"},
|
||||
< {file = "triton-2.3.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b13bf35a2b659af7159bf78e92798dc62d877aa991de723937329e2d382f1991"},
|
||||
< {file = "triton-2.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:63381e35ded3304704ea867ffde3b7cfc42c16a55b3062d41e017ef510433d66"},
|
||||
< {file = "triton-2.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1d968264523c7a07911c8fb51b4e0d1b920204dae71491b1fe7b01b62a31e124"},
|
||||
< ]
|
||||
<
|
||||
< [package.dependencies]
|
||||
< filelock = "*"
|
||||
<
|
||||
< [package.extras]
|
||||
< build = ["cmake (>=3.20)", "lit"]
|
||||
< tests = ["autopep8", "flake8", "isort", "numpy", "pytest", "scipy (>=1.7.1)", "torch"]
|
||||
< tutorials = ["matplotlib", "pandas", "tabulate", "torch"]
|
||||
<
|
||||
< [[package]]
|
||||
@@ -1,115 +0,0 @@
|
||||
# Build Encoding Worker GCE Image
|
||||
#
|
||||
# Rebuilds the custom Packer image when provisioning changes.
|
||||
# The image includes Python 3.13, FFmpeg, fonts, and dependencies
|
||||
# pre-installed to reduce VM startup time from ~10 min to ~30 sec.
|
||||
#
|
||||
# Triggers:
|
||||
# - Push to main with changes to infrastructure/packer/**
|
||||
# - Manual dispatch (workflow_dispatch)
|
||||
#
|
||||
# After image build, the encoding-worker VM will use the new image
|
||||
# on next recreation (e.g., via pulumi up or manual restart).
|
||||
|
||||
name: Build Encoding Worker Image
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'infrastructure/packer/**'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
python_version:
|
||||
description: 'Python version to install'
|
||||
required: false
|
||||
default: '3.13.1'
|
||||
|
||||
concurrency:
|
||||
group: build-encoding-worker-image
|
||||
cancel-in-progress: false # Don't cancel in-progress builds
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
name: Build Packer Image
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
id-token: write # Required for Workload Identity Federation
|
||||
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Authenticate to Google Cloud (Workload Identity)
|
||||
uses: google-github-actions/auth@v2
|
||||
with:
|
||||
workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
|
||||
service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}
|
||||
|
||||
- name: Set up Cloud SDK
|
||||
uses: google-github-actions/setup-gcloud@v2
|
||||
with:
|
||||
version: "latest"
|
||||
|
||||
- name: Install Packer
|
||||
run: |
|
||||
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
||||
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
|
||||
sudo apt-get update && sudo apt-get install -y packer
|
||||
|
||||
- name: Initialize Packer
|
||||
working-directory: infrastructure/packer
|
||||
run: packer init encoding-worker.pkr.hcl
|
||||
|
||||
- name: Validate Packer Template
|
||||
working-directory: infrastructure/packer
|
||||
run: packer validate encoding-worker.pkr.hcl
|
||||
|
||||
- name: Build Image
|
||||
working-directory: infrastructure/packer
|
||||
env:
|
||||
PYTHON_VERSION: ${{ github.event.inputs.python_version || '3.13.1' }}
|
||||
run: |
|
||||
echo "Building encoding worker image with Python ${PYTHON_VERSION}..."
|
||||
packer build \
|
||||
-var "python_version=${PYTHON_VERSION}" \
|
||||
encoding-worker.pkr.hcl
|
||||
|
||||
- name: Get Latest Image Info
|
||||
id: image-info
|
||||
run: |
|
||||
IMAGE_NAME=$(gcloud compute images list \
|
||||
--project=nomadkaraoke \
|
||||
--filter="family:encoding-worker" \
|
||||
--sort-by="~creationTimestamp" \
|
||||
--limit=1 \
|
||||
--format="value(name)")
|
||||
echo "image_name=${IMAGE_NAME}" >> $GITHUB_OUTPUT
|
||||
echo "Built image: ${IMAGE_NAME}"
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Encoding Worker Image Built" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Image:** \`${{ steps.image-info.outputs.image_name }}\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Family:** \`encoding-worker\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Project:** \`nomadkaraoke\`" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "The encoding-worker VM will use this image on next recreation." >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "To force VM recreation now:" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
|
||||
echo "gcloud compute instances delete encoding-worker --zone=us-central1-a --project=nomadkaraoke" >> $GITHUB_STEP_SUMMARY
|
||||
echo "# Then run pulumi up or wait for next infrastructure deploy" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
notify-on-failure:
|
||||
name: Notify on Failure
|
||||
runs-on: ubuntu-latest
|
||||
needs: build-image
|
||||
if: failure()
|
||||
steps:
|
||||
- name: Log Failure
|
||||
run: |
|
||||
echo "::error::Encoding worker image build failed. Check the build-image job logs."
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,174 +0,0 @@
|
||||
name: E2E Happy Path Test
|
||||
|
||||
# Comprehensive end-to-end test that validates the full karaoke generation journey
|
||||
# using ONLY browser interactions (no API shortcuts)
|
||||
#
|
||||
# See docs/E2E-HAPPY-PATH-TEST-SPEC.md for full requirements
|
||||
|
||||
on:
|
||||
# Daily run at 6 AM UTC
|
||||
schedule:
|
||||
- cron: '0 6 * * *'
|
||||
|
||||
# Manual trigger with optional parameters
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
debug_mode:
|
||||
description: 'Enable debug mode (slower, more screenshots)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
browser:
|
||||
description: 'Browser to test with'
|
||||
required: false
|
||||
default: 'chromium'
|
||||
type: choice
|
||||
options:
|
||||
- chromium
|
||||
- firefox
|
||||
- webkit
|
||||
use_test_token:
|
||||
description: 'Skip signup and use E2E_TEST_TOKEN (for iterations)'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: boolean
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
e2e-happy-path:
|
||||
name: "Full Happy Path (Real User)"
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90 # 1.5 hours max (test itself is 60 min)
|
||||
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: '20'
|
||||
cache: 'npm'
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install dependencies
|
||||
working-directory: frontend
|
||||
run: npm ci --legacy-peer-deps
|
||||
|
||||
- name: Install Playwright browsers
|
||||
working-directory: frontend
|
||||
run: |
|
||||
BROWSER=${{ inputs.browser || 'chromium' }}
|
||||
npx playwright install --with-deps $BROWSER
|
||||
|
||||
- name: Create test-results directory
|
||||
working-directory: frontend
|
||||
run: mkdir -p test-results
|
||||
|
||||
- name: Run E2E Happy Path Test
|
||||
working-directory: frontend
|
||||
env:
|
||||
TESTMAIL_API_KEY: ${{ secrets.TESTMAIL_API_KEY }}
|
||||
TESTMAIL_NAMESPACE: ${{ secrets.TESTMAIL_NAMESPACE }}
|
||||
E2E_TEST_TOKEN: ${{ inputs.use_test_token && secrets.E2E_TEST_TOKEN || '' }}
|
||||
E2E_ADMIN_TOKEN: ${{ secrets.E2E_ADMIN_TOKEN }}
|
||||
E2E_BYPASS_KEY: ${{ secrets.E2E_BYPASS_KEY }}
|
||||
DEBUG_MODE: ${{ inputs.debug_mode || 'false' }}
|
||||
run: |
|
||||
echo "Starting E2E Happy Path Test"
|
||||
echo "Browser: ${{ inputs.browser || 'chromium' }}"
|
||||
echo "Debug mode: ${{ inputs.debug_mode || 'false' }}"
|
||||
echo "Use test token: ${{ inputs.use_test_token || 'false' }}"
|
||||
echo "Time: $(date)"
|
||||
echo ""
|
||||
|
||||
# Run the real user happy path test
|
||||
npx playwright test happy-path-real-user.spec.ts \
|
||||
--config=playwright.production.config.ts \
|
||||
--project=${{ inputs.browser || 'chromium' }} \
|
||||
--reporter=list \
|
||||
--timeout=3600000 \
|
||||
2>&1 | tee test-output.log
|
||||
continue-on-error: true
|
||||
|
||||
- name: Check test result
|
||||
id: test_result
|
||||
working-directory: frontend
|
||||
run: |
|
||||
if grep -q "passed" test-output.log && ! grep -q "failed" test-output.log; then
|
||||
echo "result=success" >> $GITHUB_OUTPUT
|
||||
echo "Test PASSED"
|
||||
else
|
||||
echo "result=failure" >> $GITHUB_OUTPUT
|
||||
echo "Test FAILED - see artifacts for details"
|
||||
fi
|
||||
|
||||
- name: Upload test artifacts
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-results-${{ github.run_number }}
|
||||
path: |
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/test-output.log
|
||||
retention-days: 30
|
||||
|
||||
- name: Upload video recordings
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: e2e-happy-path-videos-${{ github.run_number }}
|
||||
path: frontend/test-results/**/*.webm
|
||||
retention-days: 30
|
||||
if-no-files-found: ignore
|
||||
|
||||
- name: Create summary
|
||||
if: always()
|
||||
run: |
|
||||
echo "## E2E Happy Path Test Results" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Run | #${{ github.run_number }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Trigger | ${{ github.event_name }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Browser | ${{ inputs.browser || 'chromium' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "| Result | ${{ steps.test_result.outputs.result || 'unknown' }} |" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
if [ -f frontend/test-output.log ]; then
|
||||
echo "### Test Output (last 100 lines)" >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
tail -100 frontend/test-output.log >> $GITHUB_STEP_SUMMARY
|
||||
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
|
||||
fi
|
||||
|
||||
- name: Send email notification
|
||||
if: always() && github.event_name == 'schedule'
|
||||
uses: dawidd6/action-send-mail@v3
|
||||
with:
|
||||
server_address: smtp.sendgrid.net
|
||||
server_port: 587
|
||||
username: apikey
|
||||
password: ${{ secrets.SENDGRID_API_KEY }}
|
||||
subject: "E2E Happy Path Test ${{ steps.test_result.outputs.result == 'success' && 'PASSED' || 'FAILED' }} - Run #${{ github.run_number }}"
|
||||
to: andrew@nomadkaraoke.com
|
||||
from: "Karaoke Gen CI <noreply@nomadkaraoke.com>"
|
||||
body: |
|
||||
E2E Happy Path Test Results
|
||||
===========================
|
||||
|
||||
Status: ${{ steps.test_result.outputs.result == 'success' && 'PASSED ✅' || 'FAILED ❌' }}
|
||||
Run: #${{ github.run_number }}
|
||||
Trigger: ${{ github.event_name }}
|
||||
Time: ${{ github.event.repository.updated_at }}
|
||||
|
||||
View details: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||||
|
||||
This is an automated notification from the daily E2E happy path test.
|
||||
|
||||
- name: Fail if test failed
|
||||
if: steps.test_result.outputs.result == 'failure'
|
||||
run: exit 1
|
||||
@@ -0,0 +1,19 @@
|
||||
name: Auto-publish
|
||||
|
||||
on: [push, workflow_dispatch]
|
||||
|
||||
jobs:
|
||||
# Auto-publish when version is increased
|
||||
publish-pypi:
|
||||
# Only publish on `main` branch
|
||||
if: github.ref == 'refs/heads/main'
|
||||
runs-on: ubuntu-latest
|
||||
permissions: # Don't forget permissions
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: etils-actions/pypi-auto-publish@v1
|
||||
with:
|
||||
pypi-token: ${{ secrets.PYPI_API_TOKEN }}
|
||||
gh-token: ${{ secrets.GITHUB_TOKEN }}
|
||||
parse-changelog: false
|
||||
+1
-62
@@ -3,65 +3,12 @@
|
||||
.vscode
|
||||
*.csv
|
||||
*.mp3
|
||||
ABBA*
|
||||
|
||||
# Andrew functional adds
|
||||
/audio/
|
||||
/lyrics/
|
||||
/karaoke/
|
||||
tests/outputs
|
||||
/input
|
||||
/output/
|
||||
|
||||
# Cloud Infra
|
||||
.emulator-logs
|
||||
|
||||
# Next.js / React Frontend (./frontend)
|
||||
node_modules/
|
||||
frontend/node_modules/
|
||||
frontend/.next/
|
||||
frontend/out/
|
||||
frontend/.swc/
|
||||
frontend/.turbo/
|
||||
frontend/build/
|
||||
frontend/dist/
|
||||
|
||||
# Next.js specific
|
||||
frontend/*.tsbuildinfo
|
||||
frontend/next-env.d.ts
|
||||
|
||||
# Environment files (local overrides)
|
||||
frontend/.env*.local
|
||||
|
||||
# Vercel
|
||||
frontend/.vercel/
|
||||
|
||||
# Package manager logs
|
||||
frontend/npm-debug.log*
|
||||
frontend/yarn-debug.log*
|
||||
frontend/yarn-error.log*
|
||||
frontend/pnpm-debug.log*
|
||||
frontend/.pnpm-debug.log*
|
||||
|
||||
# Testing
|
||||
frontend/coverage/
|
||||
frontend/test-results/
|
||||
frontend/playwright-report/
|
||||
frontend/playwright/.cache/
|
||||
frontend/e2e/screenshots/
|
||||
|
||||
# TypeScript
|
||||
frontend/*.tsbuildinfo
|
||||
|
||||
# ESLint
|
||||
frontend/.eslintcache
|
||||
|
||||
# Storybook
|
||||
frontend/storybook-static/
|
||||
|
||||
# PWA auto-generated files (our sw.js is hand-written and should be tracked)
|
||||
frontend/**/public/workbox-*.js
|
||||
frontend/**/public/swe-worker-*.js
|
||||
/tests/outputs
|
||||
|
||||
|
||||
# Byte-compiled / optimized / DLL files
|
||||
@@ -80,12 +27,8 @@ dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
# Python lib directories (not frontend)
|
||||
lib/
|
||||
lib64/
|
||||
# But allow frontend lib directories
|
||||
!frontend/lib/
|
||||
!buy-site/lib/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
@@ -228,7 +171,3 @@ cython_debug/
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
flacfetch
|
||||
|
||||
# Test output directories
|
||||
Rick Astley*/
|
||||
@@ -1,50 +0,0 @@
|
||||
# [PROJECT_NAME] Constitution
|
||||
<!-- Example: Spec Constitution, TaskFlow Constitution, etc. -->
|
||||
|
||||
## Core Principles
|
||||
|
||||
### [PRINCIPLE_1_NAME]
|
||||
<!-- Example: I. Library-First -->
|
||||
[PRINCIPLE_1_DESCRIPTION]
|
||||
<!-- Example: Every feature starts as a standalone library; Libraries must be self-contained, independently testable, documented; Clear purpose required - no organizational-only libraries -->
|
||||
|
||||
### [PRINCIPLE_2_NAME]
|
||||
<!-- Example: II. CLI Interface -->
|
||||
[PRINCIPLE_2_DESCRIPTION]
|
||||
<!-- Example: Every library exposes functionality via CLI; Text in/out protocol: stdin/args → stdout, errors → stderr; Support JSON + human-readable formats -->
|
||||
|
||||
### [PRINCIPLE_3_NAME]
|
||||
<!-- Example: III. Test-First (NON-NEGOTIABLE) -->
|
||||
[PRINCIPLE_3_DESCRIPTION]
|
||||
<!-- Example: TDD mandatory: Tests written → User approved → Tests fail → Then implement; Red-Green-Refactor cycle strictly enforced -->
|
||||
|
||||
### [PRINCIPLE_4_NAME]
|
||||
<!-- Example: IV. Integration Testing -->
|
||||
[PRINCIPLE_4_DESCRIPTION]
|
||||
<!-- Example: Focus areas requiring integration tests: New library contract tests, Contract changes, Inter-service communication, Shared schemas -->
|
||||
|
||||
### [PRINCIPLE_5_NAME]
|
||||
<!-- Example: V. Observability, VI. Versioning & Breaking Changes, VII. Simplicity -->
|
||||
[PRINCIPLE_5_DESCRIPTION]
|
||||
<!-- Example: Text I/O ensures debuggability; Structured logging required; Or: MAJOR.MINOR.BUILD format; Or: Start simple, YAGNI principles -->
|
||||
|
||||
## [SECTION_2_NAME]
|
||||
<!-- Example: Additional Constraints, Security Requirements, Performance Standards, etc. -->
|
||||
|
||||
[SECTION_2_CONTENT]
|
||||
<!-- Example: Technology stack requirements, compliance standards, deployment policies, etc. -->
|
||||
|
||||
## [SECTION_3_NAME]
|
||||
<!-- Example: Development Workflow, Review Process, Quality Gates, etc. -->
|
||||
|
||||
[SECTION_3_CONTENT]
|
||||
<!-- Example: Code review requirements, testing gates, deployment approval process, etc. -->
|
||||
|
||||
## Governance
|
||||
<!-- Example: Constitution supersedes all other practices; Amendments require documentation, approval, migration plan -->
|
||||
|
||||
[GOVERNANCE_RULES]
|
||||
<!-- Example: All PRs/reviews must verify compliance; Complexity must be justified; Use [GUIDANCE_FILE] for runtime development guidance -->
|
||||
|
||||
**Version**: [CONSTITUTION_VERSION] | **Ratified**: [RATIFICATION_DATE] | **Last Amended**: [LAST_AMENDED_DATE]
|
||||
<!-- Example: Version: 2.1.1 | Ratified: 2025-06-13 | Last Amended: 2025-07-16 -->
|
||||
@@ -1,166 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Consolidated prerequisite checking script
|
||||
#
|
||||
# This script provides unified prerequisite checking for Spec-Driven Development workflow.
|
||||
# It replaces the functionality previously spread across multiple scripts.
|
||||
#
|
||||
# Usage: ./check-prerequisites.sh [OPTIONS]
|
||||
#
|
||||
# OPTIONS:
|
||||
# --json Output in JSON format
|
||||
# --require-tasks Require tasks.md to exist (for implementation phase)
|
||||
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
# --paths-only Only output path variables (no validation)
|
||||
# --help, -h Show help message
|
||||
#
|
||||
# OUTPUTS:
|
||||
# JSON mode: {"FEATURE_DIR":"...", "AVAILABLE_DOCS":["..."]}
|
||||
# Text mode: FEATURE_DIR:... \n AVAILABLE_DOCS: \n ✓/✗ file.md
|
||||
# Paths only: REPO_ROOT: ... \n BRANCH: ... \n FEATURE_DIR: ... etc.
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
REQUIRE_TASKS=false
|
||||
INCLUDE_TASKS=false
|
||||
PATHS_ONLY=false
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--require-tasks)
|
||||
REQUIRE_TASKS=true
|
||||
;;
|
||||
--include-tasks)
|
||||
INCLUDE_TASKS=true
|
||||
;;
|
||||
--paths-only)
|
||||
PATHS_ONLY=true
|
||||
;;
|
||||
--help|-h)
|
||||
cat << 'EOF'
|
||||
Usage: check-prerequisites.sh [OPTIONS]
|
||||
|
||||
Consolidated prerequisite checking for Spec-Driven Development workflow.
|
||||
|
||||
OPTIONS:
|
||||
--json Output in JSON format
|
||||
--require-tasks Require tasks.md to exist (for implementation phase)
|
||||
--include-tasks Include tasks.md in AVAILABLE_DOCS list
|
||||
--paths-only Only output path variables (no prerequisite validation)
|
||||
--help, -h Show this help message
|
||||
|
||||
EXAMPLES:
|
||||
# Check task prerequisites (plan.md required)
|
||||
./check-prerequisites.sh --json
|
||||
|
||||
# Check implementation prerequisites (plan.md + tasks.md required)
|
||||
./check-prerequisites.sh --json --require-tasks --include-tasks
|
||||
|
||||
# Get feature paths only (no validation)
|
||||
./check-prerequisites.sh --paths-only
|
||||
|
||||
EOF
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: Unknown option '$arg'. Use --help for usage information." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Source common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get feature paths and validate branch
|
||||
eval $(get_feature_paths)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# If paths-only mode, output paths and exit (support JSON + paths-only combined)
|
||||
if $PATHS_ONLY; then
|
||||
if $JSON_MODE; then
|
||||
# Minimal JSON paths payload (no validation performed)
|
||||
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
|
||||
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
|
||||
else
|
||||
echo "REPO_ROOT: $REPO_ROOT"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "FEATURE_DIR: $FEATURE_DIR"
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "TASKS: $TASKS"
|
||||
fi
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Validate required directories and files
|
||||
if [[ ! -d "$FEATURE_DIR" ]]; then
|
||||
echo "ERROR: Feature directory not found: $FEATURE_DIR" >&2
|
||||
echo "Run /specify first to create the feature structure." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -f "$IMPL_PLAN" ]]; then
|
||||
echo "ERROR: plan.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /plan first to create the implementation plan." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check for tasks.md if required
|
||||
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
|
||||
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
|
||||
echo "Run /tasks first to create the task list." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Build list of available documents
|
||||
docs=()
|
||||
|
||||
# Always check these optional docs
|
||||
[[ -f "$RESEARCH" ]] && docs+=("research.md")
|
||||
[[ -f "$DATA_MODEL" ]] && docs+=("data-model.md")
|
||||
|
||||
# Check contracts directory (only if it exists and has files)
|
||||
if [[ -d "$CONTRACTS_DIR" ]] && [[ -n "$(ls -A "$CONTRACTS_DIR" 2>/dev/null)" ]]; then
|
||||
docs+=("contracts/")
|
||||
fi
|
||||
|
||||
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
|
||||
|
||||
# Include tasks.md if requested and it exists
|
||||
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
|
||||
docs+=("tasks.md")
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
# Build JSON array of documents
|
||||
if [[ ${#docs[@]} -eq 0 ]]; then
|
||||
json_docs="[]"
|
||||
else
|
||||
json_docs=$(printf '"%s",' "${docs[@]}")
|
||||
json_docs="[${json_docs%,}]"
|
||||
fi
|
||||
|
||||
printf '{"FEATURE_DIR":"%s","AVAILABLE_DOCS":%s}\n' "$FEATURE_DIR" "$json_docs"
|
||||
else
|
||||
# Text output
|
||||
echo "FEATURE_DIR:$FEATURE_DIR"
|
||||
echo "AVAILABLE_DOCS:"
|
||||
|
||||
# Show status of each potential document
|
||||
check_file "$RESEARCH" "research.md"
|
||||
check_file "$DATA_MODEL" "data-model.md"
|
||||
check_dir "$CONTRACTS_DIR" "contracts/"
|
||||
check_file "$QUICKSTART" "quickstart.md"
|
||||
|
||||
if $INCLUDE_TASKS; then
|
||||
check_file "$TASKS" "tasks.md"
|
||||
fi
|
||||
fi
|
||||
@@ -1,113 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Common functions and variables for all scripts
|
||||
|
||||
# Get repository root, with fallback for non-git repositories
|
||||
get_repo_root() {
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
git rev-parse --show-toplevel
|
||||
else
|
||||
# Fall back to script location for non-git repos
|
||||
local script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
(cd "$script_dir/../../.." && pwd)
|
||||
fi
|
||||
}
|
||||
|
||||
# Get current branch, with fallback for non-git repositories
|
||||
get_current_branch() {
|
||||
# First check if SPECIFY_FEATURE environment variable is set
|
||||
if [[ -n "${SPECIFY_FEATURE:-}" ]]; then
|
||||
echo "$SPECIFY_FEATURE"
|
||||
return
|
||||
fi
|
||||
|
||||
# Then check git if available
|
||||
if git rev-parse --abbrev-ref HEAD >/dev/null 2>&1; then
|
||||
git rev-parse --abbrev-ref HEAD
|
||||
return
|
||||
fi
|
||||
|
||||
# For non-git repos, try to find the latest feature directory
|
||||
local repo_root=$(get_repo_root)
|
||||
local specs_dir="$repo_root/specs"
|
||||
|
||||
if [[ -d "$specs_dir" ]]; then
|
||||
local latest_feature=""
|
||||
local highest=0
|
||||
|
||||
for dir in "$specs_dir"/*; do
|
||||
if [[ -d "$dir" ]]; then
|
||||
local dirname=$(basename "$dir")
|
||||
if [[ "$dirname" =~ ^([0-9]{3})- ]]; then
|
||||
local number=${BASH_REMATCH[1]}
|
||||
number=$((10#$number))
|
||||
if [[ "$number" -gt "$highest" ]]; then
|
||||
highest=$number
|
||||
latest_feature=$dirname
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -n "$latest_feature" ]]; then
|
||||
echo "$latest_feature"
|
||||
return
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "main" # Final fallback
|
||||
}
|
||||
|
||||
# Check if we have git available
|
||||
has_git() {
|
||||
git rev-parse --show-toplevel >/dev/null 2>&1
|
||||
}
|
||||
|
||||
check_feature_branch() {
|
||||
local branch="$1"
|
||||
local has_git_repo="$2"
|
||||
|
||||
# For non-git repos, we can't enforce branch naming but still provide output
|
||||
if [[ "$has_git_repo" != "true" ]]; then
|
||||
echo "[specify] Warning: Git repository not detected; skipped branch validation" >&2
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ ! "$branch" =~ ^[0-9]{3}- ]]; then
|
||||
echo "ERROR: Not on a feature branch. Current branch: $branch" >&2
|
||||
echo "Feature branches should be named like: 001-feature-name" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
get_feature_dir() { echo "$1/specs/$2"; }
|
||||
|
||||
get_feature_paths() {
|
||||
local repo_root=$(get_repo_root)
|
||||
local current_branch=$(get_current_branch)
|
||||
local has_git_repo="false"
|
||||
|
||||
if has_git; then
|
||||
has_git_repo="true"
|
||||
fi
|
||||
|
||||
local feature_dir=$(get_feature_dir "$repo_root" "$current_branch")
|
||||
|
||||
cat <<EOF
|
||||
REPO_ROOT='$repo_root'
|
||||
CURRENT_BRANCH='$current_branch'
|
||||
HAS_GIT='$has_git_repo'
|
||||
FEATURE_DIR='$feature_dir'
|
||||
FEATURE_SPEC='$feature_dir/spec.md'
|
||||
IMPL_PLAN='$feature_dir/plan.md'
|
||||
TASKS='$feature_dir/tasks.md'
|
||||
RESEARCH='$feature_dir/research.md'
|
||||
DATA_MODEL='$feature_dir/data-model.md'
|
||||
QUICKSTART='$feature_dir/quickstart.md'
|
||||
CONTRACTS_DIR='$feature_dir/contracts'
|
||||
EOF
|
||||
}
|
||||
|
||||
check_file() { [[ -f "$1" ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
check_dir() { [[ -d "$1" && -n $(ls -A "$1" 2>/dev/null) ]] && echo " ✓ $2" || echo " ✗ $2"; }
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
JSON_MODE=false
|
||||
ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json) JSON_MODE=true ;;
|
||||
--help|-h) echo "Usage: $0 [--json] <feature_description>"; exit 0 ;;
|
||||
*) ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
FEATURE_DESCRIPTION="${ARGS[*]}"
|
||||
if [ -z "$FEATURE_DESCRIPTION" ]; then
|
||||
echo "Usage: $0 [--json] <feature_description>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Function to find the repository root by searching for existing project markers
|
||||
find_repo_root() {
|
||||
local dir="$1"
|
||||
while [ "$dir" != "/" ]; do
|
||||
if [ -d "$dir/.git" ] || [ -d "$dir/.specify" ]; then
|
||||
echo "$dir"
|
||||
return 0
|
||||
fi
|
||||
dir="$(dirname "$dir")"
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Resolve repository root. Prefer git information when available, but fall back
|
||||
# to searching for repository markers so the workflow still functions in repositories that
|
||||
# were initialised with --no-git.
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
if git rev-parse --show-toplevel >/dev/null 2>&1; then
|
||||
REPO_ROOT=$(git rev-parse --show-toplevel)
|
||||
HAS_GIT=true
|
||||
else
|
||||
REPO_ROOT="$(find_repo_root "$SCRIPT_DIR")"
|
||||
if [ -z "$REPO_ROOT" ]; then
|
||||
echo "Error: Could not determine repository root. Please run this script from within the repository." >&2
|
||||
exit 1
|
||||
fi
|
||||
HAS_GIT=false
|
||||
fi
|
||||
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
SPECS_DIR="$REPO_ROOT/specs"
|
||||
mkdir -p "$SPECS_DIR"
|
||||
|
||||
HIGHEST=0
|
||||
if [ -d "$SPECS_DIR" ]; then
|
||||
for dir in "$SPECS_DIR"/*; do
|
||||
[ -d "$dir" ] || continue
|
||||
dirname=$(basename "$dir")
|
||||
number=$(echo "$dirname" | grep -o '^[0-9]\+' || echo "0")
|
||||
number=$((10#$number))
|
||||
if [ "$number" -gt "$HIGHEST" ]; then HIGHEST=$number; fi
|
||||
done
|
||||
fi
|
||||
|
||||
NEXT=$((HIGHEST + 1))
|
||||
FEATURE_NUM=$(printf "%03d" "$NEXT")
|
||||
|
||||
BRANCH_NAME=$(echo "$FEATURE_DESCRIPTION" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g' | sed 's/-\+/-/g' | sed 's/^-//' | sed 's/-$//')
|
||||
WORDS=$(echo "$BRANCH_NAME" | tr '-' '\n' | grep -v '^$' | head -3 | tr '\n' '-' | sed 's/-$//')
|
||||
BRANCH_NAME="${FEATURE_NUM}-${WORDS}"
|
||||
|
||||
if [ "$HAS_GIT" = true ]; then
|
||||
git checkout -b "$BRANCH_NAME"
|
||||
else
|
||||
>&2 echo "[specify] Warning: Git repository not detected; skipped branch creation for $BRANCH_NAME"
|
||||
fi
|
||||
|
||||
FEATURE_DIR="$SPECS_DIR/$BRANCH_NAME"
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/spec-template.md"
|
||||
SPEC_FILE="$FEATURE_DIR/spec.md"
|
||||
if [ -f "$TEMPLATE" ]; then cp "$TEMPLATE" "$SPEC_FILE"; else touch "$SPEC_FILE"; fi
|
||||
|
||||
# Set the SPECIFY_FEATURE environment variable for the current session
|
||||
export SPECIFY_FEATURE="$BRANCH_NAME"
|
||||
|
||||
if $JSON_MODE; then
|
||||
printf '{"BRANCH_NAME":"%s","SPEC_FILE":"%s","FEATURE_NUM":"%s"}\n' "$BRANCH_NAME" "$SPEC_FILE" "$FEATURE_NUM"
|
||||
else
|
||||
echo "BRANCH_NAME: $BRANCH_NAME"
|
||||
echo "SPEC_FILE: $SPEC_FILE"
|
||||
echo "FEATURE_NUM: $FEATURE_NUM"
|
||||
echo "SPECIFY_FEATURE environment variable set to: $BRANCH_NAME"
|
||||
fi
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -e
|
||||
|
||||
# Parse command line arguments
|
||||
JSON_MODE=false
|
||||
ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--json)
|
||||
JSON_MODE=true
|
||||
;;
|
||||
--help|-h)
|
||||
echo "Usage: $0 [--json]"
|
||||
echo " --json Output results in JSON format"
|
||||
echo " --help Show this help message"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
ARGS+=("$arg")
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
# Check if we're on a proper feature branch (only for git repos)
|
||||
check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
|
||||
|
||||
# Ensure the feature directory exists
|
||||
mkdir -p "$FEATURE_DIR"
|
||||
|
||||
# Copy plan template if it exists
|
||||
TEMPLATE="$REPO_ROOT/.specify/templates/plan-template.md"
|
||||
if [[ -f "$TEMPLATE" ]]; then
|
||||
cp "$TEMPLATE" "$IMPL_PLAN"
|
||||
echo "Copied plan template to $IMPL_PLAN"
|
||||
else
|
||||
echo "Warning: Plan template not found at $TEMPLATE"
|
||||
# Create a basic plan file if template doesn't exist
|
||||
touch "$IMPL_PLAN"
|
||||
fi
|
||||
|
||||
# Output results
|
||||
if $JSON_MODE; then
|
||||
printf '{"FEATURE_SPEC":"%s","IMPL_PLAN":"%s","SPECS_DIR":"%s","BRANCH":"%s","HAS_GIT":"%s"}\n' \
|
||||
"$FEATURE_SPEC" "$IMPL_PLAN" "$FEATURE_DIR" "$CURRENT_BRANCH" "$HAS_GIT"
|
||||
else
|
||||
echo "FEATURE_SPEC: $FEATURE_SPEC"
|
||||
echo "IMPL_PLAN: $IMPL_PLAN"
|
||||
echo "SPECS_DIR: $FEATURE_DIR"
|
||||
echo "BRANCH: $CURRENT_BRANCH"
|
||||
echo "HAS_GIT: $HAS_GIT"
|
||||
fi
|
||||
@@ -1,719 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Update agent context files with information from plan.md
|
||||
#
|
||||
# This script maintains AI agent context files by parsing feature specifications
|
||||
# and updating agent-specific configuration files with project information.
|
||||
#
|
||||
# MAIN FUNCTIONS:
|
||||
# 1. Environment Validation
|
||||
# - Verifies git repository structure and branch information
|
||||
# - Checks for required plan.md files and templates
|
||||
# - Validates file permissions and accessibility
|
||||
#
|
||||
# 2. Plan Data Extraction
|
||||
# - Parses plan.md files to extract project metadata
|
||||
# - Identifies language/version, frameworks, databases, and project types
|
||||
# - Handles missing or incomplete specification data gracefully
|
||||
#
|
||||
# 3. Agent File Management
|
||||
# - Creates new agent context files from templates when needed
|
||||
# - Updates existing agent files with new project information
|
||||
# - Preserves manual additions and custom configurations
|
||||
# - Supports multiple AI agent formats and directory structures
|
||||
#
|
||||
# 4. Content Generation
|
||||
# - Generates language-specific build/test commands
|
||||
# - Creates appropriate project directory structures
|
||||
# - Updates technology stacks and recent changes sections
|
||||
# - Maintains consistent formatting and timestamps
|
||||
#
|
||||
# 5. Multi-Agent Support
|
||||
# - Handles agent-specific file paths and naming conventions
|
||||
# - Supports: Claude, Gemini, Copilot, Cursor, Qwen, opencode, Codex, Windsurf
|
||||
# - Can update single agents or all existing agent files
|
||||
# - Creates default Claude file if no agent files exist
|
||||
#
|
||||
# Usage: ./update-agent-context.sh [agent_type]
|
||||
# Agent types: claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf
|
||||
# Leave empty to update all existing agent files
|
||||
|
||||
set -e
|
||||
|
||||
# Enable strict error handling
|
||||
set -u
|
||||
set -o pipefail
|
||||
|
||||
#==============================================================================
|
||||
# Configuration and Global Variables
|
||||
#==============================================================================
|
||||
|
||||
# Get script directory and load common functions
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/common.sh"
|
||||
|
||||
# Get all paths and variables from common functions
|
||||
eval $(get_feature_paths)
|
||||
|
||||
NEW_PLAN="$IMPL_PLAN" # Alias for compatibility with existing code
|
||||
AGENT_TYPE="${1:-}"
|
||||
|
||||
# Agent-specific file paths
|
||||
CLAUDE_FILE="$REPO_ROOT/CLAUDE.md"
|
||||
GEMINI_FILE="$REPO_ROOT/GEMINI.md"
|
||||
COPILOT_FILE="$REPO_ROOT/.github/copilot-instructions.md"
|
||||
CURSOR_FILE="$REPO_ROOT/.cursor/rules/specify-rules.mdc"
|
||||
QWEN_FILE="$REPO_ROOT/QWEN.md"
|
||||
AGENTS_FILE="$REPO_ROOT/AGENTS.md"
|
||||
WINDSURF_FILE="$REPO_ROOT/.windsurf/rules/specify-rules.md"
|
||||
KILOCODE_FILE="$REPO_ROOT/.kilocode/rules/specify-rules.md"
|
||||
AUGGIE_FILE="$REPO_ROOT/.augment/rules/specify-rules.md"
|
||||
ROO_FILE="$REPO_ROOT/.roo/rules/specify-rules.md"
|
||||
|
||||
# Template file
|
||||
TEMPLATE_FILE="$REPO_ROOT/.specify/templates/agent-file-template.md"
|
||||
|
||||
# Global variables for parsed plan data
|
||||
NEW_LANG=""
|
||||
NEW_FRAMEWORK=""
|
||||
NEW_DB=""
|
||||
NEW_PROJECT_TYPE=""
|
||||
|
||||
#==============================================================================
|
||||
# Utility Functions
|
||||
#==============================================================================
|
||||
|
||||
log_info() {
|
||||
echo "INFO: $1"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo "✓ $1"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo "ERROR: $1" >&2
|
||||
}
|
||||
|
||||
log_warning() {
|
||||
echo "WARNING: $1" >&2
|
||||
}
|
||||
|
||||
# Cleanup function for temporary files
|
||||
cleanup() {
|
||||
local exit_code=$?
|
||||
rm -f /tmp/agent_update_*_$$
|
||||
rm -f /tmp/manual_additions_$$
|
||||
exit $exit_code
|
||||
}
|
||||
|
||||
# Set up cleanup trap
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
#==============================================================================
|
||||
# Validation Functions
|
||||
#==============================================================================
|
||||
|
||||
validate_environment() {
|
||||
# Check if we have a current branch/feature (git or non-git)
|
||||
if [[ -z "$CURRENT_BRANCH" ]]; then
|
||||
log_error "Unable to determine current feature"
|
||||
if [[ "$HAS_GIT" == "true" ]]; then
|
||||
log_info "Make sure you're on a feature branch"
|
||||
else
|
||||
log_info "Set SPECIFY_FEATURE environment variable or create a feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if plan.md exists
|
||||
if [[ ! -f "$NEW_PLAN" ]]; then
|
||||
log_error "No plan.md found at $NEW_PLAN"
|
||||
log_info "Make sure you're working on a feature with a corresponding spec directory"
|
||||
if [[ "$HAS_GIT" != "true" ]]; then
|
||||
log_info "Use: export SPECIFY_FEATURE=your-feature-name or create a new feature first"
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if template exists (needed for new files)
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_warning "Template file not found at $TEMPLATE_FILE"
|
||||
log_warning "Creating new agent files will fail"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Plan Parsing Functions
|
||||
#==============================================================================
|
||||
|
||||
extract_plan_field() {
|
||||
local field_pattern="$1"
|
||||
local plan_file="$2"
|
||||
|
||||
grep "^\*\*${field_pattern}\*\*: " "$plan_file" 2>/dev/null | \
|
||||
head -1 | \
|
||||
sed "s|^\*\*${field_pattern}\*\*: ||" | \
|
||||
sed 's/^[ \t]*//;s/[ \t]*$//' | \
|
||||
grep -v "NEEDS CLARIFICATION" | \
|
||||
grep -v "^N/A$" || echo ""
|
||||
}
|
||||
|
||||
parse_plan_data() {
|
||||
local plan_file="$1"
|
||||
|
||||
if [[ ! -f "$plan_file" ]]; then
|
||||
log_error "Plan file not found: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$plan_file" ]]; then
|
||||
log_error "Plan file is not readable: $plan_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Parsing plan data from $plan_file"
|
||||
|
||||
NEW_LANG=$(extract_plan_field "Language/Version" "$plan_file")
|
||||
NEW_FRAMEWORK=$(extract_plan_field "Primary Dependencies" "$plan_file")
|
||||
NEW_DB=$(extract_plan_field "Storage" "$plan_file")
|
||||
NEW_PROJECT_TYPE=$(extract_plan_field "Project Type" "$plan_file")
|
||||
|
||||
# Log what we found
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
log_info "Found language: $NEW_LANG"
|
||||
else
|
||||
log_warning "No language information found in plan"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
log_info "Found framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
log_info "Found database: $NEW_DB"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_PROJECT_TYPE" ]]; then
|
||||
log_info "Found project type: $NEW_PROJECT_TYPE"
|
||||
fi
|
||||
}
|
||||
|
||||
format_technology_stack() {
|
||||
local lang="$1"
|
||||
local framework="$2"
|
||||
local parts=()
|
||||
|
||||
# Add non-empty parts
|
||||
[[ -n "$lang" && "$lang" != "NEEDS CLARIFICATION" ]] && parts+=("$lang")
|
||||
[[ -n "$framework" && "$framework" != "NEEDS CLARIFICATION" && "$framework" != "N/A" ]] && parts+=("$framework")
|
||||
|
||||
# Join with proper formatting
|
||||
if [[ ${#parts[@]} -eq 0 ]]; then
|
||||
echo ""
|
||||
elif [[ ${#parts[@]} -eq 1 ]]; then
|
||||
echo "${parts[0]}"
|
||||
else
|
||||
# Join multiple parts with " + "
|
||||
local result="${parts[0]}"
|
||||
for ((i=1; i<${#parts[@]}; i++)); do
|
||||
result="$result + ${parts[i]}"
|
||||
done
|
||||
echo "$result"
|
||||
fi
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Template and Content Generation Functions
|
||||
#==============================================================================
|
||||
|
||||
get_project_structure() {
|
||||
local project_type="$1"
|
||||
|
||||
if [[ "$project_type" == *"web"* ]]; then
|
||||
echo "backend/\\nfrontend/\\ntests/"
|
||||
else
|
||||
echo "src/\\ntests/"
|
||||
fi
|
||||
}
|
||||
|
||||
get_commands_for_language() {
|
||||
local lang="$1"
|
||||
|
||||
case "$lang" in
|
||||
*"Python"*)
|
||||
echo "cd src && pytest && ruff check ."
|
||||
;;
|
||||
*"Rust"*)
|
||||
echo "cargo test && cargo clippy"
|
||||
;;
|
||||
*"JavaScript"*|*"TypeScript"*)
|
||||
echo "npm test && npm run lint"
|
||||
;;
|
||||
*)
|
||||
echo "# Add commands for $lang"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
get_language_conventions() {
|
||||
local lang="$1"
|
||||
echo "$lang: Follow standard conventions"
|
||||
}
|
||||
|
||||
create_new_agent_file() {
|
||||
local target_file="$1"
|
||||
local temp_file="$2"
|
||||
local project_name="$3"
|
||||
local current_date="$4"
|
||||
|
||||
if [[ ! -f "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template not found at $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -r "$TEMPLATE_FILE" ]]; then
|
||||
log_error "Template file is not readable: $TEMPLATE_FILE"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Creating new agent context file from template..."
|
||||
|
||||
if ! cp "$TEMPLATE_FILE" "$temp_file"; then
|
||||
log_error "Failed to copy template file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Replace template placeholders
|
||||
local project_structure
|
||||
project_structure=$(get_project_structure "$NEW_PROJECT_TYPE")
|
||||
|
||||
local commands
|
||||
commands=$(get_commands_for_language "$NEW_LANG")
|
||||
|
||||
local language_conventions
|
||||
language_conventions=$(get_language_conventions "$NEW_LANG")
|
||||
|
||||
# Perform substitutions with error checking using safer approach
|
||||
# Escape special characters for sed by using a different delimiter or escaping
|
||||
local escaped_lang=$(printf '%s\n' "$NEW_LANG" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_framework=$(printf '%s\n' "$NEW_FRAMEWORK" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
local escaped_branch=$(printf '%s\n' "$CURRENT_BRANCH" | sed 's/[\[\.*^$()+{}|]/\\&/g')
|
||||
|
||||
# Build technology stack and recent change strings conditionally
|
||||
local tech_stack
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_lang + $escaped_framework ($escaped_branch)"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
tech_stack="- $escaped_lang ($escaped_branch)"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
tech_stack="- $escaped_framework ($escaped_branch)"
|
||||
else
|
||||
tech_stack="- ($escaped_branch)"
|
||||
fi
|
||||
|
||||
local recent_change
|
||||
if [[ -n "$escaped_lang" && -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang + $escaped_framework"
|
||||
elif [[ -n "$escaped_lang" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_lang"
|
||||
elif [[ -n "$escaped_framework" ]]; then
|
||||
recent_change="- $escaped_branch: Added $escaped_framework"
|
||||
else
|
||||
recent_change="- $escaped_branch: Added"
|
||||
fi
|
||||
|
||||
local substitutions=(
|
||||
"s|\[PROJECT NAME\]|$project_name|"
|
||||
"s|\[DATE\]|$current_date|"
|
||||
"s|\[EXTRACTED FROM ALL PLAN.MD FILES\]|$tech_stack|"
|
||||
"s|\[ACTUAL STRUCTURE FROM PLANS\]|$project_structure|g"
|
||||
"s|\[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES\]|$commands|"
|
||||
"s|\[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE\]|$language_conventions|"
|
||||
"s|\[LAST 3 FEATURES AND WHAT THEY ADDED\]|$recent_change|"
|
||||
)
|
||||
|
||||
for substitution in "${substitutions[@]}"; do
|
||||
if ! sed -i.bak -e "$substitution" "$temp_file"; then
|
||||
log_error "Failed to perform substitution: $substitution"
|
||||
rm -f "$temp_file" "$temp_file.bak"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Convert \n sequences to actual newlines
|
||||
newline=$(printf '\n')
|
||||
sed -i.bak2 "s/\\\\n/${newline}/g" "$temp_file"
|
||||
|
||||
# Clean up backup files
|
||||
rm -f "$temp_file.bak" "$temp_file.bak2"
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
update_existing_agent_file() {
|
||||
local target_file="$1"
|
||||
local current_date="$2"
|
||||
|
||||
log_info "Updating existing agent context file..."
|
||||
|
||||
# Use a single temporary file for atomic update
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
# Process the file in one pass
|
||||
local tech_stack=$(format_technology_stack "$NEW_LANG" "$NEW_FRAMEWORK")
|
||||
local new_tech_entries=()
|
||||
local new_change_entry=""
|
||||
|
||||
# Prepare new technology entries
|
||||
if [[ -n "$tech_stack" ]] && ! grep -q "$tech_stack" "$target_file"; then
|
||||
new_tech_entries+=("- $tech_stack ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]] && ! grep -q "$NEW_DB" "$target_file"; then
|
||||
new_tech_entries+=("- $NEW_DB ($CURRENT_BRANCH)")
|
||||
fi
|
||||
|
||||
# Prepare new change entry
|
||||
if [[ -n "$tech_stack" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $tech_stack"
|
||||
elif [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]] && [[ "$NEW_DB" != "NEEDS CLARIFICATION" ]]; then
|
||||
new_change_entry="- $CURRENT_BRANCH: Added $NEW_DB"
|
||||
fi
|
||||
|
||||
# Process file line by line
|
||||
local in_tech_section=false
|
||||
local in_changes_section=false
|
||||
local tech_entries_added=false
|
||||
local changes_entries_added=false
|
||||
local existing_changes_count=0
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
# Handle Active Technologies section
|
||||
if [[ "$line" == "## Active Technologies" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=true
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
# Add new tech entries before closing the section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
in_tech_section=false
|
||||
continue
|
||||
elif [[ $in_tech_section == true ]] && [[ -z "$line" ]]; then
|
||||
# Add new tech entries before empty line in tech section
|
||||
if [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
tech_entries_added=true
|
||||
fi
|
||||
echo "$line" >> "$temp_file"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Handle Recent Changes section
|
||||
if [[ "$line" == "## Recent Changes" ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
# Add new change entry right after the heading
|
||||
if [[ -n "$new_change_entry" ]]; then
|
||||
echo "$new_change_entry" >> "$temp_file"
|
||||
fi
|
||||
in_changes_section=true
|
||||
changes_entries_added=true
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" =~ ^##[[:space:]] ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
in_changes_section=false
|
||||
continue
|
||||
elif [[ $in_changes_section == true ]] && [[ "$line" == "- "* ]]; then
|
||||
# Keep only first 2 existing changes
|
||||
if [[ $existing_changes_count -lt 2 ]]; then
|
||||
echo "$line" >> "$temp_file"
|
||||
((existing_changes_count++))
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
|
||||
# Update timestamp
|
||||
if [[ "$line" =~ \*\*Last\ updated\*\*:.*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
|
||||
echo "$line" | sed "s/[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]/$current_date/" >> "$temp_file"
|
||||
else
|
||||
echo "$line" >> "$temp_file"
|
||||
fi
|
||||
done < "$target_file"
|
||||
|
||||
# Post-loop check: if we're still in the Active Technologies section and haven't added new entries
|
||||
if [[ $in_tech_section == true ]] && [[ $tech_entries_added == false ]] && [[ ${#new_tech_entries[@]} -gt 0 ]]; then
|
||||
printf '%s\n' "${new_tech_entries[@]}" >> "$temp_file"
|
||||
fi
|
||||
|
||||
# Move temp file to target atomically
|
||||
if ! mv "$temp_file" "$target_file"; then
|
||||
log_error "Failed to update target file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
#==============================================================================
|
||||
# Main Agent File Update Function
|
||||
#==============================================================================
|
||||
|
||||
update_agent_file() {
|
||||
local target_file="$1"
|
||||
local agent_name="$2"
|
||||
|
||||
if [[ -z "$target_file" ]] || [[ -z "$agent_name" ]]; then
|
||||
log_error "update_agent_file requires target_file and agent_name parameters"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "Updating $agent_name context file: $target_file"
|
||||
|
||||
local project_name
|
||||
project_name=$(basename "$REPO_ROOT")
|
||||
local current_date
|
||||
current_date=$(date +%Y-%m-%d)
|
||||
|
||||
# Create directory if it doesn't exist
|
||||
local target_dir
|
||||
target_dir=$(dirname "$target_file")
|
||||
if [[ ! -d "$target_dir" ]]; then
|
||||
if ! mkdir -p "$target_dir"; then
|
||||
log_error "Failed to create directory: $target_dir"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ ! -f "$target_file" ]]; then
|
||||
# Create new file from template
|
||||
local temp_file
|
||||
temp_file=$(mktemp) || {
|
||||
log_error "Failed to create temporary file"
|
||||
return 1
|
||||
}
|
||||
|
||||
if create_new_agent_file "$target_file" "$temp_file" "$project_name" "$current_date"; then
|
||||
if mv "$temp_file" "$target_file"; then
|
||||
log_success "Created new $agent_name context file"
|
||||
else
|
||||
log_error "Failed to move temporary file to $target_file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log_error "Failed to create new agent file"
|
||||
rm -f "$temp_file"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
# Update existing file
|
||||
if [[ ! -r "$target_file" ]]; then
|
||||
log_error "Cannot read existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ ! -w "$target_file" ]]; then
|
||||
log_error "Cannot write to existing file: $target_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if update_existing_agent_file "$target_file" "$current_date"; then
|
||||
log_success "Updated existing $agent_name context file"
|
||||
else
|
||||
log_error "Failed to update existing agent file"
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Agent Selection and Processing
|
||||
#==============================================================================
|
||||
|
||||
update_specific_agent() {
|
||||
local agent_type="$1"
|
||||
|
||||
case "$agent_type" in
|
||||
claude)
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
;;
|
||||
gemini)
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
;;
|
||||
copilot)
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
;;
|
||||
cursor)
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
;;
|
||||
qwen)
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
;;
|
||||
opencode)
|
||||
update_agent_file "$AGENTS_FILE" "opencode"
|
||||
;;
|
||||
codex)
|
||||
update_agent_file "$AGENTS_FILE" "Codex CLI"
|
||||
;;
|
||||
windsurf)
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
;;
|
||||
kilocode)
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
;;
|
||||
auggie)
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
;;
|
||||
roo)
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
;;
|
||||
*)
|
||||
log_error "Unknown agent type '$agent_type'"
|
||||
log_error "Expected: claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf|kilocode|auggie|roo"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
update_all_existing_agents() {
|
||||
local found_agent=false
|
||||
|
||||
# Check each possible agent file and update if it exists
|
||||
if [[ -f "$CLAUDE_FILE" ]]; then
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$GEMINI_FILE" ]]; then
|
||||
update_agent_file "$GEMINI_FILE" "Gemini CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$COPILOT_FILE" ]]; then
|
||||
update_agent_file "$COPILOT_FILE" "GitHub Copilot"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$CURSOR_FILE" ]]; then
|
||||
update_agent_file "$CURSOR_FILE" "Cursor IDE"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$QWEN_FILE" ]]; then
|
||||
update_agent_file "$QWEN_FILE" "Qwen Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AGENTS_FILE" ]]; then
|
||||
update_agent_file "$AGENTS_FILE" "Codex/opencode"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$WINDSURF_FILE" ]]; then
|
||||
update_agent_file "$WINDSURF_FILE" "Windsurf"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$KILOCODE_FILE" ]]; then
|
||||
update_agent_file "$KILOCODE_FILE" "Kilo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$AUGGIE_FILE" ]]; then
|
||||
update_agent_file "$AUGGIE_FILE" "Auggie CLI"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
if [[ -f "$ROO_FILE" ]]; then
|
||||
update_agent_file "$ROO_FILE" "Roo Code"
|
||||
found_agent=true
|
||||
fi
|
||||
|
||||
# If no agent files exist, create a default Claude file
|
||||
if [[ "$found_agent" == false ]]; then
|
||||
log_info "No existing agent files found, creating default Claude file..."
|
||||
update_agent_file "$CLAUDE_FILE" "Claude Code"
|
||||
fi
|
||||
}
|
||||
print_summary() {
|
||||
echo
|
||||
log_info "Summary of changes:"
|
||||
|
||||
if [[ -n "$NEW_LANG" ]]; then
|
||||
echo " - Added language: $NEW_LANG"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_FRAMEWORK" ]]; then
|
||||
echo " - Added framework: $NEW_FRAMEWORK"
|
||||
fi
|
||||
|
||||
if [[ -n "$NEW_DB" ]] && [[ "$NEW_DB" != "N/A" ]]; then
|
||||
echo " - Added database: $NEW_DB"
|
||||
fi
|
||||
|
||||
echo
|
||||
log_info "Usage: $0 [claude|gemini|copilot|cursor|qwen|opencode|codex|windsurf|kilocode|auggie|roo]"
|
||||
}
|
||||
|
||||
#==============================================================================
|
||||
# Main Execution
|
||||
#==============================================================================
|
||||
|
||||
main() {
|
||||
# Validate environment before proceeding
|
||||
validate_environment
|
||||
|
||||
log_info "=== Updating agent context files for feature $CURRENT_BRANCH ==="
|
||||
|
||||
# Parse the plan file to extract project information
|
||||
if ! parse_plan_data "$NEW_PLAN"; then
|
||||
log_error "Failed to parse plan data"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Process based on agent type argument
|
||||
local success=true
|
||||
|
||||
if [[ -z "$AGENT_TYPE" ]]; then
|
||||
# No specific agent provided - update all existing agent files
|
||||
log_info "No agent specified, updating all existing agent files..."
|
||||
if ! update_all_existing_agents; then
|
||||
success=false
|
||||
fi
|
||||
else
|
||||
# Specific agent provided - update only that agent
|
||||
log_info "Updating specific agent: $AGENT_TYPE"
|
||||
if ! update_specific_agent "$AGENT_TYPE"; then
|
||||
success=false
|
||||
fi
|
||||
fi
|
||||
|
||||
# Print summary
|
||||
print_summary
|
||||
|
||||
if [[ "$success" == true ]]; then
|
||||
log_success "Agent context update completed successfully"
|
||||
exit 0
|
||||
else
|
||||
log_error "Agent context update completed with errors"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Execute main function if script is run directly
|
||||
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||
main "$@"
|
||||
fi
|
||||
@@ -1,23 +0,0 @@
|
||||
# [PROJECT NAME] Development Guidelines
|
||||
|
||||
Auto-generated from all feature plans. Last updated: [DATE]
|
||||
|
||||
## Active Technologies
|
||||
[EXTRACTED FROM ALL PLAN.MD FILES]
|
||||
|
||||
## Project Structure
|
||||
```
|
||||
[ACTUAL STRUCTURE FROM PLANS]
|
||||
```
|
||||
|
||||
## Commands
|
||||
[ONLY COMMANDS FOR ACTIVE TECHNOLOGIES]
|
||||
|
||||
## Code Style
|
||||
[LANGUAGE-SPECIFIC, ONLY FOR LANGUAGES IN USE]
|
||||
|
||||
## Recent Changes
|
||||
[LAST 3 FEATURES AND WHAT THEY ADDED]
|
||||
|
||||
<!-- MANUAL ADDITIONS START -->
|
||||
<!-- MANUAL ADDITIONS END -->
|
||||
@@ -1,219 +0,0 @@
|
||||
|
||||
# Implementation Plan: [FEATURE]
|
||||
|
||||
**Branch**: `[###-feature-name]` | **Date**: [DATE] | **Spec**: [link]
|
||||
**Input**: Feature specification from `/specs/[###-feature-name]/spec.md`
|
||||
|
||||
## Execution Flow (/plan command scope)
|
||||
```
|
||||
1. Load feature spec from Input path
|
||||
→ If not found: ERROR "No feature spec at {path}"
|
||||
2. Fill Technical Context (scan for NEEDS CLARIFICATION)
|
||||
→ Detect Project Type from file system structure or context (web=frontend+backend, mobile=app+api)
|
||||
→ Set Structure Decision based on project type
|
||||
3. Fill the Constitution Check section based on the content of the constitution document.
|
||||
4. Evaluate Constitution Check section below
|
||||
→ If violations exist: Document in Complexity Tracking
|
||||
→ If no justification possible: ERROR "Simplify approach first"
|
||||
→ Update Progress Tracking: Initial Constitution Check
|
||||
5. Execute Phase 0 → research.md
|
||||
→ If NEEDS CLARIFICATION remain: ERROR "Resolve unknowns"
|
||||
6. Execute Phase 1 → contracts, data-model.md, quickstart.md, agent-specific template file (e.g., `CLAUDE.md` for Claude Code, `.github/copilot-instructions.md` for GitHub Copilot, `GEMINI.md` for Gemini CLI, `QWEN.md` for Qwen Code or `AGENTS.md` for opencode).
|
||||
7. Re-evaluate Constitution Check section
|
||||
→ If new violations: Refactor design, return to Phase 1
|
||||
→ Update Progress Tracking: Post-Design Constitution Check
|
||||
8. Plan Phase 2 → Describe task generation approach (DO NOT create tasks.md)
|
||||
9. STOP - Ready for /tasks command
|
||||
```
|
||||
|
||||
**IMPORTANT**: The /plan command STOPS at step 7. Phases 2-4 are executed by other commands:
|
||||
- Phase 2: /tasks command creates tasks.md
|
||||
- Phase 3-4: Implementation execution (manual or via tools)
|
||||
|
||||
## Summary
|
||||
[Extract from feature spec: primary requirement + technical approach from research]
|
||||
|
||||
## Technical Context
|
||||
**Language/Version**: [e.g., Python 3.11, Swift 5.9, Rust 1.75 or NEEDS CLARIFICATION]
|
||||
**Primary Dependencies**: [e.g., FastAPI, UIKit, LLVM or NEEDS CLARIFICATION]
|
||||
**Storage**: [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
|
||||
**Testing**: [e.g., pytest, XCTest, cargo test or NEEDS CLARIFICATION]
|
||||
**Target Platform**: [e.g., Linux server, iOS 15+, WASM or NEEDS CLARIFICATION]
|
||||
**Project Type**: [single/web/mobile - determines source structure]
|
||||
**Performance Goals**: [domain-specific, e.g., 1000 req/s, 10k lines/sec, 60 fps or NEEDS CLARIFICATION]
|
||||
**Constraints**: [domain-specific, e.g., <200ms p95, <100MB memory, offline-capable or NEEDS CLARIFICATION]
|
||||
**Scale/Scope**: [domain-specific, e.g., 10k users, 1M LOC, 50 screens or NEEDS CLARIFICATION]
|
||||
|
||||
## Constitution Check
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
[Gates determined based on constitution file]
|
||||
|
||||
## Project Structure
|
||||
|
||||
### Documentation (this feature)
|
||||
```
|
||||
specs/[###-feature]/
|
||||
├── plan.md # This file (/plan command output)
|
||||
├── research.md # Phase 0 output (/plan command)
|
||||
├── data-model.md # Phase 1 output (/plan command)
|
||||
├── quickstart.md # Phase 1 output (/plan command)
|
||||
├── contracts/ # Phase 1 output (/plan command)
|
||||
└── tasks.md # Phase 2 output (/tasks command - NOT created by /plan)
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
<!--
|
||||
ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
|
||||
for this feature. Delete unused options and expand the chosen structure with
|
||||
real paths (e.g., apps/admin, packages/something). The delivered plan must
|
||||
not include Option labels.
|
||||
-->
|
||||
```
|
||||
# [REMOVE IF UNUSED] Option 1: Single project (DEFAULT)
|
||||
src/
|
||||
├── models/
|
||||
├── services/
|
||||
├── cli/
|
||||
└── lib/
|
||||
|
||||
tests/
|
||||
├── contract/
|
||||
├── integration/
|
||||
└── unit/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 2: Web application (when "frontend" + "backend" detected)
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── models/
|
||||
│ ├── services/
|
||||
│ └── api/
|
||||
└── tests/
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ ├── pages/
|
||||
│ └── services/
|
||||
└── tests/
|
||||
|
||||
# [REMOVE IF UNUSED] Option 3: Mobile + API (when "iOS/Android" detected)
|
||||
api/
|
||||
└── [same as backend above]
|
||||
|
||||
ios/ or android/
|
||||
└── [platform-specific structure: feature modules, UI flows, platform tests]
|
||||
```
|
||||
|
||||
**Structure Decision**: [Document the selected structure and reference the real
|
||||
directories captured above]
|
||||
|
||||
## Phase 0: Outline & Research
|
||||
1. **Extract unknowns from Technical Context** above:
|
||||
- For each NEEDS CLARIFICATION → research task
|
||||
- For each dependency → best practices task
|
||||
- For each integration → patterns task
|
||||
|
||||
2. **Generate and dispatch research agents**:
|
||||
```
|
||||
For each unknown in Technical Context:
|
||||
Task: "Research {unknown} for {feature context}"
|
||||
For each technology choice:
|
||||
Task: "Find best practices for {tech} in {domain}"
|
||||
```
|
||||
|
||||
3. **Consolidate findings** in `research.md` using format:
|
||||
- Decision: [what was chosen]
|
||||
- Rationale: [why chosen]
|
||||
- Alternatives considered: [what else evaluated]
|
||||
|
||||
**Output**: research.md with all NEEDS CLARIFICATION resolved
|
||||
|
||||
## Phase 1: Design & Contracts
|
||||
*Prerequisites: research.md complete*
|
||||
|
||||
1. **Extract entities from feature spec** → `data-model.md`:
|
||||
- Entity name, fields, relationships
|
||||
- Validation rules from requirements
|
||||
- State transitions if applicable
|
||||
|
||||
2. **Generate API contracts** from functional requirements:
|
||||
- For each user action → endpoint
|
||||
- Use standard REST/GraphQL patterns
|
||||
- Output OpenAPI/GraphQL schema to `/contracts/`
|
||||
|
||||
3. **Generate contract tests** from contracts:
|
||||
- One test file per endpoint
|
||||
- Assert request/response schemas
|
||||
- Tests must fail (no implementation yet)
|
||||
|
||||
4. **Extract test scenarios** from user stories:
|
||||
- Each story → integration test scenario
|
||||
- Quickstart test = story validation steps
|
||||
|
||||
5. **Update agent file incrementally** (O(1) operation):
|
||||
- Run `.specify/scripts/bash/update-agent-context.sh cursor`
|
||||
**IMPORTANT**: Execute it exactly as specified above. Do not add or remove any arguments.
|
||||
- If exists: Add only NEW tech from current plan
|
||||
- Preserve manual additions between markers
|
||||
- Update recent changes (keep last 3)
|
||||
- Keep under 150 lines for token efficiency
|
||||
- Output to repository root
|
||||
|
||||
**Output**: data-model.md, /contracts/*, failing tests, quickstart.md, agent-specific file
|
||||
|
||||
## Phase 2: Task Planning Approach
|
||||
*This section describes what the /tasks command will do - DO NOT execute during /plan*
|
||||
|
||||
**Task Generation Strategy**:
|
||||
- Load `.specify/templates/tasks-template.md` as base
|
||||
- Generate tasks from Phase 1 design docs (contracts, data model, quickstart)
|
||||
- Each contract → contract test task [P]
|
||||
- Each entity → model creation task [P]
|
||||
- Each user story → integration test task
|
||||
- Implementation tasks to make tests pass
|
||||
|
||||
**Ordering Strategy**:
|
||||
- TDD order: Tests before implementation
|
||||
- Dependency order: Models before services before UI
|
||||
- Mark [P] for parallel execution (independent files)
|
||||
|
||||
**Estimated Output**: 25-30 numbered, ordered tasks in tasks.md
|
||||
|
||||
**IMPORTANT**: This phase is executed by the /tasks command, NOT by /plan
|
||||
|
||||
## Phase 3+: Future Implementation
|
||||
*These phases are beyond the scope of the /plan command*
|
||||
|
||||
**Phase 3**: Task execution (/tasks command creates tasks.md)
|
||||
**Phase 4**: Implementation (execute tasks.md following constitutional principles)
|
||||
**Phase 5**: Validation (run tests, execute quickstart.md, performance validation)
|
||||
|
||||
## Complexity Tracking
|
||||
*Fill ONLY if Constitution Check has violations that must be justified*
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| [e.g., 4th project] | [current need] | [why 3 projects insufficient] |
|
||||
| [e.g., Repository pattern] | [specific problem] | [why direct DB access insufficient] |
|
||||
|
||||
|
||||
## Progress Tracking
|
||||
*This checklist is updated during execution flow*
|
||||
|
||||
**Phase Status**:
|
||||
- [ ] Phase 0: Research complete (/plan command)
|
||||
- [ ] Phase 1: Design complete (/plan command)
|
||||
- [ ] Phase 2: Task planning complete (/plan command - describe approach only)
|
||||
- [ ] Phase 3: Tasks generated (/tasks command)
|
||||
- [ ] Phase 4: Implementation complete
|
||||
- [ ] Phase 5: Validation passed
|
||||
|
||||
**Gate Status**:
|
||||
- [ ] Initial Constitution Check: PASS
|
||||
- [ ] Post-Design Constitution Check: PASS
|
||||
- [ ] All NEEDS CLARIFICATION resolved
|
||||
- [ ] Complexity deviations documented
|
||||
|
||||
---
|
||||
*Based on Constitution v2.1.1 - See `/memory/constitution.md`*
|
||||
@@ -1,116 +0,0 @@
|
||||
# Feature Specification: [FEATURE NAME]
|
||||
|
||||
**Feature Branch**: `[###-feature-name]`
|
||||
**Created**: [DATE]
|
||||
**Status**: Draft
|
||||
**Input**: User description: "$ARGUMENTS"
|
||||
|
||||
## Execution Flow (main)
|
||||
```
|
||||
1. Parse user description from Input
|
||||
→ If empty: ERROR "No feature description provided"
|
||||
2. Extract key concepts from description
|
||||
→ Identify: actors, actions, data, constraints
|
||||
3. For each unclear aspect:
|
||||
→ Mark with [NEEDS CLARIFICATION: specific question]
|
||||
4. Fill User Scenarios & Testing section
|
||||
→ If no clear user flow: ERROR "Cannot determine user scenarios"
|
||||
5. Generate Functional Requirements
|
||||
→ Each requirement must be testable
|
||||
→ Mark ambiguous requirements
|
||||
6. Identify Key Entities (if data involved)
|
||||
7. Run Review Checklist
|
||||
→ If any [NEEDS CLARIFICATION]: WARN "Spec has uncertainties"
|
||||
→ If implementation details found: ERROR "Remove tech details"
|
||||
8. Return: SUCCESS (spec ready for planning)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Quick Guidelines
|
||||
- ✅ Focus on WHAT users need and WHY
|
||||
- ❌ Avoid HOW to implement (no tech stack, APIs, code structure)
|
||||
- 👥 Written for business stakeholders, not developers
|
||||
|
||||
### Section Requirements
|
||||
- **Mandatory sections**: Must be completed for every feature
|
||||
- **Optional sections**: Include only when relevant to the feature
|
||||
- When a section doesn't apply, remove it entirely (don't leave as "N/A")
|
||||
|
||||
### For AI Generation
|
||||
When creating this spec from a user prompt:
|
||||
1. **Mark all ambiguities**: Use [NEEDS CLARIFICATION: specific question] for any assumption you'd need to make
|
||||
2. **Don't guess**: If the prompt doesn't specify something (e.g., "login system" without auth method), mark it
|
||||
3. **Think like a tester**: Every vague requirement should fail the "testable and unambiguous" checklist item
|
||||
4. **Common underspecified areas**:
|
||||
- User types and permissions
|
||||
- Data retention/deletion policies
|
||||
- Performance targets and scale
|
||||
- Error handling behaviors
|
||||
- Integration requirements
|
||||
- Security/compliance needs
|
||||
|
||||
---
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### Primary User Story
|
||||
[Describe the main user journey in plain language]
|
||||
|
||||
### Acceptance Scenarios
|
||||
1. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
2. **Given** [initial state], **When** [action], **Then** [expected outcome]
|
||||
|
||||
### Edge Cases
|
||||
- What happens when [boundary condition]?
|
||||
- How does system handle [error scenario]?
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
- **FR-001**: System MUST [specific capability, e.g., "allow users to create accounts"]
|
||||
- **FR-002**: System MUST [specific capability, e.g., "validate email addresses"]
|
||||
- **FR-003**: Users MUST be able to [key interaction, e.g., "reset their password"]
|
||||
- **FR-004**: System MUST [data requirement, e.g., "persist user preferences"]
|
||||
- **FR-005**: System MUST [behavior, e.g., "log all security events"]
|
||||
|
||||
*Example of marking unclear requirements:*
|
||||
- **FR-006**: System MUST authenticate users via [NEEDS CLARIFICATION: auth method not specified - email/password, SSO, OAuth?]
|
||||
- **FR-007**: System MUST retain user data for [NEEDS CLARIFICATION: retention period not specified]
|
||||
|
||||
### Key Entities *(include if feature involves data)*
|
||||
- **[Entity 1]**: [What it represents, key attributes without implementation]
|
||||
- **[Entity 2]**: [What it represents, relationships to other entities]
|
||||
|
||||
---
|
||||
|
||||
## Review & Acceptance Checklist
|
||||
*GATE: Automated checks run during main() execution*
|
||||
|
||||
### Content Quality
|
||||
- [ ] No implementation details (languages, frameworks, APIs)
|
||||
- [ ] Focused on user value and business needs
|
||||
- [ ] Written for non-technical stakeholders
|
||||
- [ ] All mandatory sections completed
|
||||
|
||||
### Requirement Completeness
|
||||
- [ ] No [NEEDS CLARIFICATION] markers remain
|
||||
- [ ] Requirements are testable and unambiguous
|
||||
- [ ] Success criteria are measurable
|
||||
- [ ] Scope is clearly bounded
|
||||
- [ ] Dependencies and assumptions identified
|
||||
|
||||
---
|
||||
|
||||
## Execution Status
|
||||
*Updated by main() during processing*
|
||||
|
||||
- [ ] User description parsed
|
||||
- [ ] Key concepts extracted
|
||||
- [ ] Ambiguities marked
|
||||
- [ ] User scenarios defined
|
||||
- [ ] Requirements generated
|
||||
- [ ] Entities identified
|
||||
- [ ] Review checklist passed
|
||||
|
||||
---
|
||||
@@ -1,127 +0,0 @@
|
||||
# Tasks: [FEATURE NAME]
|
||||
|
||||
**Input**: Design documents from `/specs/[###-feature-name]/`
|
||||
**Prerequisites**: plan.md (required), research.md, data-model.md, contracts/
|
||||
|
||||
## Execution Flow (main)
|
||||
```
|
||||
1. Load plan.md from feature directory
|
||||
→ If not found: ERROR "No implementation plan found"
|
||||
→ Extract: tech stack, libraries, structure
|
||||
2. Load optional design documents:
|
||||
→ data-model.md: Extract entities → model tasks
|
||||
→ contracts/: Each file → contract test task
|
||||
→ research.md: Extract decisions → setup tasks
|
||||
3. Generate tasks by category:
|
||||
→ Setup: project init, dependencies, linting
|
||||
→ Tests: contract tests, integration tests
|
||||
→ Core: models, services, CLI commands
|
||||
→ Integration: DB, middleware, logging
|
||||
→ Polish: unit tests, performance, docs
|
||||
4. Apply task rules:
|
||||
→ Different files = mark [P] for parallel
|
||||
→ Same file = sequential (no [P])
|
||||
→ Tests before implementation (TDD)
|
||||
5. Number tasks sequentially (T001, T002...)
|
||||
6. Generate dependency graph
|
||||
7. Create parallel execution examples
|
||||
8. Validate task completeness:
|
||||
→ All contracts have tests?
|
||||
→ All entities have models?
|
||||
→ All endpoints implemented?
|
||||
9. Return: SUCCESS (tasks ready for execution)
|
||||
```
|
||||
|
||||
## Format: `[ID] [P?] Description`
|
||||
- **[P]**: Can run in parallel (different files, no dependencies)
|
||||
- Include exact file paths in descriptions
|
||||
|
||||
## Path Conventions
|
||||
- **Single project**: `src/`, `tests/` at repository root
|
||||
- **Web app**: `backend/src/`, `frontend/src/`
|
||||
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
|
||||
- Paths shown below assume single project - adjust based on plan.md structure
|
||||
|
||||
## Phase 3.1: Setup
|
||||
- [ ] T001 Create project structure per implementation plan
|
||||
- [ ] T002 Initialize [language] project with [framework] dependencies
|
||||
- [ ] T003 [P] Configure linting and formatting tools
|
||||
|
||||
## Phase 3.2: Tests First (TDD) ⚠️ MUST COMPLETE BEFORE 3.3
|
||||
**CRITICAL: These tests MUST be written and MUST FAIL before ANY implementation**
|
||||
- [ ] T004 [P] Contract test POST /api/users in tests/contract/test_users_post.py
|
||||
- [ ] T005 [P] Contract test GET /api/users/{id} in tests/contract/test_users_get.py
|
||||
- [ ] T006 [P] Integration test user registration in tests/integration/test_registration.py
|
||||
- [ ] T007 [P] Integration test auth flow in tests/integration/test_auth.py
|
||||
|
||||
## Phase 3.3: Core Implementation (ONLY after tests are failing)
|
||||
- [ ] T008 [P] User model in src/models/user.py
|
||||
- [ ] T009 [P] UserService CRUD in src/services/user_service.py
|
||||
- [ ] T010 [P] CLI --create-user in src/cli/user_commands.py
|
||||
- [ ] T011 POST /api/users endpoint
|
||||
- [ ] T012 GET /api/users/{id} endpoint
|
||||
- [ ] T013 Input validation
|
||||
- [ ] T014 Error handling and logging
|
||||
|
||||
## Phase 3.4: Integration
|
||||
- [ ] T015 Connect UserService to DB
|
||||
- [ ] T016 Auth middleware
|
||||
- [ ] T017 Request/response logging
|
||||
- [ ] T018 CORS and security headers
|
||||
|
||||
## Phase 3.5: Polish
|
||||
- [ ] T019 [P] Unit tests for validation in tests/unit/test_validation.py
|
||||
- [ ] T020 Performance tests (<200ms)
|
||||
- [ ] T021 [P] Update docs/api.md
|
||||
- [ ] T022 Remove duplication
|
||||
- [ ] T023 Run manual-testing.md
|
||||
|
||||
## Dependencies
|
||||
- Tests (T004-T007) before implementation (T008-T014)
|
||||
- T008 blocks T009, T015
|
||||
- T016 blocks T018
|
||||
- Implementation before polish (T019-T023)
|
||||
|
||||
## Parallel Example
|
||||
```
|
||||
# Launch T004-T007 together:
|
||||
Task: "Contract test POST /api/users in tests/contract/test_users_post.py"
|
||||
Task: "Contract test GET /api/users/{id} in tests/contract/test_users_get.py"
|
||||
Task: "Integration test registration in tests/integration/test_registration.py"
|
||||
Task: "Integration test auth in tests/integration/test_auth.py"
|
||||
```
|
||||
|
||||
## Notes
|
||||
- [P] tasks = different files, no dependencies
|
||||
- Verify tests fail before implementing
|
||||
- Commit after each task
|
||||
- Avoid: vague tasks, same file conflicts
|
||||
|
||||
## Task Generation Rules
|
||||
*Applied during main() execution*
|
||||
|
||||
1. **From Contracts**:
|
||||
- Each contract file → contract test task [P]
|
||||
- Each endpoint → implementation task
|
||||
|
||||
2. **From Data Model**:
|
||||
- Each entity → model creation task [P]
|
||||
- Relationships → service layer tasks
|
||||
|
||||
3. **From User Stories**:
|
||||
- Each story → integration test [P]
|
||||
- Quickstart scenarios → validation tasks
|
||||
|
||||
4. **Ordering**:
|
||||
- Setup → Tests → Models → Services → Endpoints → Polish
|
||||
- Dependencies block parallel execution
|
||||
|
||||
## Validation Checklist
|
||||
*GATE: Checked by main() before returning*
|
||||
|
||||
- [ ] All contracts have corresponding tests
|
||||
- [ ] All entities have model tasks
|
||||
- [ ] All tests come before implementation
|
||||
- [ ] Parallel tasks truly independent
|
||||
- [ ] Each task specifies exact file path
|
||||
- [ ] No task modifies same file as another [P] task
|
||||
@@ -1,187 +0,0 @@
|
||||
# Karaoke-Gen AI Assistant Guidelines
|
||||
|
||||
## Project Overview
|
||||
|
||||
**Karaoke video generation platform** - CLI tool and web service that creates professional karaoke videos with synchronized lyrics.
|
||||
|
||||
- **Production**: <https://gen.nomadkaraoke.com> (frontend), <https://api.nomadkaraoke.com> (backend)
|
||||
- **CLI**: `karaoke-gen` (local), `karaoke-gen-remote` (cloud)
|
||||
- **Repo**: <https://github.com/nomadkaraoke/karaoke-gen>
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| What | Where |
|
||||
|------|-------|
|
||||
| Product vision & goals | `docs/PRODUCT-VISION.md` |
|
||||
| Current status | `docs/README.md` |
|
||||
| Architecture | `docs/ARCHITECTURE.md` |
|
||||
| Dev setup & testing | `docs/DEVELOPMENT.md` |
|
||||
| **Testing & code quality** | `docs/TESTING.md` |
|
||||
| API reference | `docs/API.md` |
|
||||
| Past learnings | `docs/LESSONS-LEARNED.md` |
|
||||
| **Product communication** | `docs/PRODUCT-COMMUNICATION-GUIDE.md` |
|
||||
| Brand style guide | `docs/BRAND-STYLE-GUIDE.md` |
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Backend**: FastAPI on Cloud Run, Firestore, GCS, Secret Manager
|
||||
- **Frontend**: Next.js on Cloudflare Pages
|
||||
- **Processing**: Modal (GPU audio separation), AudioShake/Whisper (transcription)
|
||||
- **Infra**: Pulumi IaC, GitHub Actions CI/CD
|
||||
|
||||
## Essential Rules
|
||||
|
||||
### Git Workflow
|
||||
- **Never commit directly to `main`** - use `/new-worktree <description>` to start
|
||||
- **Follow global workflow** - see `~/.claude/CLAUDE.md` for command sequence
|
||||
- Create PR with summary, changes, testing info
|
||||
- Merge only after CI passes
|
||||
|
||||
### Testing (Required)
|
||||
**Before planning any implementation work**, read `docs/TESTING.md` and ensure your plan includes a thorough testing strategy that follows the guidance there. This includes:
|
||||
- Which test types are needed (unit, integration, E2E)
|
||||
- Where tests should be placed
|
||||
- What mocking approach to use
|
||||
- Coverage expectations
|
||||
- **Production E2E tests for user-facing features** - see below
|
||||
|
||||
**Critical:** If your plan includes "Manual Testing" steps, convert them to automated Playwright E2E tests. Production E2E tests (in `frontend/e2e/production/`) are valuable even if only run once after deployment - they codify expected behavior and verify the deployed code works. See `docs/TESTING.md` for details.
|
||||
|
||||
**Before committing**, run all tests:
|
||||
```bash
|
||||
make test 2>&1 | tail -n 500
|
||||
```
|
||||
|
||||
This single command:
|
||||
- Installs dependencies automatically (backend + frontend)
|
||||
- Runs all backend tests (unit, integration, emulator)
|
||||
- Runs all frontend tests (Jest unit + Playwright E2E)
|
||||
- Takes ~5-10 minutes
|
||||
|
||||
**All tests must pass.** Don't dismiss failures as "pre-existing" - investigate and fix them.
|
||||
|
||||
### Testing in Production (for Agents)
|
||||
|
||||
When asked to "test it yourself in prod" or verify a fix in production:
|
||||
|
||||
```bash
|
||||
# Get admin token
|
||||
export KARAOKE_ADMIN_TOKEN=$(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke | cut -d',' -f1)
|
||||
|
||||
# Option 1: Use the debug template directly
|
||||
cd frontend && node e2e/helpers/debug-prod-template.mjs
|
||||
|
||||
# Option 2: Copy template for customization (gitignored)
|
||||
cp frontend/e2e/helpers/debug-prod-template.mjs frontend/test-my-issue.local.mjs
|
||||
# Edit the script, then run it
|
||||
node test-my-issue.local.mjs
|
||||
|
||||
# Option 3: Run existing production E2E tests
|
||||
KARAOKE_ADMIN_TOKEN=$KARAOKE_ADMIN_TOKEN npx playwright test e2e/production/admin-dashboard.spec.ts
|
||||
```
|
||||
|
||||
Files matching `test-*.local.*` and `debug-*.local.*` are gitignored - you can hardcode tokens in them safely.
|
||||
|
||||
See `docs/TESTING.md` § "Ad-Hoc Production Debugging" for full details.
|
||||
|
||||
### Version Bumping
|
||||
- Bump `tool.poetry.version` in `pyproject.toml` for code changes
|
||||
- Skip for docs-only changes
|
||||
|
||||
### Infrastructure
|
||||
- **All GCP changes via Pulumi PRs** - changes in `infrastructure/` deploy automatically on merge to main
|
||||
- Never modify GCP resources directly via console or `gcloud` CLI
|
||||
- `gcloud` CLI for reading/debugging only (e.g., checking logs, SSH to VMs)
|
||||
- Local `pulumi preview` works but `pulumi up` requires broader permissions than CI
|
||||
- Stop and notify user on auth issues
|
||||
|
||||
## API Authentication (for Agents)
|
||||
|
||||
When you need to call backend APIs programmatically:
|
||||
|
||||
### Internal Worker Endpoints
|
||||
Use the `X-Admin-Token` header with the token from Secret Manager:
|
||||
```bash
|
||||
# Trigger video worker
|
||||
curl -X POST "https://api.nomadkaraoke.com/api/internal/workers/video" \
|
||||
-H "X-Admin-Token: $(gcloud secrets versions access latest --secret=admin-tokens --project=nomadkaraoke)" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"job_id": "YOUR_JOB_ID"}'
|
||||
|
||||
# Other internal endpoints follow the same pattern
|
||||
```
|
||||
|
||||
### Firestore Direct Access
|
||||
```bash
|
||||
# Set the correct project
|
||||
export GOOGLE_CLOUD_PROJECT=nomadkaraoke
|
||||
|
||||
# Use Python with google-cloud-firestore
|
||||
python3 << 'EOF'
|
||||
import os
|
||||
os.environ['GOOGLE_CLOUD_PROJECT'] = 'nomadkaraoke'
|
||||
from google.cloud import firestore
|
||||
db = firestore.Client(project='nomadkaraoke')
|
||||
# Query/update documents...
|
||||
EOF
|
||||
```
|
||||
|
||||
### GCE Encoding Worker
|
||||
```bash
|
||||
# SSH to restart service (clears in-memory job queue)
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="sudo systemctl restart encoding-worker"
|
||||
|
||||
# Check health
|
||||
gcloud compute ssh encoding-worker --zone=us-central1-c --project=nomadkaraoke \
|
||||
--command="curl -s http://localhost:8080/health"
|
||||
```
|
||||
|
||||
### What Doesn't Work
|
||||
- `gcloud auth print-identity-token` - Wrong token type for internal endpoints
|
||||
- Cloud Tasks without proper service account - Gets 401 from Cloud Run
|
||||
|
||||
## Documentation Maintenance
|
||||
|
||||
### Structure
|
||||
```text
|
||||
docs/
|
||||
├── README.md # Current status + navigation (UPDATE THIS)
|
||||
├── ARCHITECTURE.md # System design
|
||||
├── DEVELOPMENT.md # Dev setup, testing, deployment
|
||||
├── API.md # Backend API reference
|
||||
├── LESSONS-LEARNED.md # Key insights for future agents
|
||||
└── archive/ # Historical docs (YYYY-MM-DD-topic.md)
|
||||
```
|
||||
|
||||
### Rules
|
||||
1. **Before merging PRs**: Run `/docs-review` to check if docs need updating
|
||||
2. **Periodically**: Run `/docs-maintain` to verify docs are organized
|
||||
3. **For significant work**: Create `docs/archive/YYYY-MM-DD-topic.md`
|
||||
4. **Add learnings**: Update `docs/LESSONS-LEARNED.md` with insights
|
||||
5. **Keep status current**: Update `docs/README.md` when project state changes
|
||||
|
||||
### What Goes Where
|
||||
- **README.md**: Current status, known issues, quick links
|
||||
- **ARCHITECTURE.md**: System design, data flow, tech decisions
|
||||
- **DEVELOPMENT.md**: Setup, testing, deployment, CI/CD
|
||||
- **API.md**: Endpoints, request/response formats
|
||||
- **LESSONS-LEARNED.md**: What worked, what didn't, gotchas
|
||||
- **archive/**: Completed work, session summaries, old plans
|
||||
|
||||
## Related Projects
|
||||
|
||||
### flacfetch
|
||||
Location: `/Users/andrew/Projects/flacfetch`
|
||||
|
||||
Workflow: Make changes on `main`, bump version, push, wait for CI, then `poetry update flacfetch` in karaoke-gen.
|
||||
|
||||
## Pre-PR Checklist
|
||||
|
||||
Follow global workflow (`~/.claude/CLAUDE.md`):
|
||||
|
||||
- [ ] Tests pass: `/test` (or `make test` / `npm run test:all`)
|
||||
- [ ] Version bumped (if code changed)
|
||||
- [ ] Docs checked: `/docs-review`
|
||||
- [ ] Code reviewed: `/review` (CodeRabbit CLI)
|
||||
- [ ] PR created: `/pr` (adds @coderabbitai ignore)
|
||||
@@ -1,6 +1,6 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Nomad Karaoke LLC
|
||||
Copyright (c) 2025 Nomad Karaoke LLC
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
|
||||
@@ -1,112 +0,0 @@
|
||||
# Makefile for karaoke-gen project
|
||||
# Run `make help` to see available commands
|
||||
|
||||
.PHONY: help install install-backend install-frontend build-frontend test test-unit test-backend test-e2e test-frontend test-all lint clean emulators-start emulators-stop
|
||||
|
||||
# Default target
|
||||
help:
|
||||
@echo "Available commands:"
|
||||
@echo " make test - Run ALL tests (backend + frontend, installs deps automatically)"
|
||||
@echo " make test-backend - Run backend tests only (unit + emulator)"
|
||||
@echo " make test-frontend - Run frontend tests only (unit + E2E)"
|
||||
@echo " make test-unit - Run unit tests only (karaoke_gen package)"
|
||||
@echo " make test-e2e - Run emulator tests with auto-start/stop"
|
||||
@echo " make install - Install all dependencies (backend + frontend)"
|
||||
@echo " make build-frontend - Build frontend and copy to Python package (for local testing)"
|
||||
@echo " make lint - Run linter checks"
|
||||
@echo " make emulators-start - Start GCP emulators for local development"
|
||||
@echo " make emulators-stop - Stop GCP emulators"
|
||||
@echo ""
|
||||
@echo "Before committing, run: make test"
|
||||
|
||||
# Install dependencies (only if needed)
|
||||
install-backend:
|
||||
@if [ ! -d "$$(poetry env info --path 2>/dev/null)" ] || ! poetry run python -c "import fastapi" 2>/dev/null; then \
|
||||
echo "=== Installing backend dependencies ==="; \
|
||||
poetry install; \
|
||||
fi
|
||||
|
||||
install-frontend:
|
||||
@if [ ! -d "frontend/node_modules" ]; then \
|
||||
echo "=== Installing frontend dependencies ==="; \
|
||||
cd frontend && npm install; \
|
||||
fi
|
||||
|
||||
install: install-backend install-frontend
|
||||
|
||||
# Build frontend and copy to Python package (for local CLI testing)
|
||||
build-frontend: install-frontend
|
||||
@echo "=== Building frontend ==="
|
||||
cd frontend && npm run build
|
||||
@echo "=== Copying build to Python package ==="
|
||||
cp -r frontend/out/* karaoke_gen/nextjs_frontend/out/
|
||||
@echo "✅ Frontend built and ready for local testing"
|
||||
|
||||
# Run unit tests for karaoke_gen package
|
||||
test-unit: install-backend
|
||||
@echo "=== Running karaoke_gen unit tests ==="
|
||||
poetry run pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing --cov-fail-under=69
|
||||
|
||||
# Run backend unit tests (excludes emulator tests)
|
||||
test-backend-unit: install-backend
|
||||
@echo "=== Running backend unit tests ==="
|
||||
poetry run pytest backend/tests/ --ignore=backend/tests/emulator -v
|
||||
|
||||
# Run E2E integration tests with emulators
|
||||
test-e2e: install-backend
|
||||
@echo "=== Running E2E integration tests with emulators ==="
|
||||
@./scripts/start-emulators.sh || (echo "Failed to start emulators" && exit 1)
|
||||
@export FIRESTORE_EMULATOR_HOST=127.0.0.1:8080 && \
|
||||
export STORAGE_EMULATOR_HOST=http://127.0.0.1:4443 && \
|
||||
export GOOGLE_CLOUD_PROJECT=test-project && \
|
||||
export GCS_BUCKET_NAME=test-bucket && \
|
||||
export ADMIN_TOKENS=test-admin-token && \
|
||||
export ENVIRONMENT=test && \
|
||||
poetry run pytest backend/tests/emulator/ -v; \
|
||||
TEST_RESULT=$$?; \
|
||||
./scripts/stop-emulators.sh; \
|
||||
exit $$TEST_RESULT
|
||||
|
||||
# Run all backend tests (unit + emulator)
|
||||
test-backend: test-unit test-backend-unit test-e2e
|
||||
@echo ""
|
||||
@echo "✅ Backend tests passed!"
|
||||
|
||||
# Run frontend tests (unit + E2E)
|
||||
test-frontend: install-frontend
|
||||
@echo "=== Running frontend tests ==="
|
||||
cd frontend && npm run test:all
|
||||
|
||||
# Run ALL tests (backend + frontend) - use this before committing!
|
||||
test: test-backend test-frontend
|
||||
@echo ""
|
||||
@echo "✅ All tests passed! Ready to commit."
|
||||
|
||||
# Alias for test
|
||||
test-all: test
|
||||
|
||||
# Lint checks
|
||||
lint:
|
||||
@echo "=== Running lint checks ==="
|
||||
poetry run ruff check karaoke_gen/ backend/
|
||||
poetry run ruff format --check karaoke_gen/ backend/
|
||||
|
||||
# Start emulators for local development
|
||||
emulators-start:
|
||||
@echo "=== Starting GCP emulators ==="
|
||||
./scripts/start-emulators.sh
|
||||
|
||||
# Stop emulators
|
||||
emulators-stop:
|
||||
@echo "=== Stopping GCP emulators ==="
|
||||
./scripts/stop-emulators.sh
|
||||
|
||||
# Clean up temporary files
|
||||
clean:
|
||||
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type d -name ".coverage" -exec rm -rf {} + 2>/dev/null || true
|
||||
find . -type f -name "*.pyc" -delete 2>/dev/null || true
|
||||
rm -rf htmlcov/ .coverage coverage.xml 2>/dev/null || true
|
||||
@echo "Cleaned up temporary files"
|
||||
|
||||
@@ -1,678 +1,144 @@
|
||||
# Karaoke Generator 🎶 🎥 🚀
|
||||
# Karaoke Gen
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||
Generate karaoke videos with synchronized lyrics. Handles the entire process from downloading audio and lyrics to creating the final video with title screens.
|
||||
|
||||
Generate professional karaoke videos with instrumental audio and synchronized lyrics. Available as a **local CLI** (`karaoke-gen`) or **cloud-based CLI** (`karaoke-gen-remote`) that offloads processing to Google Cloud.
|
||||
## Overview
|
||||
|
||||
## ✨ Two Ways to Generate Karaoke
|
||||
Karaoke Gen is a comprehensive tool for creating high-quality karaoke videos. It automates the entire workflow:
|
||||
|
||||
### 1. Local CLI (`karaoke-gen`)
|
||||
Run all processing locally on your machine. Requires GPU for optimal audio separation performance.
|
||||
1. **Download** audio and lyrics for a specified song
|
||||
2. **Separate** audio stems (vocals, instrumental)
|
||||
3. **Synchronize** lyrics with the audio
|
||||
4. **Generate** title and end screens
|
||||
5. **Combine** everything into a polished final video
|
||||
6. **Organize** and **share** the output files
|
||||
|
||||
```bash
|
||||
karaoke-gen "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
### 2. Remote CLI (`karaoke-gen-remote`)
|
||||
Offload all processing to a cloud backend. No GPU required - just authenticate and submit jobs.
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
```
|
||||
|
||||
Both CLIs produce identical outputs: 4K karaoke videos, CDG+MP3 packages, audio stems, and more.
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Features
|
||||
|
||||
### Core Pipeline
|
||||
- **Audio Separation**: AI-powered vocal/instrumental separation using MDX and Demucs models
|
||||
- **Lyrics Transcription**: Word-level timestamps via AudioShake API
|
||||
- **Lyrics Correction**: Match transcription against online lyrics (Genius, Spotify, Musixmatch)
|
||||
- **Human Review**: Interactive UI for correcting lyrics before final render
|
||||
- **Video Rendering**: High-quality 4K karaoke videos with customizable styles
|
||||
- **Multiple Outputs**: MP4 (4K lossless/lossy, 720p), MKV, CDG+MP3, TXT+MP3
|
||||
|
||||
### Distribution Features
|
||||
- **YouTube Upload**: Automatic upload to your YouTube channel
|
||||
- **Dropbox Integration**: Organize output in brand-coded folders
|
||||
- **Google Drive**: Upload to public share folders
|
||||
- **Discord Notifications**: Webhook notifications on completion
|
||||
|
||||
---
|
||||
|
||||
## 📦 Installation
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install karaoke-gen
|
||||
```
|
||||
|
||||
This installs both `karaoke-gen` (local) and `karaoke-gen-remote` (cloud) CLIs.
|
||||
|
||||
### Requirements
|
||||
- Python 3.10-3.13
|
||||
- FFmpeg
|
||||
- For local processing: CUDA-capable GPU or Apple Silicon CPU recommended
|
||||
|
||||
### Transcription Provider Setup
|
||||
|
||||
**Transcription is required** for creating karaoke videos with synchronized lyrics. The system needs word-level timing data to display lyrics in sync with the music.
|
||||
|
||||
#### Option 1: AudioShake (Recommended)
|
||||
Commercial service with high-quality transcription. Best for production use.
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
export AUDIOSHAKE_API_TOKEN="your_audioshake_token"
|
||||
```
|
||||
# Generate a karaoke video from a YouTube URL
|
||||
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
Get an API key at [https://www.audioshake.ai/](https://www.audioshake.ai/) - business only, at time of writing this.
|
||||
|
||||
#### Option 2: Local Whisper (No Cloud Required)
|
||||
Run Whisper directly on your local machine using whisper-timestamped. Works on CPU, NVIDIA GPU (CUDA), or Apple Silicon.
|
||||
|
||||
```bash
|
||||
# Install with local Whisper support
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
|
||||
# Optional: Configure model size (tiny, base, small, medium, large)
|
||||
export WHISPER_MODEL_SIZE="medium"
|
||||
|
||||
# Optional: Force specific device (cpu, cuda, mps)
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
**Model Size Guide:**
|
||||
| Model | VRAM | Speed | Quality |
|
||||
|-------|------|-------|---------|
|
||||
| tiny | ~1GB | Fast | Lower |
|
||||
| base | ~1GB | Fast | Basic |
|
||||
| small | ~2GB | Medium | Good |
|
||||
| medium | ~5GB | Slower | Better |
|
||||
| large | ~10GB | Slowest | Best |
|
||||
|
||||
**CPU-Only Installation** (no GPU required):
|
||||
```bash
|
||||
# Pre-install CPU-only PyTorch first
|
||||
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
```
|
||||
|
||||
Local Whisper runs automatically as a fallback when no cloud transcription services are configured.
|
||||
|
||||
#### Option 3: Whisper via RunPod
|
||||
Cloud-based alternative using OpenAI's Whisper model on RunPod infrastructure.
|
||||
|
||||
```bash
|
||||
export RUNPOD_API_KEY="your_runpod_key"
|
||||
export WHISPER_RUNPOD_ID="your_whisper_endpoint_id"
|
||||
```
|
||||
|
||||
Set up a Whisper endpoint at [https://www.runpod.io/](https://www.runpod.io/)
|
||||
|
||||
#### Without Transcription (Instrumental Only)
|
||||
If you don't need synchronized lyrics, use the `--skip-lyrics` flag:
|
||||
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
This creates an instrumental-only karaoke video without lyrics overlay.
|
||||
|
||||
> **Note:** See `lyrics_transcriber_temp/README.md` for detailed transcription provider configuration options.
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ Local CLI (`karaoke-gen`)
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Generate from local audio file
|
||||
karaoke-gen ./song.mp3 "Artist Name" "Song Title"
|
||||
|
||||
# Search and download audio automatically
|
||||
# Or let it search YouTube for you
|
||||
karaoke-gen "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Process from YouTube URL
|
||||
karaoke-gen "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "Rick Astley" "Never Gonna Give You Up"
|
||||
# Process multiple tracks in bulk from a CSV file
|
||||
karaoke-bulk input.csv --style_params_json=style.json
|
||||
```
|
||||
|
||||
### Remote Audio Separation (Optional)
|
||||
## Workflow Options
|
||||
|
||||
Offload just the GPU-intensive audio separation to Modal.com while keeping other processing local:
|
||||
Karaoke Gen supports different workflow options to fit your needs:
|
||||
|
||||
```bash
|
||||
export AUDIO_SEPARATOR_API_URL="https://USERNAME--audio-separator-api.modal.run"
|
||||
karaoke-gen "Artist" "Title"
|
||||
# Run only the preparation phase (download, separate stems, create title screens)
|
||||
karaoke-gen --prep-only "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Run only the finalisation phase (must be run in a directory prepared by the prep phase)
|
||||
karaoke-gen --finalise-only
|
||||
|
||||
# Skip automatic lyrics transcription/synchronization (for manual syncing)
|
||||
karaoke-gen --skip-transcription "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Skip audio separation (if you already have instrumental)
|
||||
karaoke-gen --skip-separation --existing-instrumental="path/to/instrumental.mp3" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Key Options
|
||||
## Advanced Features
|
||||
|
||||
### Audio Processing
|
||||
|
||||
```bash
|
||||
# Custom styling
|
||||
karaoke-gen --style_params_json="./styles.json" "Artist" "Title"
|
||||
|
||||
# Generate CDG and TXT packages
|
||||
karaoke-gen --enable_cdg --enable_txt "Artist" "Title"
|
||||
|
||||
# Skip video encoding (CDG/TXT only, faster)
|
||||
karaoke-gen --no-video --enable_cdg "Artist" "Title"
|
||||
|
||||
# YouTube upload
|
||||
karaoke-gen --enable_youtube_upload --youtube_description_file="./desc.txt" "Artist" "Title"
|
||||
|
||||
# Full production run
|
||||
karaoke-gen \
|
||||
--style_params_json="./branding.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix="BRAND" \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./description.txt" \
|
||||
"Artist" "Title"
|
||||
# Specify custom audio separation models
|
||||
karaoke-gen --clean_instrumental_model="model_name.ckpt" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Full Options Reference
|
||||
### Lyrics Handling
|
||||
|
||||
```bash
|
||||
# Use a local lyrics file instead of fetching from online
|
||||
karaoke-gen --lyrics_file="path/to/lyrics.txt" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Adjust subtitle timing
|
||||
karaoke-gen --subtitle_offset_ms=500 "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
### Finalisation Options
|
||||
|
||||
```bash
|
||||
# Enable CDG ZIP generation
|
||||
karaoke-gen --enable_cdg --style_params_json="path/to/style.json" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Enable TXT ZIP generation
|
||||
karaoke-gen --enable_txt "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Upload to YouTube
|
||||
karaoke-gen --youtube_client_secrets_file="path/to/client_secret.json" --youtube_description_file="path/to/description.txt" "Rick Astley" "Never Gonna Give You Up"
|
||||
|
||||
# Organize files with brand code
|
||||
karaoke-gen --brand_prefix="BRAND" --organised_dir="path/to/Tracks-Organized" "Rick Astley" "Never Gonna Give You Up"
|
||||
```
|
||||
|
||||
## Python API
|
||||
|
||||
Karaoke Gen can also be used as a Python library:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from karaoke_gen import KaraokeController, ProjectConfig
|
||||
|
||||
async def generate_karaoke():
|
||||
# Create a configuration
|
||||
config = ProjectConfig(
|
||||
artist="Rick Astley",
|
||||
title="Never Gonna Give You Up",
|
||||
input_media="https://www.youtube.com/watch?v=dQw4w9WgXcQ",
|
||||
output_dir="output",
|
||||
create_track_subfolders=True,
|
||||
)
|
||||
|
||||
# Create a controller
|
||||
controller = KaraokeController(config)
|
||||
|
||||
# Process the track
|
||||
tracks = await controller.process()
|
||||
|
||||
# Print the output files
|
||||
for track in tracks:
|
||||
print(f"Final video: {track.final_video}")
|
||||
|
||||
# Run the async function
|
||||
asyncio.run(generate_karaoke())
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
Karaoke Gen has a modular architecture with the following components:
|
||||
|
||||
- **Controller**: Orchestrates the entire process
|
||||
- **Services**: Handle specific aspects of the process
|
||||
- **Media Service**: Downloads and extracts media
|
||||
- **Audio Service**: Separates and processes audio
|
||||
- **Lyrics Service**: Fetches and synchronizes lyrics
|
||||
- **Video Service**: Renders video with synchronized lyrics
|
||||
- **Distribution Service**: Organizes and shares the output files
|
||||
- **Utilities**: Common utility functions for file handling, logging, etc.
|
||||
|
||||
## Full Command Reference
|
||||
|
||||
For a complete list of options:
|
||||
|
||||
```bash
|
||||
karaoke-gen --help
|
||||
karaoke-bulk --help
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ☁️ Remote CLI (`karaoke-gen-remote`)
|
||||
|
||||
The remote CLI submits jobs to a Google Cloud backend that handles all processing. You don't need a GPU or any audio processing libraries installed locally.
|
||||
|
||||
### Setup
|
||||
|
||||
1. **Set the backend URL:**
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://api.nomadkaraoke.com" # Or your own backend
|
||||
```
|
||||
|
||||
2. **Authenticate with Google Cloud:**
|
||||
```bash
|
||||
gcloud auth login
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Submit a job
|
||||
karaoke-gen-remote ./song.flac "ABBA" "Waterloo"
|
||||
|
||||
# The CLI will:
|
||||
# 1. Upload your audio file
|
||||
# 2. Monitor processing progress
|
||||
# 3. Open lyrics review UI when ready
|
||||
# 4. Prompt for instrumental selection
|
||||
# 5. Download all outputs when complete
|
||||
```
|
||||
|
||||
### Job Management
|
||||
|
||||
```bash
|
||||
# List all jobs
|
||||
karaoke-gen-remote --list
|
||||
|
||||
# Resume monitoring an existing job
|
||||
karaoke-gen-remote --resume abc12345
|
||||
|
||||
# Cancel a running job
|
||||
karaoke-gen-remote --cancel abc12345
|
||||
|
||||
# Delete a job and its files
|
||||
karaoke-gen-remote --delete abc12345
|
||||
```
|
||||
|
||||
### Full Production Run
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote \
|
||||
--style_params_json="./karaoke-styles.json" \
|
||||
--enable_cdg \
|
||||
--enable_txt \
|
||||
--brand_prefix=NOMAD \
|
||||
--enable_youtube_upload \
|
||||
--youtube_description_file="./youtube-description.txt" \
|
||||
./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `KARAOKE_GEN_URL` | Backend service URL | Required |
|
||||
| `KARAOKE_GEN_AUTH_TOKEN` | Admin auth token (for protected endpoints) | Optional |
|
||||
| `REVIEW_UI_URL` | Lyrics review UI URL | `https://gen.nomadkaraoke.com/lyrics/` |
|
||||
| `POLL_INTERVAL` | Seconds between status polls | `5` |
|
||||
|
||||
**Note:** The `REVIEW_UI_URL` defaults to the hosted lyrics review UI. For local development, set it to `http://localhost:5173` if you're running the frontend dev server.
|
||||
|
||||
### Authentication
|
||||
|
||||
The backend uses token-based authentication for admin operations (bulk delete, internal worker triggers). For basic job submission and monitoring, authentication is optional.
|
||||
|
||||
**For admin access:**
|
||||
```bash
|
||||
export KARAOKE_GEN_AUTH_TOKEN="your-admin-token"
|
||||
```
|
||||
|
||||
The token must match one of the tokens configured in the backend's `ADMIN_TOKENS` environment variable.
|
||||
|
||||
### Non-Interactive Mode
|
||||
|
||||
For automated/CI usage:
|
||||
|
||||
```bash
|
||||
karaoke-gen-remote -y ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
The `-y` flag auto-accepts default corrections and selects clean instrumental.
|
||||
|
||||
---
|
||||
|
||||
## 🎨 Style Configuration
|
||||
|
||||
Create a `styles.json` file to customize the karaoke video appearance:
|
||||
|
||||
```json
|
||||
{
|
||||
"intro": {
|
||||
"video_duration": 5,
|
||||
"background_image": "/path/to/title-background.png",
|
||||
"font": "/path/to/Font.ttf",
|
||||
"artist_color": "#ffdf6b",
|
||||
"title_color": "#ffffff"
|
||||
},
|
||||
"karaoke": {
|
||||
"background_image": "/path/to/karaoke-background.png",
|
||||
"font_path": "/path/to/Font.ttf"
|
||||
},
|
||||
"end": {
|
||||
"background_image": "/path/to/end-background.png"
|
||||
},
|
||||
"cdg": {
|
||||
"font_path": "/path/to/Font.ttf",
|
||||
"instrumental_background": "/path/to/cdg-background.png"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When using `karaoke-gen-remote`, all referenced files are automatically uploaded with your job.
|
||||
|
||||
---
|
||||
|
||||
## 📤 Output Files
|
||||
|
||||
A completed job produces:
|
||||
|
||||
```
|
||||
BRAND-1234 - Artist - Title/
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mp4 # ProRes 4K
|
||||
├── Artist - Title (Final Karaoke Lossless 4k).mkv # FLAC audio 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 4k).mp4 # H.264 4K
|
||||
├── Artist - Title (Final Karaoke Lossy 720p).mp4 # H.264 720p
|
||||
├── Artist - Title (Final Karaoke CDG).zip # CDG+MP3 package
|
||||
├── Artist - Title (Final Karaoke TXT).zip # TXT+MP3 package
|
||||
├── Artist - Title (Karaoke).cdg # Individual CDG
|
||||
├── Artist - Title (Karaoke).mp3 # Karaoke audio
|
||||
├── Artist - Title (Karaoke).lrc # LRC lyrics
|
||||
├── Artist - Title (Karaoke).ass # ASS subtitles
|
||||
├── Artist - Title (Title).mov # Title screen video
|
||||
├── Artist - Title (End).mov # End screen video
|
||||
├── Artist - Title (Instrumental...).flac # Clean instrumental
|
||||
├── Artist - Title (Instrumental +BV...).flac # With backing vocals
|
||||
└── stems/ # All audio stems
|
||||
├── ...Vocals....flac
|
||||
├── ...Bass....flac
|
||||
├── ...Drums....flac
|
||||
└── ...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Deploy Your Own Backend
|
||||
|
||||
The cloud backend runs on Google Cloud Platform using:
|
||||
- **Cloud Run**: Serverless API hosting
|
||||
- **Firestore**: Job state management
|
||||
- **Cloud Storage**: File uploads and outputs
|
||||
- **Modal.com**: GPU-accelerated audio separation
|
||||
- **AudioShake**: Lyrics transcription API
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Google Cloud account with billing enabled
|
||||
- [Pulumi CLI](https://www.pulumi.com/docs/install/)
|
||||
- Modal.com account (for audio separation)
|
||||
- AudioShake API key
|
||||
|
||||
### Infrastructure Setup
|
||||
|
||||
```bash
|
||||
cd infrastructure
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Login to Pulumi
|
||||
pulumi login
|
||||
|
||||
# Create a stack
|
||||
pulumi stack init prod
|
||||
|
||||
# Configure GCP project
|
||||
pulumi config set gcp:project your-project-id
|
||||
pulumi config set gcp:region us-central1
|
||||
|
||||
# Deploy infrastructure
|
||||
pulumi up
|
||||
```
|
||||
|
||||
This creates:
|
||||
- Firestore database
|
||||
- Cloud Storage bucket
|
||||
- Artifact Registry
|
||||
- Service account with IAM roles
|
||||
- Secret Manager secrets (you add values)
|
||||
|
||||
### Add Secret Values
|
||||
|
||||
```bash
|
||||
# AudioShake API key
|
||||
echo -n "your-audioshake-key" | gcloud secrets versions add audioshake-api-key --data-file=-
|
||||
|
||||
# Genius API key
|
||||
echo -n "your-genius-key" | gcloud secrets versions add genius-api-key --data-file=-
|
||||
|
||||
# Modal API URL
|
||||
echo -n "https://your-modal-url" | gcloud secrets versions add audio-separator-api-url --data-file=-
|
||||
|
||||
# YouTube OAuth credentials (JSON)
|
||||
gcloud secrets versions add youtube-oauth-credentials --data-file=./youtube-creds.json
|
||||
|
||||
# Dropbox OAuth credentials (JSON)
|
||||
gcloud secrets versions add dropbox-oauth-credentials --data-file=./dropbox-creds.json
|
||||
|
||||
# Google Drive service account (JSON)
|
||||
gcloud secrets versions add gdrive-service-account --data-file=./gdrive-sa.json
|
||||
```
|
||||
|
||||
### Deploy Cloud Run
|
||||
|
||||
Deployments happen automatically via GitHub Actions CI when pushing to `main`.
|
||||
See `.github/workflows/ci.yml` for the full deployment workflow.
|
||||
|
||||
### Point CLI to Your Backend
|
||||
|
||||
```bash
|
||||
export KARAOKE_GEN_URL="https://your-backend.run.app"
|
||||
karaoke-gen-remote ./song.flac "Artist" "Title"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔌 Backend API Reference
|
||||
|
||||
The backend exposes a REST API for job management.
|
||||
|
||||
### Job Submission
|
||||
|
||||
**POST** `/api/jobs/upload`
|
||||
|
||||
Submit a new karaoke generation job with audio file and options.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/upload" \
|
||||
-F "file=@song.flac" \
|
||||
-F "artist=ABBA" \
|
||||
-F "title=Waterloo" \
|
||||
-F "enable_cdg=true" \
|
||||
-F "enable_txt=true" \
|
||||
-F "brand_prefix=NOMAD" \
|
||||
-F "style_params=@styles.json" \
|
||||
-F "style_karaoke_background=@background.png"
|
||||
```
|
||||
|
||||
### Job Status
|
||||
|
||||
**GET** `/api/jobs/{job_id}`
|
||||
|
||||
Get job status and details.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs/abc12345"
|
||||
```
|
||||
|
||||
### List Jobs
|
||||
|
||||
**GET** `/api/jobs`
|
||||
|
||||
List all jobs with optional status filter.
|
||||
|
||||
```bash
|
||||
curl "https://api.example.com/api/jobs?status=complete&limit=10"
|
||||
```
|
||||
|
||||
### Cancel Job
|
||||
|
||||
**POST** `/api/jobs/{job_id}/cancel`
|
||||
|
||||
Cancel a running job.
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/cancel" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"reason": "User cancelled"}'
|
||||
```
|
||||
|
||||
### Delete Job
|
||||
|
||||
**DELETE** `/api/jobs/{job_id}`
|
||||
|
||||
Delete a job and its files.
|
||||
|
||||
```bash
|
||||
curl -X DELETE "https://api.example.com/api/jobs/abc12345?delete_files=true"
|
||||
```
|
||||
|
||||
### Lyrics Review
|
||||
|
||||
**GET** `/api/review/{job_id}/correction-data`
|
||||
|
||||
Get correction data for lyrics review.
|
||||
|
||||
**POST** `/api/review/{job_id}/complete`
|
||||
|
||||
Submit corrected lyrics and trigger video rendering.
|
||||
|
||||
### Instrumental Selection
|
||||
|
||||
**GET** `/api/jobs/{job_id}/instrumental-options`
|
||||
|
||||
Get available instrumental options.
|
||||
|
||||
**POST** `/api/jobs/{job_id}/select-instrumental`
|
||||
|
||||
Submit instrumental selection (clean or with_backing).
|
||||
|
||||
```bash
|
||||
curl -X POST "https://api.example.com/api/jobs/abc12345/select-instrumental" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"selection": "clean"}'
|
||||
```
|
||||
|
||||
### Download Files
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download-urls`
|
||||
|
||||
Get download URLs for all output files.
|
||||
|
||||
**GET** `/api/jobs/{job_id}/download/{category}/{file_key}`
|
||||
|
||||
Stream download a specific file.
|
||||
|
||||
### Health Check
|
||||
|
||||
**GET** `/api/health`
|
||||
|
||||
Check backend health status.
|
||||
|
||||
---
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### "No suitable files found for processing"
|
||||
|
||||
This error occurs during the finalisation step when the `(With Vocals).mkv` file is missing. This file is created during lyrics transcription.
|
||||
|
||||
**Most common cause:** No transcription provider configured.
|
||||
|
||||
**Quick fix:**
|
||||
1. Check if transcription providers are configured:
|
||||
```bash
|
||||
echo $AUDIOSHAKE_API_TOKEN
|
||||
echo $RUNPOD_API_KEY
|
||||
```
|
||||
|
||||
2. If both are empty, set up a provider (see [Transcription Provider Setup](#transcription-provider-setup))
|
||||
|
||||
3. Or use `--skip-lyrics` for instrumental-only karaoke:
|
||||
```bash
|
||||
karaoke-gen --skip-lyrics "Artist" "Title"
|
||||
```
|
||||
|
||||
**Other causes:**
|
||||
- Invalid API credentials - verify your tokens are correct and active
|
||||
- API service unavailable - check service status pages
|
||||
- Network connectivity issues - ensure you can reach the API endpoints
|
||||
- Transcription timeout - try again or use a different provider
|
||||
|
||||
### Transcription Fails Silently
|
||||
|
||||
If karaoke-gen runs without errors but produces no synchronized lyrics:
|
||||
|
||||
1. **Check logs** - Run with `--log_level debug` for detailed output:
|
||||
```bash
|
||||
karaoke-gen --log_level debug "Artist" "Title"
|
||||
```
|
||||
|
||||
2. **Verify environment variables** - Ensure API tokens are exported in your shell:
|
||||
```bash
|
||||
# Check if set
|
||||
printenv | grep -E "(AUDIOSHAKE|RUNPOD|WHISPER)"
|
||||
|
||||
# Set in current session
|
||||
export AUDIOSHAKE_API_TOKEN="your_token"
|
||||
```
|
||||
|
||||
3. **Test API connectivity** - Verify you can reach the transcription service
|
||||
|
||||
### "No lyrics found from any source"
|
||||
|
||||
This warning means no reference lyrics were fetched from online sources (Genius, Spotify, Musixmatch). The transcription will still work, but auto-correction may be less accurate.
|
||||
|
||||
**To fix:**
|
||||
- Set `GENIUS_API_TOKEN` for Genius lyrics
|
||||
- Set `SPOTIFY_COOKIE_SP_DC` for Spotify lyrics
|
||||
- Set `RAPIDAPI_KEY` for Musixmatch lyrics
|
||||
- Or provide lyrics manually with `--lyrics_file /path/to/lyrics.txt`
|
||||
|
||||
### Video Quality Issues
|
||||
|
||||
If the output video has quality problems:
|
||||
- Ensure FFmpeg is properly installed: `ffmpeg -version`
|
||||
- Check available codecs: `ffmpeg -codecs`
|
||||
- For 4K output, ensure sufficient disk space (10GB+ per track)
|
||||
|
||||
### Local Whisper Issues
|
||||
|
||||
#### GPU Out of Memory
|
||||
If you get CUDA out of memory errors:
|
||||
```bash
|
||||
# Use a smaller model
|
||||
export WHISPER_MODEL_SIZE="small" # or "tiny"
|
||||
|
||||
# Or force CPU mode
|
||||
export WHISPER_DEVICE="cpu"
|
||||
```
|
||||
|
||||
#### Slow Transcription on CPU
|
||||
CPU transcription is significantly slower than GPU. For faster processing:
|
||||
- Use a smaller model (`tiny` or `base`)
|
||||
- Consider using cloud transcription (AudioShake or RunPod)
|
||||
- On Apple Silicon, the `small` model offers good speed/quality balance
|
||||
|
||||
#### Model Download Issues
|
||||
Whisper models are downloaded on first use (~1-3GB depending on size). If downloads fail:
|
||||
- Check your internet connection
|
||||
- Set a custom cache directory: `export WHISPER_CACHE_DIR="/path/with/space"`
|
||||
- Models are cached in `~/.cache/whisper/` by default
|
||||
|
||||
#### whisper-timestamped Not Found
|
||||
If you get "whisper-timestamped is not installed":
|
||||
```bash
|
||||
pip install "karaoke-gen[local-whisper]"
|
||||
# Or install directly:
|
||||
pip install whisper-timestamped
|
||||
```
|
||||
|
||||
#### Disabling Local Whisper
|
||||
If you want to disable local Whisper (e.g., to force cloud transcription):
|
||||
```bash
|
||||
export ENABLE_LOCAL_WHISPER="false"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Development
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest tests/ backend/tests/ -v
|
||||
|
||||
# Run only unit tests
|
||||
pytest tests/unit/ -v
|
||||
|
||||
# Run with coverage
|
||||
pytest tests/unit/ -v --cov=karaoke_gen --cov-report=term-missing
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
karaoke-gen/
|
||||
├── karaoke_gen/ # Core CLI package
|
||||
│ ├── utils/
|
||||
│ │ ├── gen_cli.py # Local CLI (karaoke-gen)
|
||||
│ │ └── remote_cli.py # Remote CLI (karaoke-gen-remote)
|
||||
│ ├── karaoke_finalise/ # Video encoding, packaging, distribution
|
||||
│ └── style_loader.py # Unified style configuration
|
||||
├── backend/ # Cloud backend (FastAPI)
|
||||
│ ├── api/routes/ # API endpoints
|
||||
│ ├── workers/ # Background processing workers
|
||||
│ └── services/ # Business logic services
|
||||
├── infrastructure/ # Pulumi IaC for GCP
|
||||
├── docs/ # Documentation
|
||||
└── tests/ # Test suite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see our contributing guidelines.
|
||||
@@ -1,75 +0,0 @@
|
||||
[
|
||||
{
|
||||
"test_name": "Sound-alike error (wurld -> world)",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "ReplaceWord",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w1"
|
||||
],
|
||||
"replacement_text": "world",
|
||||
"confidence": 0.75,
|
||||
"reason": "Sound-alike error. Reference suggests: 'world'. The transcription 'wurld' is a phonetic misspelling or sound-alike of the word 'world' which appears in both reference sources (Genius and LRCLIB).",
|
||||
"gap_category": "GapCategory.SOUND_ALIKE",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Punctuation difference",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w2"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The gap text 'Hello' followed by the context 'is it me you're looking for' matches the Genius reference lyrics exactly, excluding punctuation. According to the guidelines, if any reference source matches the transcription text (ignoring punctuation), it should be classified as NO_ERROR.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Extra filler word",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "DeleteWord",
|
||||
"word_id": "w3",
|
||||
"word_ids": null,
|
||||
"replacement_text": null,
|
||||
"confidence": 0.8,
|
||||
"reason": "Extra filler word 'And' at sentence start not in reference. The transcription includes the filler word 'And' at the beginning of the phrase, which is not present in the reference lyrics. This matches the definition of EXTRA_WORDS.",
|
||||
"gap_category": "GapCategory.EXTRA_WORDS",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"test_name": "Complex multi-word error",
|
||||
"success": true,
|
||||
"proposals": [
|
||||
{
|
||||
"action": "NoAction",
|
||||
"word_id": null,
|
||||
"word_ids": [
|
||||
"w7",
|
||||
"w8",
|
||||
"w9",
|
||||
"w10"
|
||||
],
|
||||
"replacement_text": null,
|
||||
"confidence": 0.99,
|
||||
"reason": "Transcription matches at least one reference source. The transcription text 'gonna give you up' matches the Genius reference lyrics exactly when ignoring punctuation and capitalization. In context, the full phrase 'never gonna give you up' is correctly transcribed and aligns perfectly with the known lyrics and title of the song.",
|
||||
"gap_category": "GapCategory.NO_ERROR",
|
||||
"requires_human_review": false
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -1,20 +0,0 @@
|
||||
[run]
|
||||
source = backend
|
||||
omit =
|
||||
backend/tests/*
|
||||
backend/validate.py
|
||||
backend/tests/emulator/*
|
||||
# Workers that require external APIs are tested via integration tests
|
||||
backend/workers/screens_worker.py
|
||||
backend/workers/video_worker.py
|
||||
|
||||
[report]
|
||||
exclude_lines =
|
||||
pragma: no cover
|
||||
def __repr__
|
||||
raise NotImplementedError
|
||||
if __name__ == .__main__.:
|
||||
pass
|
||||
|
||||
show_missing = True
|
||||
|
||||
@@ -1,37 +0,0 @@
|
||||
# Ignore Python cache and virtual environments
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Environment files
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# IDE
|
||||
.vscode/
|
||||
.idea/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# Temporary files
|
||||
*.log
|
||||
temp/
|
||||
tmp/
|
||||
|
||||
# Google Cloud
|
||||
.gcloud/
|
||||
*.json
|
||||
|
||||
# Testing
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
# Fast application image - uses pre-built base with all dependencies
|
||||
#
|
||||
# The base image (karaoke-backend-base) contains:
|
||||
# - Python 3.11, ffmpeg, sox, system deps
|
||||
# - All pip dependencies from requirements-frozen.txt (NO code)
|
||||
#
|
||||
# This Dockerfile adds the actual application code on top.
|
||||
# Build time: ~1-2 minutes (deps already cached in base)
|
||||
#
|
||||
# If the base image doesn't exist yet, build it first:
|
||||
# gcloud builds submit --config=cloudbuild-base.yaml --project=nomadkaraoke
|
||||
|
||||
# Use pre-built base image with all dependencies
|
||||
# Override with --build-arg BASE_IMAGE=karaoke-backend-base:local for local testing
|
||||
ARG BASE_IMAGE=us-central1-docker.pkg.dev/nomadkaraoke/karaoke-repo/karaoke-backend-base:latest
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
# Set working directory (should already exist from base)
|
||||
WORKDIR /app
|
||||
|
||||
# Copy backend code (this is the only thing that changes frequently)
|
||||
COPY backend /app/backend
|
||||
|
||||
# Copy and reinstall karaoke_gen from local source
|
||||
# This ensures new modules are included without rebuilding base image
|
||||
# Note: lyrics_transcriber is now a subpackage of karaoke_gen
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Set environment variables
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV PORT=8080
|
||||
|
||||
# Expose port
|
||||
EXPOSE 8080
|
||||
|
||||
# Run uvicorn directly
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
# Using --loop asyncio (instead of uvloop) to support nest_asyncio for remote flacfetch calls
|
||||
CMD ["uvicorn", "backend.main:app", "--host", "0.0.0.0", "--port", "8080", "--loop", "asyncio"]
|
||||
@@ -1,75 +0,0 @@
|
||||
# Base image with all Python dependencies pre-installed
|
||||
# This image is rebuilt only when poetry.lock changes
|
||||
#
|
||||
# Build with: gcloud builds submit --config=cloudbuild-base.yaml
|
||||
# Or manually: docker build -f backend/Dockerfile.base -t karaoke-backend-base .
|
||||
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install system dependencies required by karaoke-gen
|
||||
# Note: Transmission daemon removed - torrent downloads now handled by
|
||||
# dedicated flacfetch VM service (see docs/00-current-plan/TORRENT-SERVICE-PLAN.md)
|
||||
RUN apt-get update && apt-get install -y \
|
||||
libsndfile1 \
|
||||
libsox-dev \
|
||||
sox \
|
||||
build-essential \
|
||||
curl \
|
||||
xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install comprehensive font support for international lyrics and symbols
|
||||
# Noto fonts provide full Unicode coverage including:
|
||||
# - Latin, Greek, Cyrillic (core)
|
||||
# - CJK: Chinese, Japanese, Korean (~150MB)
|
||||
# - Arabic, Hebrew, Thai, Hindi, and other scripts (extra)
|
||||
# - Color emoji (limited support in libass, but available)
|
||||
# - Musical symbols (♪ ♫ etc.)
|
||||
# Note: This adds ~200-300MB to the image but ensures all languages render correctly
|
||||
RUN apt-get update && apt-get install -y \
|
||||
fonts-noto-core \
|
||||
fonts-noto-cjk \
|
||||
fonts-noto-extra \
|
||||
fonts-noto-color-emoji \
|
||||
fontconfig \
|
||||
&& fc-cache -f \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install optimized static FFmpeg build (7.0.2) instead of Debian package (5.1.8)
|
||||
# John Van Sickle's builds are compiled with better optimizations for x86_64
|
||||
# This provides ~20-40% performance improvement for encoding tasks
|
||||
# Source: https://johnvansickle.com/ffmpeg/
|
||||
RUN curl -L https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz -o /tmp/ffmpeg.tar.xz && \
|
||||
tar -xf /tmp/ffmpeg.tar.xz -C /tmp && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ && \
|
||||
cp /tmp/ffmpeg-*-amd64-static/ffprobe /usr/local/bin/ && \
|
||||
chmod +x /usr/local/bin/ffmpeg /usr/local/bin/ffprobe && \
|
||||
rm -rf /tmp/ffmpeg* && \
|
||||
ffmpeg -version
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy dependency files and source code needed for pip install
|
||||
COPY pyproject.toml README.md LICENSE /app/
|
||||
COPY karaoke_gen /app/karaoke_gen
|
||||
COPY lyrics_transcriber_temp /app/lyrics_transcriber_temp
|
||||
COPY backend /app/backend
|
||||
|
||||
# Install all Python dependencies (the slow part - cached in base image)
|
||||
RUN pip install --no-cache-dir --upgrade pip && \
|
||||
pip install --no-cache-dir -e .
|
||||
|
||||
# Download spaCy language model for lyrics transcription
|
||||
# This must be done at build time, not runtime (Cloud Run network restrictions)
|
||||
RUN python -m spacy download en_core_web_sm
|
||||
|
||||
# Label with build info
|
||||
ARG BUILD_DATE
|
||||
ARG BUILD_ID
|
||||
ARG BASE_CONTENT_HASH
|
||||
LABEL org.opencontainers.image.created="${BUILD_DATE}"
|
||||
LABEL org.opencontainers.image.revision="${BUILD_ID}"
|
||||
LABEL org.opencontainers.image.title="karaoke-backend-base"
|
||||
LABEL org.opencontainers.image.description="Base image with Python dependencies and system tools for karaoke-backend"
|
||||
LABEL base.content.hash="${BASE_CONTENT_HASH}"
|
||||
@@ -1,242 +0,0 @@
|
||||
# Backend Development Setup
|
||||
|
||||
## One-Time Setup
|
||||
|
||||
### 1. Create Virtual Environment
|
||||
|
||||
```bash
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
python3 -m venv backend/venv
|
||||
```
|
||||
|
||||
### 2. Activate Virtual Environment
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
```
|
||||
|
||||
### 3. Install Dependencies
|
||||
|
||||
**IMPORTANT:** Use Python 3.12 (not 3.13/3.14 - pydantic has compatibility issues)
|
||||
|
||||
```bash
|
||||
# Backend dependencies
|
||||
pip install -r backend/requirements.txt
|
||||
|
||||
# Install karaoke_gen in development mode
|
||||
pip install -e .
|
||||
|
||||
# Install lyrics_transcriber submodule
|
||||
pip install -e lyrics_transcriber_local/
|
||||
|
||||
# Install Google API client (for YouTube upload)
|
||||
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
|
||||
```
|
||||
|
||||
### 4. Add Shell Alias (Optional)
|
||||
|
||||
Add to `~/.zshrc`:
|
||||
|
||||
```bash
|
||||
alias backend='cd /Users/andrew/Projects/karaoke-gen && source backend/venv/bin/activate'
|
||||
```
|
||||
|
||||
Then you can just run:
|
||||
```bash
|
||||
backend
|
||||
# Now you're in the project directory with venv activated!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Daily Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Or if you set up the alias:
|
||||
backend
|
||||
```
|
||||
|
||||
### After Making Changes
|
||||
|
||||
```bash
|
||||
# 1. Run full validation (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
|
||||
# 2. If validation passes, deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Options
|
||||
|
||||
### Full Validation (Recommended - Requires venv)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
python3 backend/validate.py
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ **Import errors** (like the processing_service issue!)
|
||||
- ✅ Configuration issues
|
||||
- ✅ FastAPI app creation problems
|
||||
- ✅ Missing dependencies
|
||||
|
||||
**This would have caught the ModuleNotFoundError!**
|
||||
|
||||
### Quick Check (No venv needed)
|
||||
|
||||
```bash
|
||||
./backend/quick-check.sh
|
||||
```
|
||||
|
||||
**Catches:**
|
||||
- ✅ Syntax errors
|
||||
- ✅ Missing files
|
||||
- ✅ Some obvious import issues
|
||||
|
||||
**Good for quick checks, but won't catch all import errors**
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Development Session
|
||||
|
||||
```bash
|
||||
# 1. Activate environment
|
||||
backend # or: source backend/venv/bin/activate
|
||||
|
||||
# 2. Make your changes
|
||||
vim backend/api/routes/jobs.py
|
||||
|
||||
# 3. Validate (catches import errors!)
|
||||
python3 backend/validate.py
|
||||
# ✅ All validations passed!
|
||||
|
||||
# 4. Deploy
|
||||
./scripts/deploy.sh
|
||||
|
||||
# 5. Test
|
||||
export BACKEND_URL="https://karaoke-backend-ipzqd2k4yq-uc.a.run.app"
|
||||
export AUTH_TOKEN=$(gcloud auth print-identity-token)
|
||||
curl -H "Authorization: Bearer $AUTH_TOKEN" $BACKEND_URL/api/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What About karaoke_gen Imports?
|
||||
|
||||
The backend imports from `karaoke_gen` (the CLI package):
|
||||
|
||||
```python
|
||||
from karaoke_gen.karaoke_gen import KaraokePrep
|
||||
from karaoke_gen.audio_processor import AudioProcessor
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
### Option 1: Install in development mode (Recommended)
|
||||
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
This makes `karaoke_gen` available in the venv without copying files.
|
||||
|
||||
### Option 2: Use system Python
|
||||
|
||||
Your system Python already has `karaoke_gen` installed, so validation will work even if the venv doesn't have it (Python will fall back to system packages).
|
||||
|
||||
---
|
||||
|
||||
## Running the Backend Locally
|
||||
|
||||
```bash
|
||||
# Activate venv
|
||||
source backend/venv/bin/activate
|
||||
|
||||
# Set required environment variables
|
||||
export GOOGLE_CLOUD_PROJECT="nomadkaraoke"
|
||||
export GCS_BUCKET_NAME="karaoke-gen-storage-nomadkaraoke"
|
||||
|
||||
# Optional: Point to local credentials
|
||||
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
|
||||
|
||||
# Run the server
|
||||
cd backend
|
||||
uvicorn main:app --reload --port 8080
|
||||
```
|
||||
|
||||
Visit: http://localhost:8080/api/health
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No module named 'fastapi'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
pip install -r backend/requirements.txt
|
||||
```
|
||||
|
||||
### "No module named 'karaoke_gen'"
|
||||
|
||||
**Fix:**
|
||||
```bash
|
||||
source backend/venv/bin/activate
|
||||
cd /Users/andrew/Projects/karaoke-gen
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### Validation passes locally but fails in Cloud Run
|
||||
|
||||
**Check:**
|
||||
1. Is `requirements.txt` up to date?
|
||||
2. Did you add a new dependency without adding it to `requirements.txt`?
|
||||
3. Run: `pip freeze | grep <package-name>`
|
||||
|
||||
---
|
||||
|
||||
## Updating Dependencies
|
||||
|
||||
When you add new packages:
|
||||
|
||||
```bash
|
||||
# 1. Install in venv
|
||||
source backend/venv/bin/activate
|
||||
pip install new-package
|
||||
|
||||
# 2. Update requirements.txt
|
||||
pip freeze > backend/requirements.txt
|
||||
|
||||
# 3. Or manually add just the new package:
|
||||
echo "new-package==1.2.3" >> backend/requirements.txt
|
||||
|
||||
# 4. Validate
|
||||
python3 backend/validate.py
|
||||
|
||||
# 5. Deploy
|
||||
./scripts/deploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Best practice workflow:**
|
||||
|
||||
1. **One-time setup:** Create venv and install dependencies
|
||||
2. **Daily:** Activate venv when working on backend
|
||||
3. **Before deploy:** Run `python3 backend/validate.py` (catches import errors!)
|
||||
4. **Deploy:** Use `./scripts/deploy.sh`
|
||||
|
||||
**The import error would have been caught if you ran validation in a venv!** ✅
|
||||
Whitespace-only changes.
Whitespace-only changes.
@@ -1,343 +0,0 @@
|
||||
"""
|
||||
FastAPI dependencies for authentication and authorization.
|
||||
"""
|
||||
import logging
|
||||
import secrets
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from typing import Optional, Tuple, Callable, Union
|
||||
from fastapi import Depends, HTTPException, Header, Query, Request, Path
|
||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
||||
|
||||
from backend.services.auth_service import get_auth_service, UserType, AuthService, AuthResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def generate_review_token() -> str:
|
||||
"""Generate a cryptographically secure review token."""
|
||||
return secrets.token_urlsafe(32)
|
||||
|
||||
|
||||
def get_review_token_expiry(hours: int = 24) -> datetime:
|
||||
"""Get expiry time for a review token."""
|
||||
return datetime.now(UTC) + timedelta(hours=hours)
|
||||
|
||||
|
||||
# HTTP Bearer security scheme (Authorization: Bearer <token>)
|
||||
security = HTTPBearer(auto_error=False)
|
||||
|
||||
|
||||
async def get_token_from_request(
|
||||
request: Request,
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, description="Access token (alternative to Bearer header)")
|
||||
) -> Optional[str]:
|
||||
"""
|
||||
Extract access token from request.
|
||||
|
||||
Supports three methods (in priority order):
|
||||
1. X-Admin-Token header: Used by Cloud Tasks (since OIDC overwrites Authorization)
|
||||
2. Authorization header: Authorization: Bearer <token>
|
||||
3. Query parameter: ?token=<token> (for download links)
|
||||
|
||||
Returns:
|
||||
Token string or None
|
||||
"""
|
||||
# Check X-Admin-Token header first (used by Cloud Tasks)
|
||||
# Cloud Tasks OIDC token overwrites Authorization header, so we use a custom header
|
||||
admin_token = request.headers.get("X-Admin-Token")
|
||||
if admin_token:
|
||||
return admin_token
|
||||
|
||||
# Try Authorization header
|
||||
if credentials:
|
||||
return credentials.credentials
|
||||
|
||||
# Try query parameter
|
||||
if token:
|
||||
return token
|
||||
|
||||
return None
|
||||
|
||||
|
||||
async def require_auth(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require authentication for an endpoint.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication context including:
|
||||
- is_valid, user_type, remaining_uses, message (backward compatible via tuple unpacking)
|
||||
- user_email: Email of authenticated user (if session/API key auth)
|
||||
- is_admin: Whether user has admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
"""
|
||||
# Get token from request
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
# Validate token using the full method
|
||||
auth_result = auth_service.validate_token_full(token_str)
|
||||
|
||||
# Get request_id from middleware for correlation
|
||||
request_id = getattr(request.state, "request_id", None)
|
||||
|
||||
if not auth_result.is_valid:
|
||||
# Log auth failure with request_id for correlation
|
||||
auth_header = request.headers.get("Authorization", "")
|
||||
logger.warning(
|
||||
"auth_failed",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"audit_type": "auth_event",
|
||||
"auth_message": auth_result.message,
|
||||
"token_provided": bool(token_str),
|
||||
"token_length": len(token_str) if token_str else 0,
|
||||
"auth_header_present": bool(auth_header),
|
||||
}
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {auth_result.message}"
|
||||
)
|
||||
|
||||
# Log successful auth with request_id for correlation with request audit
|
||||
logger.info(
|
||||
"auth_success",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"user_email": auth_result.user_email,
|
||||
"user_type": auth_result.user_type.value if auth_result.user_type else None,
|
||||
"is_admin": auth_result.is_admin,
|
||||
"remaining_uses": auth_result.remaining_uses,
|
||||
"audit_type": "auth_event",
|
||||
}
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
async def require_auth_legacy(
|
||||
request: Request,
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Tuple[str, UserType, int]:
|
||||
"""
|
||||
Legacy authentication dependency that returns tuple.
|
||||
|
||||
DEPRECATED: Use require_auth which returns AuthResult instead.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses)
|
||||
"""
|
||||
token_str = await get_token_from_request(request, credentials, token)
|
||||
|
||||
if not token_str:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide token via Authorization header or ?token= parameter"
|
||||
)
|
||||
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail=f"Authentication failed: {message}"
|
||||
)
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
async def require_admin(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
) -> AuthResult:
|
||||
"""
|
||||
Require admin access for an endpoint.
|
||||
|
||||
Admin access is granted if:
|
||||
- Using an admin token from ADMIN_TOKENS env var
|
||||
- User email is from admin domain (e.g., @nomadkaraoke.com)
|
||||
- User role is set to ADMIN in database
|
||||
|
||||
Returns:
|
||||
AuthResult with admin privileges
|
||||
|
||||
Raises:
|
||||
HTTPException: 403 if user is not admin
|
||||
"""
|
||||
if not auth_result.is_admin:
|
||||
logger.warning(
|
||||
f"Admin access denied for {auth_result.user_type} user "
|
||||
f"(email: {auth_result.user_email or 'unknown'})"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Admin access required"
|
||||
)
|
||||
|
||||
return auth_result
|
||||
|
||||
|
||||
def optional_auth(
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None)
|
||||
) -> Optional[Tuple[str, UserType, int]]:
|
||||
"""
|
||||
Optional authentication - doesn't fail if no token provided.
|
||||
|
||||
Useful for endpoints that have different behavior for authenticated users.
|
||||
|
||||
Returns:
|
||||
(token, user_type, remaining_uses) if authenticated, None otherwise
|
||||
"""
|
||||
# Get token
|
||||
token_str = None
|
||||
if credentials:
|
||||
token_str = credentials.credentials
|
||||
elif token:
|
||||
token_str = token
|
||||
|
||||
if not token_str:
|
||||
return None
|
||||
|
||||
# Validate token
|
||||
is_valid, user_type, remaining_uses, message = auth_service.validate_token(token_str)
|
||||
|
||||
if not is_valid:
|
||||
logger.debug(f"Optional auth failed: {message}")
|
||||
return None
|
||||
|
||||
return token_str, user_type, remaining_uses
|
||||
|
||||
|
||||
def require_review_auth_factory(job_id_param: str = "job_id"):
|
||||
"""
|
||||
Factory to create a review authentication dependency.
|
||||
|
||||
Accepts either:
|
||||
1. Full user authentication (admin/user token) - also validates job ownership
|
||||
2. Job-specific review token (only valid for the specific job)
|
||||
|
||||
Args:
|
||||
job_id_param: Name of the path parameter containing the job ID
|
||||
|
||||
Returns:
|
||||
Dependency function that validates review access
|
||||
"""
|
||||
async def require_review_auth(
|
||||
request: Request,
|
||||
job_id: str = Path(...),
|
||||
review_token: Optional[str] = Query(None, description="Job-specific review token"),
|
||||
auth_service: AuthService = Depends(get_auth_service),
|
||||
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security),
|
||||
token: Optional[str] = Query(None, alias="token", description="Full access token")
|
||||
) -> Tuple[str, str]:
|
||||
"""
|
||||
Validate review access for a job.
|
||||
|
||||
Returns:
|
||||
(job_id, auth_type) where auth_type is "full" or "review_token"
|
||||
|
||||
Raises:
|
||||
HTTPException: 401 if authentication fails
|
||||
HTTPException: 403 if user doesn't own the job (for full auth)
|
||||
"""
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.job_manager import JobManager
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
# Try full authentication first
|
||||
full_token = None
|
||||
if credentials:
|
||||
full_token = credentials.credentials
|
||||
elif token:
|
||||
full_token = token
|
||||
|
||||
if full_token:
|
||||
auth_result = auth_service.validate_token_full(full_token)
|
||||
if auth_result.is_valid:
|
||||
# For full auth, also verify job ownership
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Check ownership: admin can access all, users only their own jobs
|
||||
if not auth_result.is_admin:
|
||||
if auth_result.user_email and job.user_email:
|
||||
if auth_result.user_email.lower() != job.user_email.lower():
|
||||
logger.warning(
|
||||
f"Review access denied: user {auth_result.user_email} "
|
||||
f"tried to access job {job_id} owned by {job.user_email}"
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
elif job.user_email:
|
||||
# Token auth without email trying to access a job with owner
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="You don't have permission to access this job's review"
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via full auth ({auth_result.user_type}) for job {job_id}")
|
||||
return job_id, "full"
|
||||
|
||||
# Try review token
|
||||
if review_token:
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Validate review token matches
|
||||
if job.review_token and secrets.compare_digest(job.review_token, review_token):
|
||||
# Check expiry if set
|
||||
if job.review_token_expires_at:
|
||||
now = datetime.now(UTC)
|
||||
# Handle timezone-naive datetimes from Firestore
|
||||
expiry = job.review_token_expires_at
|
||||
if expiry.tzinfo is None:
|
||||
expiry = expiry.replace(tzinfo=UTC)
|
||||
|
||||
if now > expiry:
|
||||
logger.warning(f"Review token expired for job {job_id}")
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Review token has expired. Please request a new review link."
|
||||
)
|
||||
|
||||
logger.info(f"Review access granted via review_token for job {job_id}")
|
||||
return job_id, "review_token"
|
||||
else:
|
||||
logger.warning(f"Invalid review token for job {job_id}")
|
||||
|
||||
# No valid authentication
|
||||
raise HTTPException(
|
||||
status_code=401,
|
||||
detail="Authentication required. Provide either a full access token or a valid review_token for this job."
|
||||
)
|
||||
|
||||
return require_review_auth
|
||||
|
||||
|
||||
# Default instance for most review endpoints
|
||||
require_review_auth = require_review_auth_factory()
|
||||
|
||||
|
||||
Whitespace-only changes.
File diff suppressed because it is too large.
Load diff
@@ -1,918 +0,0 @@
|
||||
"""
|
||||
Audio search API routes for artist+title search mode.
|
||||
|
||||
This module provides endpoints for:
|
||||
1. Creating a job with audio search (artist+title without file)
|
||||
2. Getting search results for a job
|
||||
3. Selecting an audio source to download
|
||||
|
||||
The flow is:
|
||||
1. POST /api/audio-search/search - Create job and search for audio
|
||||
2. GET /api/audio-search/{job_id}/results - Get search results
|
||||
3. POST /api/audio-search/{job_id}/select - Select audio source
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import mimetypes
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Request, Depends
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from backend.models.job import JobCreate, JobStatus
|
||||
from karaoke_gen.utils import normalize_text
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.worker_service import get_worker_service
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
from backend.services.audio_search_service import (
|
||||
get_audio_search_service,
|
||||
AudioSearchResult,
|
||||
AudioSearchError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
from backend.services.youtube_download_service import (
|
||||
get_youtube_download_service,
|
||||
YouTubeDownloadError,
|
||||
)
|
||||
from backend.services.theme_service import get_theme_service
|
||||
from backend.services.job_defaults_service import resolve_cdg_txt_defaults
|
||||
from backend.config import get_settings
|
||||
from backend.version import VERSION
|
||||
from backend.api.dependencies import require_auth
|
||||
from backend.services.auth_service import UserType, AuthResult
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
from pathlib import Path
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Valid style file types (from file_upload.py)
|
||||
STYLE_FILE_TYPES = {
|
||||
'style_params': {'.json'},
|
||||
'style_intro_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_karaoke_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_end_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_font': {'.ttf', '.otf'},
|
||||
'style_cdg_instrumental_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_title_background': {'.jpg', '.jpeg', '.png'},
|
||||
'style_cdg_outro_background': {'.jpg', '.jpeg', '.png'},
|
||||
}
|
||||
router = APIRouter(tags=["audio-search"])
|
||||
|
||||
# Initialize services
|
||||
job_manager = JobManager()
|
||||
storage_service = StorageService()
|
||||
worker_service = get_worker_service()
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Pydantic models
|
||||
# ============================================================================
|
||||
|
||||
class StyleFileRequest(BaseModel):
|
||||
"""Information about a style file to be uploaded."""
|
||||
filename: str = Field(..., description="Original filename with extension")
|
||||
content_type: str = Field(..., description="MIME type of the file")
|
||||
file_type: str = Field(..., description="Type of file: 'style_params', 'style_intro_background', 'style_karaoke_background', 'style_end_background', 'style_font', 'style_cdg_instrumental_background', 'style_cdg_title_background', 'style_cdg_outro_background'")
|
||||
|
||||
|
||||
class StyleUploadUrl(BaseModel):
|
||||
"""Signed URL for uploading a style file."""
|
||||
file_type: str = Field(..., description="Type of file being uploaded")
|
||||
gcs_path: str = Field(..., description="Destination path in GCS")
|
||||
upload_url: str = Field(..., description="Signed URL to PUT the file to")
|
||||
|
||||
|
||||
class AudioSearchRequest(BaseModel):
|
||||
"""Request to search for audio by artist and title."""
|
||||
artist: str = Field(..., description="Artist name to search for")
|
||||
title: str = Field(..., description="Song title to search for")
|
||||
|
||||
# Auto-download mode
|
||||
auto_download: bool = Field(False, description="Automatically select best result and download")
|
||||
|
||||
# Theme configuration
|
||||
theme_id: Optional[str] = Field(None, description="Theme ID to use (e.g., 'nomad', 'default'). If set, CDG/TXT are enabled by default.")
|
||||
color_overrides: Optional[Dict[str, str]] = Field(None, description="Color overrides: artist_color, title_color, sung_lyrics_color, unsung_lyrics_color (hex #RRGGBB)")
|
||||
|
||||
# Processing options (CDG/TXT require style config or theme, disabled by default unless theme is set)
|
||||
enable_cdg: Optional[bool] = Field(None, description="Generate CDG+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
enable_txt: Optional[bool] = Field(None, description="Generate TXT+MP3 package. Default: True if theme_id set, False otherwise")
|
||||
|
||||
# Finalisation options
|
||||
brand_prefix: Optional[str] = Field(None, description="Brand code prefix (e.g., NOMAD)")
|
||||
enable_youtube_upload: Optional[bool] = Field(None, description="Upload to YouTube. None = use server default")
|
||||
youtube_description: Optional[str] = Field(None, description="YouTube video description text")
|
||||
discord_webhook_url: Optional[str] = Field(None, description="Discord webhook URL for notifications")
|
||||
|
||||
# Distribution options
|
||||
dropbox_path: Optional[str] = Field(None, description="Dropbox folder path for organized output")
|
||||
gdrive_folder_id: Optional[str] = Field(None, description="Google Drive folder ID for public share uploads")
|
||||
|
||||
# Lyrics configuration
|
||||
lyrics_artist: Optional[str] = Field(None, description="Override artist name for lyrics search")
|
||||
lyrics_title: Optional[str] = Field(None, description="Override title for lyrics search")
|
||||
subtitle_offset_ms: int = Field(0, description="Subtitle timing offset in milliseconds")
|
||||
|
||||
# Display overrides (optional - if empty, search values are used for display)
|
||||
display_artist: Optional[str] = Field(None, description="Artist name for title screens/filenames. If empty, uses search artist.")
|
||||
display_title: Optional[str] = Field(None, description="Title for title screens/filenames. If empty, uses search title.")
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = Field(None, description="Model for clean instrumental separation")
|
||||
backing_vocals_models: Optional[List[str]] = Field(None, description="Models for backing vocals separation")
|
||||
other_stems_models: Optional[List[str]] = Field(None, description="Models for other stems")
|
||||
|
||||
# Style file uploads (optional)
|
||||
# Style params JSON and related assets (background images, fonts)
|
||||
style_files: Optional[List[StyleFileRequest]] = Field(None, description="Style files to upload (style_params JSON, backgrounds, fonts)")
|
||||
|
||||
# Non-interactive mode
|
||||
non_interactive: bool = Field(False, description="Skip interactive steps (lyrics review, instrumental selection)")
|
||||
|
||||
@validator('artist', 'title', 'lyrics_artist', 'lyrics_title', 'display_artist', 'display_title')
|
||||
def normalize_text_fields(cls, v):
|
||||
"""Normalize text fields to standardize Unicode characters."""
|
||||
if v is not None and isinstance(v, str):
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class AudioSearchResultResponse(BaseModel):
|
||||
"""A single audio search result.
|
||||
|
||||
Contains all fields needed for rich display using flacfetch's
|
||||
shared formatting functions.
|
||||
"""
|
||||
index: int
|
||||
title: str
|
||||
artist: str
|
||||
provider: str # Maps to source_name in Release
|
||||
url: Optional[str] = None # Maps to download_url in Release (may be None for remote)
|
||||
duration: Optional[int] = None # Maps to duration_seconds in Release
|
||||
quality: Optional[str] = None # Stringified quality
|
||||
source_id: Optional[str] = None # Maps to info_hash in Release
|
||||
seeders: Optional[int] = None
|
||||
target_file: Optional[str] = None
|
||||
# Additional fields for rich display (from Release.to_dict())
|
||||
year: Optional[int] = None
|
||||
label: Optional[str] = None
|
||||
edition_info: Optional[str] = None
|
||||
release_type: Optional[str] = None
|
||||
channel: Optional[str] = None # For YouTube
|
||||
view_count: Optional[int] = None # For YouTube
|
||||
size_bytes: Optional[int] = None
|
||||
target_file_size: Optional[int] = None
|
||||
track_pattern: Optional[str] = None
|
||||
match_score: Optional[float] = None
|
||||
# Pre-computed display fields
|
||||
formatted_size: Optional[str] = None
|
||||
formatted_duration: Optional[str] = None
|
||||
formatted_views: Optional[str] = None
|
||||
is_lossless: Optional[bool] = None
|
||||
quality_str: Optional[str] = None
|
||||
# Full quality object for Release.from_dict()
|
||||
quality_data: Optional[Dict[str, Any]] = None
|
||||
|
||||
|
||||
class AudioSearchResponse(BaseModel):
|
||||
"""Response from audio search."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
results: Optional[List[AudioSearchResultResponse]] = None
|
||||
results_count: int = 0
|
||||
auto_download: bool = False
|
||||
server_version: str
|
||||
# Style file upload URLs (when style_files are specified in request)
|
||||
style_upload_urls: Optional[List[StyleUploadUrl]] = None
|
||||
|
||||
|
||||
class AudioSelectRequest(BaseModel):
|
||||
"""Request to select an audio source."""
|
||||
selection_index: int = Field(..., description="Index of the selected audio source from search results")
|
||||
|
||||
|
||||
class AudioSelectResponse(BaseModel):
|
||||
"""Response from audio source selection."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
selected_index: int
|
||||
selected_title: str
|
||||
selected_artist: str
|
||||
selected_provider: str
|
||||
|
||||
|
||||
def extract_request_metadata(request: Request, created_from: str = "audio_search") -> Dict[str, Any]:
|
||||
"""Extract metadata from request for job tracking."""
|
||||
headers = dict(request.headers)
|
||||
|
||||
client_ip = headers.get('x-forwarded-for', '').split(',')[0].strip()
|
||||
if not client_ip and request.client:
|
||||
client_ip = request.client.host
|
||||
|
||||
user_agent = headers.get('user-agent', '')
|
||||
environment = headers.get('x-environment', '')
|
||||
client_id = headers.get('x-client-id', '')
|
||||
|
||||
custom_headers = {}
|
||||
for key, value in headers.items():
|
||||
if key.lower().startswith('x-') and key.lower() not in ('x-forwarded-for', 'x-forwarded-proto', 'x-forwarded-host'):
|
||||
custom_headers[key] = value
|
||||
|
||||
return {
|
||||
'client_ip': client_ip,
|
||||
'user_agent': user_agent,
|
||||
'environment': environment,
|
||||
'client_id': client_id,
|
||||
'server_version': VERSION,
|
||||
'created_from': created_from,
|
||||
'custom_headers': custom_headers,
|
||||
}
|
||||
|
||||
|
||||
async def _trigger_workers_parallel(job_id: str) -> None:
|
||||
"""Trigger both audio and lyrics workers in parallel."""
|
||||
await asyncio.gather(
|
||||
worker_service.trigger_audio_worker(job_id),
|
||||
worker_service.trigger_lyrics_worker(job_id)
|
||||
)
|
||||
|
||||
|
||||
def _extract_gcs_path(filepath: str) -> str:
|
||||
"""
|
||||
Extract the path portion from a GCS path or gs:// URL.
|
||||
|
||||
Args:
|
||||
filepath: Either a gs:// URL or a plain path
|
||||
|
||||
Returns:
|
||||
The path portion (e.g., "uploads/job123/audio/file.flac")
|
||||
"""
|
||||
if filepath.startswith("gs://"):
|
||||
# Format: gs://bucket/path/to/file
|
||||
parts = filepath.replace("gs://", "").split("/", 1)
|
||||
if len(parts) == 2:
|
||||
return parts[1]
|
||||
return filepath
|
||||
|
||||
|
||||
async def _download_and_start_processing(
|
||||
job_id: str,
|
||||
selection_index: int,
|
||||
audio_search_service,
|
||||
background_tasks: BackgroundTasks,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download selected audio and start job processing.
|
||||
|
||||
This is called either:
|
||||
- Immediately after search if auto_download=True
|
||||
- When user calls the select endpoint
|
||||
|
||||
For remote flacfetch downloads (torrent sources), the file is downloaded
|
||||
on the flacfetch VM and uploaded directly to GCS. For local downloads
|
||||
(YouTube), the file is downloaded locally and then uploaded to GCS.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Get search results from state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results available")
|
||||
|
||||
if selection_index < 0 or selection_index >= len(search_results):
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid selection index {selection_index}. Valid range: 0-{len(search_results)-1}"
|
||||
)
|
||||
|
||||
selected = search_results[selection_index]
|
||||
|
||||
# Get remote_search_id from state_data (stored during initial search)
|
||||
remote_search_id = job.state_data.get('remote_search_id')
|
||||
|
||||
# Transition to downloading state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING_AUDIO,
|
||||
progress=10,
|
||||
message=f"Downloading from {selected['provider']}: {selected['artist']} - {selected['title']}",
|
||||
state_data_updates={
|
||||
'selected_audio_index': selection_index,
|
||||
'selected_audio_provider': selected['provider'],
|
||||
}
|
||||
)
|
||||
|
||||
try:
|
||||
# Get source info from selected result
|
||||
source_name = selected.get('provider')
|
||||
source_id = selected.get('source_id')
|
||||
target_file = selected.get('target_file')
|
||||
download_url = selected.get('url')
|
||||
|
||||
# Route to appropriate download handler based on source type
|
||||
if source_name == 'YouTube':
|
||||
# Use YouTubeDownloadService for all YouTube downloads
|
||||
# This ensures consistent handling whether remote is configured or not
|
||||
youtube_service = get_youtube_download_service()
|
||||
|
||||
if not source_id and download_url:
|
||||
# Extract video ID from URL if not in source_id
|
||||
source_id = youtube_service._extract_video_id(download_url)
|
||||
|
||||
if not source_id:
|
||||
raise DownloadError(f"No video ID available for YouTube download")
|
||||
|
||||
logger.info(f"YouTube download via YouTubeDownloadService: video_id={source_id}")
|
||||
|
||||
try:
|
||||
audio_gcs_path = await youtube_service.download_by_id(
|
||||
video_id=source_id,
|
||||
job_id=job_id,
|
||||
artist=selected.get('artist'),
|
||||
title=selected.get('title'),
|
||||
)
|
||||
filename = os.path.basename(audio_gcs_path)
|
||||
except YouTubeDownloadError as e:
|
||||
raise DownloadError(f"YouTube download failed: {e}")
|
||||
|
||||
elif source_name in ['RED', 'OPS']:
|
||||
# Torrent sources - must use remote flacfetch
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
)
|
||||
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
|
||||
if source_id:
|
||||
logger.info(f"Torrent download via download_by_id: {source_name} ID={source_id}")
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="",
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
else:
|
||||
logger.info(f"Torrent download via search-based download")
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir="",
|
||||
gcs_path=gcs_destination,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
# Extract GCS path from result
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
logger.info(f"Torrent download complete: {audio_gcs_path}")
|
||||
|
||||
elif source_name == 'Spotify':
|
||||
# Spotify downloads - use audio_search_service
|
||||
if not audio_search_service.is_remote_enabled():
|
||||
raise DownloadError(
|
||||
f"Cannot download from Spotify without remote flacfetch service. "
|
||||
"Configure FLACFETCH_API_URL."
|
||||
)
|
||||
|
||||
gcs_destination = f"uploads/{job_id}/audio/"
|
||||
logger.info(f"Spotify download: source_id={source_id}")
|
||||
|
||||
result = audio_search_service.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir="",
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_destination,
|
||||
)
|
||||
|
||||
audio_gcs_path = _extract_gcs_path(result.filepath)
|
||||
filename = os.path.basename(result.filepath)
|
||||
logger.info(f"Spotify download complete: {audio_gcs_path}")
|
||||
|
||||
else:
|
||||
# Unknown source - try generic download
|
||||
logger.warning(f"Unknown source type: {source_name}, attempting generic download")
|
||||
temp_dir = tempfile.mkdtemp(prefix=f"audio_download_{job_id}_")
|
||||
|
||||
try:
|
||||
result = audio_search_service.download(
|
||||
result_index=selection_index,
|
||||
output_dir=temp_dir,
|
||||
remote_search_id=remote_search_id,
|
||||
)
|
||||
|
||||
filename = os.path.basename(result.filepath)
|
||||
audio_gcs_path = f"uploads/{job_id}/audio/{filename}"
|
||||
|
||||
content_type, _ = mimetypes.guess_type(result.filepath)
|
||||
with open(result.filepath, 'rb') as f:
|
||||
storage_service.upload_fileobj(
|
||||
f,
|
||||
audio_gcs_path,
|
||||
content_type=content_type or 'application/octet-stream'
|
||||
)
|
||||
|
||||
logger.info(f"Generic download uploaded to GCS: {audio_gcs_path}")
|
||||
finally:
|
||||
# Clean up temp files
|
||||
try:
|
||||
if os.path.exists(temp_dir):
|
||||
import shutil
|
||||
shutil.rmtree(temp_dir)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to clean up temp files: {e}")
|
||||
|
||||
# Update job with GCS path and transition to DOWNLOADING
|
||||
job_manager.update_job(job_id, {
|
||||
'input_media_gcs_path': audio_gcs_path,
|
||||
'filename': filename,
|
||||
})
|
||||
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.DOWNLOADING,
|
||||
progress=15,
|
||||
message="Audio downloaded, starting processing"
|
||||
)
|
||||
|
||||
# Trigger workers
|
||||
background_tasks.add_task(_trigger_workers_parallel, job_id)
|
||||
|
||||
return {
|
||||
'selected_index': selection_index,
|
||||
'selected_title': selected['title'],
|
||||
'selected_artist': selected['artist'],
|
||||
'selected_provider': selected['provider'],
|
||||
}
|
||||
|
||||
except DownloadError as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
except Exception as e:
|
||||
job_manager.fail_job(job_id, f"Audio download failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Download failed: {e}")
|
||||
|
||||
|
||||
@router.post("/audio-search/search", response_model=AudioSearchResponse)
|
||||
async def search_audio(
|
||||
request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSearchRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Search for audio by artist and title, creating a new job.
|
||||
|
||||
This endpoint:
|
||||
1. Creates a job in PENDING state
|
||||
2. Searches for audio using flacfetch
|
||||
3. Either returns search results for user selection, or
|
||||
4. If auto_download=True, automatically selects best and starts processing
|
||||
|
||||
Use cases:
|
||||
- Interactive mode (default): Returns results, user calls /select endpoint
|
||||
- Auto mode (auto_download=True): Automatically selects and downloads best
|
||||
"""
|
||||
# Check tenant feature flag
|
||||
tenant_config = get_tenant_config_from_request(request)
|
||||
if tenant_config and not tenant_config.features.audio_search:
|
||||
raise HTTPException(
|
||||
status_code=403,
|
||||
detail="Audio search is not available for this portal"
|
||||
)
|
||||
|
||||
try:
|
||||
# Apply default distribution settings
|
||||
settings = get_settings()
|
||||
effective_dropbox_path = body.dropbox_path or settings.default_dropbox_path
|
||||
effective_gdrive_folder_id = body.gdrive_folder_id or settings.default_gdrive_folder_id
|
||||
effective_discord_webhook_url = body.discord_webhook_url or settings.default_discord_webhook_url
|
||||
|
||||
# Apply defaults for YouTube/Dropbox distribution (for web service)
|
||||
# Use explicit value if provided, otherwise fall back to server default
|
||||
effective_enable_youtube_upload = body.enable_youtube_upload if body.enable_youtube_upload is not None else settings.default_enable_youtube_upload
|
||||
effective_brand_prefix = body.brand_prefix or settings.default_brand_prefix
|
||||
effective_youtube_description = body.youtube_description or settings.default_youtube_description
|
||||
|
||||
# Validate credentials if distribution services are requested
|
||||
invalid_services = []
|
||||
credential_manager = get_credential_manager()
|
||||
|
||||
if effective_enable_youtube_upload:
|
||||
result = credential_manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"youtube ({result.message})")
|
||||
|
||||
if effective_dropbox_path:
|
||||
result = credential_manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"dropbox ({result.message})")
|
||||
|
||||
if effective_gdrive_folder_id:
|
||||
result = credential_manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append(f"gdrive ({result.message})")
|
||||
|
||||
if invalid_services:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail={
|
||||
"error": "credentials_invalid",
|
||||
"message": f"Distribution services need re-authorization: {', '.join(invalid_services)}",
|
||||
"invalid_services": invalid_services,
|
||||
}
|
||||
)
|
||||
|
||||
# Extract request metadata
|
||||
request_metadata = extract_request_metadata(request, created_from="audio_search")
|
||||
|
||||
# Apply default theme if none specified
|
||||
# This ensures all karaoke videos use the Nomad theme by default
|
||||
effective_theme_id = body.theme_id
|
||||
if effective_theme_id is None:
|
||||
theme_service = get_theme_service()
|
||||
effective_theme_id = theme_service.get_default_theme_id()
|
||||
if effective_theme_id:
|
||||
logger.info(f"Applying default theme: {effective_theme_id}")
|
||||
|
||||
# Resolve CDG/TXT defaults based on theme
|
||||
resolved_cdg, resolved_txt = resolve_cdg_txt_defaults(
|
||||
effective_theme_id, body.enable_cdg, body.enable_txt
|
||||
)
|
||||
|
||||
# Use authenticated user's email
|
||||
effective_user_email = auth_result.user_email
|
||||
|
||||
# Determine display values - use display_* if provided, otherwise fall back to search values
|
||||
# Display values are used for title screens, filenames, YouTube, etc.
|
||||
# Search values (body.artist, body.title) are used for audio search
|
||||
effective_display_artist = body.display_artist.strip() if body.display_artist else body.artist
|
||||
effective_display_title = body.display_title.strip() if body.display_title else body.title
|
||||
|
||||
# Create job
|
||||
job_create = JobCreate(
|
||||
artist=effective_display_artist, # Display value for title screens, filenames
|
||||
title=effective_display_title, # Display value for title screens, filenames
|
||||
theme_id=effective_theme_id,
|
||||
color_overrides=body.color_overrides or {},
|
||||
enable_cdg=resolved_cdg,
|
||||
enable_txt=resolved_txt,
|
||||
brand_prefix=effective_brand_prefix,
|
||||
enable_youtube_upload=effective_enable_youtube_upload,
|
||||
youtube_description=effective_youtube_description,
|
||||
youtube_description_template=effective_youtube_description, # video_worker reads this field
|
||||
discord_webhook_url=effective_discord_webhook_url,
|
||||
dropbox_path=effective_dropbox_path,
|
||||
gdrive_folder_id=effective_gdrive_folder_id,
|
||||
user_email=effective_user_email,
|
||||
lyrics_artist=body.lyrics_artist,
|
||||
lyrics_title=body.lyrics_title,
|
||||
subtitle_offset_ms=body.subtitle_offset_ms,
|
||||
clean_instrumental_model=body.clean_instrumental_model,
|
||||
backing_vocals_models=body.backing_vocals_models,
|
||||
other_stems_models=body.other_stems_models,
|
||||
audio_search_artist=body.artist,
|
||||
audio_search_title=body.title,
|
||||
auto_download=body.auto_download,
|
||||
request_metadata=request_metadata,
|
||||
non_interactive=body.non_interactive,
|
||||
# Tenant scoping
|
||||
tenant_id=tenant_config.id if tenant_config else None,
|
||||
)
|
||||
job = job_manager.create_job(job_create, is_admin=auth_result.is_admin)
|
||||
job_id = job.job_id
|
||||
|
||||
logger.info(f"Created job {job_id} for audio search: {body.artist} - {body.title}")
|
||||
|
||||
# Update job with audio search fields
|
||||
job_manager.update_job(job_id, {
|
||||
'audio_search_artist': body.artist,
|
||||
'audio_search_title': body.title,
|
||||
'auto_download': body.auto_download,
|
||||
})
|
||||
|
||||
# If theme is set and no custom style files are being uploaded, prepare theme style now
|
||||
# This copies the theme's style_params.json to the job folder so LyricsTranscriber
|
||||
# can access the style configuration for preview videos
|
||||
if effective_theme_id and not body.style_files:
|
||||
from backend.api.routes.file_upload import _prepare_theme_for_job
|
||||
try:
|
||||
style_params_path, theme_style_assets, youtube_desc = _prepare_theme_for_job(
|
||||
job_id, effective_theme_id, body.color_overrides
|
||||
)
|
||||
theme_update = {
|
||||
'style_params_gcs_path': style_params_path,
|
||||
'style_assets': theme_style_assets,
|
||||
}
|
||||
if youtube_desc and not effective_youtube_description:
|
||||
theme_update['youtube_description_template'] = youtube_desc
|
||||
job_manager.update_job(job_id, theme_update)
|
||||
logger.info(f"Applied theme '{effective_theme_id}' to job {job_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to prepare theme '{effective_theme_id}' for job {job_id}: {e}")
|
||||
# Continue without theme - job can still be processed with defaults
|
||||
|
||||
# Handle style file uploads if provided
|
||||
style_upload_urls: List[StyleUploadUrl] = []
|
||||
style_assets = {}
|
||||
|
||||
if body.style_files:
|
||||
logger.info(f"Processing {len(body.style_files)} style file uploads for job {job_id}")
|
||||
|
||||
for file_info in body.style_files:
|
||||
# Validate file type
|
||||
if file_info.file_type not in STYLE_FILE_TYPES:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid file type '{file_info.file_type}'. Must be one of: {', '.join(STYLE_FILE_TYPES.keys())}"
|
||||
)
|
||||
|
||||
# Validate extension
|
||||
ext = Path(file_info.filename).suffix.lower()
|
||||
allowed_exts = STYLE_FILE_TYPES[file_info.file_type]
|
||||
if ext not in allowed_exts:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid extension '{ext}' for {file_info.file_type}. Allowed: {', '.join(allowed_exts)}"
|
||||
)
|
||||
|
||||
# Generate GCS path
|
||||
if file_info.file_type == 'style_params':
|
||||
gcs_path = f"uploads/{job_id}/style/style_params.json"
|
||||
else:
|
||||
# style_intro_background -> intro_background, etc.
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
gcs_path = f"uploads/{job_id}/style/{asset_key}{ext}"
|
||||
|
||||
# Generate signed upload URL
|
||||
signed_url = storage_service.generate_signed_upload_url(
|
||||
gcs_path,
|
||||
content_type=file_info.content_type,
|
||||
expiration_minutes=60
|
||||
)
|
||||
|
||||
style_upload_urls.append(StyleUploadUrl(
|
||||
file_type=file_info.file_type,
|
||||
gcs_path=gcs_path,
|
||||
upload_url=signed_url
|
||||
))
|
||||
|
||||
# Track the expected asset paths
|
||||
if file_info.file_type == 'style_params':
|
||||
style_assets['style_params'] = gcs_path
|
||||
else:
|
||||
asset_key = file_info.file_type.replace('style_', '')
|
||||
style_assets[asset_key] = gcs_path
|
||||
|
||||
# Update job with style asset expectations
|
||||
style_update = {'style_assets': style_assets}
|
||||
if 'style_params' in style_assets:
|
||||
style_update['style_params_gcs_path'] = style_assets['style_params']
|
||||
job_manager.update_job(job_id, style_update)
|
||||
|
||||
logger.info(f"Generated {len(style_upload_urls)} style upload URLs for job {job_id}")
|
||||
|
||||
# Transition to searching state
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.SEARCHING_AUDIO,
|
||||
progress=5,
|
||||
message=f"Searching for: {body.artist} - {body.title}"
|
||||
)
|
||||
|
||||
# Perform search
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
try:
|
||||
search_results = audio_search_service.search(body.artist, body.title)
|
||||
except NoResultsError as e:
|
||||
job_manager.fail_job(job_id, f"No audio sources found for: {body.artist} - {body.title}")
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail={
|
||||
"error": "no_results",
|
||||
"message": str(e),
|
||||
"job_id": job_id,
|
||||
}
|
||||
)
|
||||
except AudioSearchError as e:
|
||||
job_manager.fail_job(job_id, f"Audio search failed: {e}")
|
||||
raise HTTPException(status_code=500, detail=f"Search failed: {e}")
|
||||
|
||||
# Store results in job state_data, including remote_search_id if available
|
||||
results_dicts = [r.to_dict() for r in search_results]
|
||||
state_data_update = {
|
||||
'audio_search_results': results_dicts,
|
||||
'audio_search_count': len(results_dicts),
|
||||
}
|
||||
# Store remote_search_id for use during download (important for concurrent requests)
|
||||
if audio_search_service.last_remote_search_id:
|
||||
state_data_update['remote_search_id'] = audio_search_service.last_remote_search_id
|
||||
job_manager.update_job(job_id, {
|
||||
'state_data': state_data_update
|
||||
})
|
||||
|
||||
# If auto_download, select best and start processing
|
||||
if body.auto_download:
|
||||
best_index = audio_search_service.select_best(search_results)
|
||||
|
||||
logger.info(f"Auto-download enabled, selecting result {best_index}")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=best_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message=f"Audio found and download started: {selection_info['selected_artist']} - {selection_info['selected_title']} ({selection_info['selected_provider']})",
|
||||
results=None, # Don't return results in auto mode
|
||||
results_count=len(search_results),
|
||||
auto_download=True,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
# Interactive mode: return results for user selection
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.AWAITING_AUDIO_SELECTION,
|
||||
progress=10,
|
||||
message=f"Found {len(search_results)} audio sources. Waiting for selection."
|
||||
)
|
||||
|
||||
# Convert to response format with full Release data for rich display
|
||||
result_responses = []
|
||||
for r in search_results:
|
||||
# Get full serialized data from raw_result if available
|
||||
# raw_result can be either:
|
||||
# - A dict (from remote flacfetch API)
|
||||
# - A Release object (from local flacfetch)
|
||||
raw_dict = {}
|
||||
if r.raw_result:
|
||||
if isinstance(r.raw_result, dict):
|
||||
# Remote flacfetch API returns dicts directly
|
||||
raw_dict = r.raw_result
|
||||
else:
|
||||
# Local flacfetch returns Release objects
|
||||
try:
|
||||
raw_dict = r.raw_result.to_dict()
|
||||
except AttributeError:
|
||||
pass # Not a Release object
|
||||
|
||||
result_responses.append(
|
||||
AudioSearchResultResponse(
|
||||
index=r.index,
|
||||
title=r.title,
|
||||
artist=r.artist,
|
||||
provider=r.provider,
|
||||
url=r.url,
|
||||
duration=r.duration,
|
||||
quality=r.quality,
|
||||
source_id=r.source_id,
|
||||
seeders=raw_dict.get('seeders') or r.seeders,
|
||||
target_file=raw_dict.get('target_file') or r.target_file,
|
||||
# Additional fields for rich display
|
||||
year=raw_dict.get('year'),
|
||||
label=raw_dict.get('label'),
|
||||
edition_info=raw_dict.get('edition_info'),
|
||||
release_type=raw_dict.get('release_type'),
|
||||
channel=raw_dict.get('channel'),
|
||||
view_count=raw_dict.get('view_count'),
|
||||
size_bytes=raw_dict.get('size_bytes'),
|
||||
target_file_size=raw_dict.get('target_file_size'),
|
||||
track_pattern=raw_dict.get('track_pattern'),
|
||||
match_score=raw_dict.get('match_score'),
|
||||
# Pre-computed display fields
|
||||
formatted_size=raw_dict.get('formatted_size'),
|
||||
formatted_duration=raw_dict.get('formatted_duration'),
|
||||
formatted_views=raw_dict.get('formatted_views'),
|
||||
is_lossless=raw_dict.get('is_lossless'),
|
||||
quality_str=raw_dict.get('quality_str'),
|
||||
# Full quality object for Release.from_dict()
|
||||
# Remote API uses 'quality_data', local uses 'quality' for the dict
|
||||
quality_data=raw_dict.get('quality_data') or raw_dict.get('quality'),
|
||||
)
|
||||
)
|
||||
|
||||
return AudioSearchResponse(
|
||||
status="awaiting_selection",
|
||||
job_id=job_id,
|
||||
message=f"Found {len(search_results)} audio sources. Call /api/audio-search/{job_id}/select to choose one.",
|
||||
results=result_responses,
|
||||
results_count=len(search_results),
|
||||
auto_download=False,
|
||||
server_version=VERSION,
|
||||
style_upload_urls=style_upload_urls if style_upload_urls else None,
|
||||
)
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error in audio search: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/audio-search/{job_id}/results")
|
||||
async def get_audio_search_results(
|
||||
job_id: str,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Get audio search results for a job.
|
||||
|
||||
Returns the cached search results so user can select one.
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
|
||||
if not search_results:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="No search results available for this job"
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"job_id": job_id,
|
||||
"job_status": job.status,
|
||||
"artist": job.audio_search_artist or job.artist,
|
||||
"title": job.audio_search_title or job.title,
|
||||
"results": search_results,
|
||||
"results_count": len(search_results),
|
||||
}
|
||||
|
||||
|
||||
@router.post("/audio-search/{job_id}/select", response_model=AudioSelectResponse)
|
||||
async def select_audio_source(
|
||||
job_id: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
body: AudioSelectRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Select an audio source and start job processing.
|
||||
|
||||
This endpoint:
|
||||
1. Validates the job is awaiting selection
|
||||
2. Downloads the selected audio
|
||||
3. Uploads to GCS
|
||||
4. Triggers audio and lyrics workers
|
||||
"""
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
|
||||
|
||||
# Verify job is awaiting selection
|
||||
if job.status != JobStatus.AWAITING_AUDIO_SELECTION:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job is not awaiting audio selection (status: {job.status})"
|
||||
)
|
||||
|
||||
# Get search service instance
|
||||
# Note: With download_by_id, we no longer need to re-search to populate the cache.
|
||||
# The source_id stored in job.state_data['audio_search_results'] is sufficient.
|
||||
audio_search_service = get_audio_search_service()
|
||||
|
||||
# Validate search results exist in job state_data
|
||||
search_results = job.state_data.get('audio_search_results', [])
|
||||
if not search_results:
|
||||
raise HTTPException(status_code=400, detail="No search results cached for this job")
|
||||
|
||||
selection_info = await _download_and_start_processing(
|
||||
job_id=job_id,
|
||||
selection_index=body.selection_index,
|
||||
audio_search_service=audio_search_service,
|
||||
background_tasks=background_tasks,
|
||||
)
|
||||
|
||||
return AudioSelectResponse(
|
||||
status="success",
|
||||
job_id=job_id,
|
||||
message="Audio selected and download started",
|
||||
selected_index=selection_info['selected_index'],
|
||||
selected_title=selection_info['selected_title'],
|
||||
selected_artist=selection_info['selected_artist'],
|
||||
selected_provider=selection_info['selected_provider'],
|
||||
)
|
||||
|
||||
|
||||
@@ -1,348 +0,0 @@
|
||||
"""
|
||||
OAuth credential management API routes.
|
||||
|
||||
Provides endpoints for:
|
||||
1. Checking credential status
|
||||
2. Device authorization flow for re-authentication
|
||||
3. Credential validation before job submission
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.services.credential_manager import (
|
||||
get_credential_manager,
|
||||
CredentialStatus,
|
||||
CredentialCheckResult,
|
||||
)
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Request/Response Models
|
||||
# ===========================================================================
|
||||
|
||||
class CredentialStatusResponse(BaseModel):
|
||||
"""Response model for credential status."""
|
||||
service: str
|
||||
status: str
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
|
||||
|
||||
class AllCredentialsStatusResponse(BaseModel):
|
||||
"""Response model for all credentials status."""
|
||||
youtube: CredentialStatusResponse
|
||||
gdrive: CredentialStatusResponse
|
||||
dropbox: CredentialStatusResponse
|
||||
all_valid: bool
|
||||
services_needing_auth: list[str]
|
||||
|
||||
|
||||
class DeviceAuthStartRequest(BaseModel):
|
||||
"""Request to start device authorization flow."""
|
||||
client_id: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
client_secret: Optional[str] = None # Optional - reads from Secret Manager if not provided
|
||||
|
||||
|
||||
class DeviceAuthStartResponse(BaseModel):
|
||||
"""Response with device authorization info."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
instructions: str
|
||||
|
||||
|
||||
class DeviceAuthPollResponse(BaseModel):
|
||||
"""Response from polling device authorization."""
|
||||
status: str # "pending", "complete", "expired", "error"
|
||||
message: Optional[str] = None
|
||||
credentials_saved: bool = False
|
||||
|
||||
|
||||
class CredentialValidationRequest(BaseModel):
|
||||
"""Request to validate credentials for specific services."""
|
||||
youtube: bool = False
|
||||
gdrive: bool = False
|
||||
dropbox: bool = False
|
||||
|
||||
|
||||
class CredentialValidationResponse(BaseModel):
|
||||
"""Response from credential validation."""
|
||||
valid: bool
|
||||
invalid_services: list[str]
|
||||
message: str
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.get("/status", response_model=AllCredentialsStatusResponse)
|
||||
async def get_credentials_status():
|
||||
"""
|
||||
Get the status of all OAuth credentials.
|
||||
|
||||
Returns the validation status for YouTube, Google Drive, and Dropbox
|
||||
credentials, including whether they are valid, expired, or need
|
||||
re-authorization.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
def to_response(result: CredentialCheckResult) -> CredentialStatusResponse:
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
services_needing_auth = [
|
||||
name for name, result in results.items()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED, CredentialStatus.NOT_CONFIGURED)
|
||||
]
|
||||
|
||||
return AllCredentialsStatusResponse(
|
||||
youtube=to_response(results["youtube"]),
|
||||
gdrive=to_response(results["gdrive"]),
|
||||
dropbox=to_response(results["dropbox"]),
|
||||
all_valid=len(services_needing_auth) == 0,
|
||||
services_needing_auth=services_needing_auth
|
||||
)
|
||||
|
||||
|
||||
@router.get("/status/{service}", response_model=CredentialStatusResponse)
|
||||
async def get_service_credential_status(service: str):
|
||||
"""
|
||||
Get the status of a specific service's OAuth credentials.
|
||||
|
||||
Args:
|
||||
service: One of "youtube", "gdrive", "dropbox"
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
if service == "youtube":
|
||||
result = manager.check_youtube_credentials()
|
||||
elif service == "gdrive":
|
||||
result = manager.check_gdrive_credentials()
|
||||
elif service == "dropbox":
|
||||
result = manager.check_dropbox_credentials()
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail=f"Unknown service: {service}")
|
||||
|
||||
return CredentialStatusResponse(
|
||||
service=result.service,
|
||||
status=result.status.value,
|
||||
message=result.message,
|
||||
last_checked=result.last_checked,
|
||||
expires_at=result.expires_at
|
||||
)
|
||||
|
||||
|
||||
@router.post("/validate", response_model=CredentialValidationResponse)
|
||||
async def validate_credentials(request: CredentialValidationRequest):
|
||||
"""
|
||||
Validate that credentials are available for requested services.
|
||||
|
||||
Use this before submitting a job to ensure all required
|
||||
credentials are valid.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
invalid_services = []
|
||||
|
||||
if request.youtube:
|
||||
result = manager.check_youtube_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("youtube")
|
||||
|
||||
if request.gdrive:
|
||||
result = manager.check_gdrive_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("gdrive")
|
||||
|
||||
if request.dropbox:
|
||||
result = manager.check_dropbox_credentials()
|
||||
if result.status != CredentialStatus.VALID:
|
||||
invalid_services.append("dropbox")
|
||||
|
||||
if invalid_services:
|
||||
return CredentialValidationResponse(
|
||||
valid=False,
|
||||
invalid_services=invalid_services,
|
||||
message=f"The following services need re-authorization: {', '.join(invalid_services)}"
|
||||
)
|
||||
|
||||
return CredentialValidationResponse(
|
||||
valid=True,
|
||||
invalid_services=[],
|
||||
message="All requested credentials are valid"
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Device Authorization Flow Endpoints
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/youtube/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_youtube_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
This initiates a device auth flow that allows authorization
|
||||
from any device. The user must visit the verification URL
|
||||
and enter the user code.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('youtube-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/youtube/device/{device_code} to
|
||||
check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_youtube_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start YouTube device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/youtube/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_youtube_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for YouTube device authorization completion.
|
||||
|
||||
Call this endpoint at the interval specified in the device auth
|
||||
start response. Keep polling while status is "pending".
|
||||
|
||||
Status values:
|
||||
- pending: User has not yet authorized
|
||||
- complete: Authorization successful, credentials saved
|
||||
- expired: Device code expired, start new flow
|
||||
- error: An error occurred
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("youtube", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
@router.post("/gdrive/device", response_model=DeviceAuthStartResponse)
|
||||
async def start_gdrive_device_auth(request: Optional[DeviceAuthStartRequest] = None):
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Client credentials are loaded from Secret Manager ('gdrive-client-credentials')
|
||||
unless explicitly provided in the request body.
|
||||
|
||||
After starting, poll /auth/gdrive/device/{device_code} to check for completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
try:
|
||||
device_info = manager.start_gdrive_device_auth(
|
||||
client_id=request.client_id if request else None,
|
||||
client_secret=request.client_secret if request else None
|
||||
)
|
||||
|
||||
return DeviceAuthStartResponse(
|
||||
device_code=device_info.device_code,
|
||||
user_code=device_info.user_code,
|
||||
verification_url=device_info.verification_url,
|
||||
expires_in=device_info.expires_in,
|
||||
interval=device_info.interval,
|
||||
instructions=f"Visit {device_info.verification_url} and enter code: {device_info.user_code}"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to start Google Drive device auth: {e}")
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
||||
|
||||
@router.get("/gdrive/device/{device_code}", response_model=DeviceAuthPollResponse)
|
||||
async def poll_gdrive_device_auth(device_code: str):
|
||||
"""
|
||||
Poll for Google Drive device authorization completion.
|
||||
"""
|
||||
manager = get_credential_manager()
|
||||
|
||||
status, data = manager.poll_device_auth("gdrive", device_code)
|
||||
|
||||
return DeviceAuthPollResponse(
|
||||
status=status,
|
||||
message=data.get("message") if data else None,
|
||||
credentials_saved=(status == "complete")
|
||||
)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Alert Testing
|
||||
# ===========================================================================
|
||||
|
||||
@router.post("/test-alert")
|
||||
async def test_credential_alert(background_tasks: BackgroundTasks):
|
||||
"""
|
||||
Test the credential alert mechanism.
|
||||
|
||||
Sends a test alert to Discord if configured.
|
||||
"""
|
||||
settings = get_settings()
|
||||
discord_url = settings.get_secret("discord-alert-webhook")
|
||||
|
||||
if not discord_url:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Discord alert webhook not configured"
|
||||
)
|
||||
|
||||
manager = get_credential_manager()
|
||||
|
||||
# Create a fake invalid result for testing
|
||||
from backend.services.credential_manager import CredentialCheckResult
|
||||
test_results = [
|
||||
CredentialCheckResult(
|
||||
service="test",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="This is a test alert",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
]
|
||||
|
||||
background_tasks.add_task(
|
||||
manager.send_credential_alert,
|
||||
test_results,
|
||||
discord_url
|
||||
)
|
||||
|
||||
return {"message": "Test alert queued"}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,440 +0,0 @@
|
||||
"""
|
||||
Health check routes.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
import subprocess
|
||||
import platform
|
||||
from fastapi import APIRouter
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.version import VERSION
|
||||
from backend.services.flacfetch_client import get_flacfetch_client
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.services.spacy_preloader import get_preloaded_model, is_model_preloaded
|
||||
from backend.services.nltk_preloader import get_preloaded_cmudict, is_cmudict_preloaded
|
||||
from backend.services.langfuse_preloader import get_preloaded_langfuse_handler, is_langfuse_preloaded, is_langfuse_configured
|
||||
|
||||
router = APIRouter()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_system_info() -> Dict[str, Any]:
|
||||
"""Get system information for performance analysis."""
|
||||
info = {
|
||||
"platform": platform.system(),
|
||||
"platform_release": platform.release(),
|
||||
"python_version": platform.python_version(),
|
||||
}
|
||||
|
||||
# Get CPU info
|
||||
try:
|
||||
cpu_count = os.cpu_count() or 0
|
||||
info["cpu_count"] = cpu_count
|
||||
|
||||
# Try to get more detailed CPU info on Linux
|
||||
if platform.system() == "Linux":
|
||||
try:
|
||||
with open("/proc/cpuinfo", "r") as f:
|
||||
for line in f:
|
||||
if "model name" in line:
|
||||
info["cpu_model"] = line.split(":")[1].strip()
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
except Exception as e:
|
||||
info["cpu_error"] = str(e)
|
||||
|
||||
# Get memory info
|
||||
try:
|
||||
if platform.system() == "Linux":
|
||||
with open("/proc/meminfo", "r") as f:
|
||||
for line in f:
|
||||
if line.startswith("MemTotal:"):
|
||||
mem_kb = int(line.split()[1])
|
||||
info["memory_gb"] = round(mem_kb / (1024 * 1024), 1)
|
||||
break
|
||||
except Exception as e:
|
||||
info["memory_error"] = str(e)
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def get_ffmpeg_info() -> Dict[str, Any]:
|
||||
"""Get FFmpeg version and configuration for debugging encoding performance."""
|
||||
try:
|
||||
# Get FFmpeg version
|
||||
result = subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if result.returncode == 0:
|
||||
lines = result.stdout.split("\n")
|
||||
version_line = lines[0] if lines else "unknown"
|
||||
|
||||
# Check for hardware acceleration support
|
||||
hw_accel = {}
|
||||
config_result = subprocess.run(
|
||||
["ffmpeg", "-hwaccels"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if config_result.returncode == 0:
|
||||
hw_lines = config_result.stdout.strip().split("\n")
|
||||
hw_accel["available"] = [h.strip() for h in hw_lines[1:] if h.strip()]
|
||||
|
||||
# Check encoder availability
|
||||
encoders = {}
|
||||
encoder_result = subprocess.run(
|
||||
["ffmpeg", "-encoders"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=10
|
||||
)
|
||||
if encoder_result.returncode == 0:
|
||||
encoder_output = encoder_result.stdout
|
||||
# Check for specific encoders we care about
|
||||
encoders["libx264"] = "libx264" in encoder_output
|
||||
encoders["h264_nvenc"] = "h264_nvenc" in encoder_output
|
||||
encoders["h264_vaapi"] = "h264_vaapi" in encoder_output
|
||||
encoders["hevc_nvenc"] = "hevc_nvenc" in encoder_output
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"version": version_line,
|
||||
"hw_acceleration": hw_accel,
|
||||
"encoders": encoders,
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"available": False,
|
||||
"error": result.stderr[:500] if result.stderr else "Unknown error",
|
||||
}
|
||||
except subprocess.TimeoutExpired:
|
||||
return {"available": False, "error": "Timeout getting FFmpeg info"}
|
||||
except FileNotFoundError:
|
||||
return {"available": False, "error": "FFmpeg not found"}
|
||||
except Exception as e:
|
||||
return {"available": False, "error": str(e)}
|
||||
|
||||
|
||||
def check_transmission_status() -> Dict[str, Any]:
|
||||
"""Check if Transmission daemon is available and responsive."""
|
||||
try:
|
||||
import transmission_rpc
|
||||
host = os.environ.get("TRANSMISSION_HOST", "localhost")
|
||||
port = int(os.environ.get("TRANSMISSION_PORT", "9091"))
|
||||
|
||||
client = transmission_rpc.Client(host=host, port=port, timeout=5)
|
||||
stats = client.session_stats()
|
||||
|
||||
# Get detailed torrent info
|
||||
torrents = client.get_torrents()
|
||||
torrent_details = []
|
||||
for t in torrents:
|
||||
torrent_info = {
|
||||
"name": t.name,
|
||||
"progress": round(t.progress, 1),
|
||||
"status": str(t.status),
|
||||
"peers": t.peers_connected if hasattr(t, 'peers_connected') else 0,
|
||||
"download_speed": round(t.rate_download / 1024, 1) if hasattr(t, 'rate_download') else 0, # KB/s
|
||||
}
|
||||
# Check if stalled (downloading but no progress and no peers)
|
||||
if t.status.downloading and t.progress < 100 and torrent_info['peers'] == 0:
|
||||
torrent_info['stalled'] = True
|
||||
torrent_details.append(torrent_info)
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"host": host,
|
||||
"port": port,
|
||||
"download_dir": stats.download_dir if hasattr(stats, 'download_dir') else None,
|
||||
"active_torrent_count": stats.active_torrent_count if hasattr(stats, 'active_torrent_count') else 0,
|
||||
"torrents": torrent_details,
|
||||
}
|
||||
except ImportError as e:
|
||||
return {
|
||||
"available": False,
|
||||
"error": f"transmission_rpc not installed: {e}",
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"available": False,
|
||||
"host": os.environ.get("TRANSMISSION_HOST", "localhost"),
|
||||
"port": int(os.environ.get("TRANSMISSION_PORT", "9091")),
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def health_check() -> Dict[str, str]:
|
||||
"""Health check endpoint."""
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
|
||||
async def check_flacfetch_service_status() -> Dict[str, Any]:
|
||||
"""Check if remote flacfetch service is available and healthy."""
|
||||
client = get_flacfetch_client()
|
||||
|
||||
if not client:
|
||||
return {
|
||||
"configured": False,
|
||||
"message": "Remote flacfetch service not configured (FLACFETCH_API_URL not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await client.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"available": True,
|
||||
"status": health.get("status"),
|
||||
"version": health.get("version"),
|
||||
"transmission": health.get("transmission", {}),
|
||||
"disk": health.get("disk", {}),
|
||||
"providers": health.get("providers", {}),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
async def check_encoding_worker_status() -> Dict[str, Any]:
|
||||
"""Check if GCE encoding worker is available and healthy."""
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
if not encoding_service.is_configured:
|
||||
return {
|
||||
"configured": False,
|
||||
"enabled": encoding_service.settings.use_gce_encoding,
|
||||
"message": "GCE encoding worker not configured (ENCODING_WORKER_URL or API key not set)",
|
||||
}
|
||||
|
||||
try:
|
||||
health = await encoding_service.health_check()
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": health.get("status") == "ok",
|
||||
"status": health.get("status"),
|
||||
"active_jobs": health.get("active_jobs", 0),
|
||||
"queue_length": health.get("queue_length", 0),
|
||||
"ffmpeg_version": health.get("ffmpeg_version"),
|
||||
"wheel_version": health.get("wheel_version"),
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"configured": True,
|
||||
"enabled": encoding_service.is_enabled,
|
||||
"available": False,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/encoding-worker")
|
||||
async def encoding_worker_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check encoding worker status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_encoding_worker_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("wheel_version"),
|
||||
"active_jobs": status.get("active_jobs", 0),
|
||||
"queue_length": status.get("queue_length", 0),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/flacfetch")
|
||||
async def flacfetch_health() -> Dict[str, Any]:
|
||||
"""
|
||||
Lightweight endpoint to check flacfetch service status.
|
||||
|
||||
Returns minimal info for frontend footer display.
|
||||
No authentication required.
|
||||
"""
|
||||
status = await check_flacfetch_service_status()
|
||||
|
||||
# Return simplified response for frontend
|
||||
if not status.get("configured"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "not_configured",
|
||||
}
|
||||
|
||||
if not status.get("available"):
|
||||
return {
|
||||
"available": False,
|
||||
"status": "offline",
|
||||
"error": status.get("error"),
|
||||
}
|
||||
|
||||
return {
|
||||
"available": True,
|
||||
"status": "ok",
|
||||
"version": status.get("version"),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/detailed")
|
||||
async def detailed_health_check() -> Dict[str, Any]:
|
||||
"""
|
||||
Detailed health check including dependencies.
|
||||
|
||||
Use this to debug issues with Transmission, flacfetch service, etc.
|
||||
"""
|
||||
transmission_status = check_transmission_status()
|
||||
flacfetch_status = await check_flacfetch_service_status()
|
||||
encoding_status = await check_encoding_worker_status()
|
||||
|
||||
# Check email service
|
||||
email_service = get_email_service()
|
||||
email_status = {
|
||||
"configured": email_service.is_configured(),
|
||||
"provider": type(email_service.provider).__name__,
|
||||
}
|
||||
|
||||
# Check Stripe service
|
||||
stripe_service = get_stripe_service()
|
||||
stripe_status = {
|
||||
"configured": stripe_service.is_configured(),
|
||||
}
|
||||
|
||||
# Log for debugging
|
||||
if not transmission_status.get("available"):
|
||||
logger.warning(f"Local Transmission not available: {transmission_status.get('error')}")
|
||||
else:
|
||||
logger.info(f"Local Transmission available at {transmission_status.get('host')}:{transmission_status.get('port')}")
|
||||
|
||||
if flacfetch_status.get("configured") and not flacfetch_status.get("available"):
|
||||
logger.warning(f"Remote flacfetch service not available: {flacfetch_status.get('error')}")
|
||||
elif flacfetch_status.get("available"):
|
||||
logger.info(f"Remote flacfetch service healthy: {flacfetch_status.get('status')}")
|
||||
|
||||
if not email_status["configured"]:
|
||||
logger.warning("Email service not configured - magic links will not work")
|
||||
|
||||
if not stripe_status["configured"]:
|
||||
logger.warning("Stripe service not configured - payments will not work")
|
||||
|
||||
# Get system and FFmpeg info for performance analysis
|
||||
system_info = get_system_info()
|
||||
ffmpeg_info = get_ffmpeg_info()
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"system": system_info,
|
||||
"ffmpeg": ffmpeg_info,
|
||||
"dependencies": {
|
||||
"transmission_local": transmission_status,
|
||||
"flacfetch_remote": flacfetch_status,
|
||||
"encoding_worker": encoding_status,
|
||||
},
|
||||
"services": {
|
||||
"email": email_status,
|
||||
"stripe": stripe_status,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/health/preload-status")
|
||||
async def preload_status() -> Dict[str, Any]:
|
||||
"""
|
||||
Check status of preloaded resources for performance optimization.
|
||||
|
||||
Use this endpoint to verify that NLTK, SpaCy, and Langfuse resources
|
||||
were successfully preloaded at container startup. If any show as
|
||||
not preloaded, check Cloud Run startup logs for errors.
|
||||
|
||||
Expected state after successful deployment:
|
||||
- spacy.preloaded: true
|
||||
- nltk.preloaded: true
|
||||
- langfuse.preloaded: true (if configured) or configured: false
|
||||
"""
|
||||
# SpaCy status
|
||||
spacy_model = get_preloaded_model("en_core_web_sm")
|
||||
spacy_status = {
|
||||
"preloaded": is_model_preloaded("en_core_web_sm"),
|
||||
"model": "en_core_web_sm",
|
||||
}
|
||||
if spacy_model:
|
||||
spacy_status["vocab_size"] = len(spacy_model.vocab)
|
||||
|
||||
# NLTK status
|
||||
cmudict = get_preloaded_cmudict()
|
||||
nltk_status = {
|
||||
"preloaded": is_cmudict_preloaded(),
|
||||
"resource": "cmudict",
|
||||
}
|
||||
if cmudict:
|
||||
nltk_status["entries"] = len(cmudict)
|
||||
|
||||
# Langfuse status
|
||||
langfuse_handler = get_preloaded_langfuse_handler()
|
||||
langfuse_status = {
|
||||
"configured": is_langfuse_configured(),
|
||||
"preloaded": is_langfuse_preloaded(),
|
||||
}
|
||||
if langfuse_handler:
|
||||
langfuse_status["handler_type"] = type(langfuse_handler).__name__
|
||||
|
||||
# Overall status
|
||||
all_preloaded = (
|
||||
spacy_status["preloaded"]
|
||||
and nltk_status["preloaded"]
|
||||
and (langfuse_status["preloaded"] or not langfuse_status["configured"])
|
||||
)
|
||||
|
||||
return {
|
||||
"status": "ok" if all_preloaded else "degraded",
|
||||
"message": "All resources preloaded" if all_preloaded else "Some resources not preloaded - check startup logs",
|
||||
"spacy": spacy_status,
|
||||
"nltk": nltk_status,
|
||||
"langfuse": langfuse_status,
|
||||
"performance_impact": {
|
||||
"spacy_preload": "Saves ~60s on first lyrics correction",
|
||||
"nltk_preload": "Saves ~100-150s on SyllablesMatchHandler init",
|
||||
"langfuse_preload": "Saves ~200s on AgenticCorrector init",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@router.get("/readiness")
|
||||
async def readiness_check() -> Dict[str, str]:
|
||||
"""Readiness check endpoint for Cloud Run."""
|
||||
return {
|
||||
"status": "ready",
|
||||
"service": "karaoke-gen-backend"
|
||||
}
|
||||
|
||||
@@ -1,636 +0,0 @@
|
||||
"""
|
||||
Internal API routes for worker coordination.
|
||||
|
||||
These endpoints are for internal use only (backend → workers).
|
||||
They are protected by admin authentication.
|
||||
|
||||
With Cloud Tasks integration, these endpoints may be called multiple times
|
||||
(retry on failure). Idempotency checks prevent duplicate processing.
|
||||
|
||||
Observability:
|
||||
- Extracts trace context from incoming requests (propagated via Cloud Tasks)
|
||||
- Creates worker spans linked to the original request trace
|
||||
- All logs include job_id for easy filtering in Cloud Logging
|
||||
"""
|
||||
import logging
|
||||
import asyncio
|
||||
import time
|
||||
from typing import Tuple, Optional
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Request
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.workers.audio_worker import process_audio_separation
|
||||
from backend.workers.lyrics_worker import process_lyrics_transcription
|
||||
from backend.workers.screens_worker import generate_screens
|
||||
from backend.workers.video_worker import generate_video
|
||||
from backend.workers.render_video_worker import process_render_video
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.tracing import (
|
||||
extract_trace_context,
|
||||
start_span_with_context,
|
||||
add_span_attribute,
|
||||
add_span_event,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/internal", tags=["internal"])
|
||||
|
||||
|
||||
class WorkerRequest(BaseModel):
|
||||
"""Request to trigger a worker."""
|
||||
job_id: str
|
||||
|
||||
|
||||
class WorkerResponse(BaseModel):
|
||||
"""Response from worker trigger."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
|
||||
def _check_worker_idempotency(job_id: str, worker_name: str) -> Optional[WorkerResponse]:
|
||||
"""
|
||||
Check if a worker is already running or completed for this job.
|
||||
|
||||
This provides idempotency for Cloud Tasks retries - if a task is retried
|
||||
but the worker is already running or has completed, we skip processing.
|
||||
|
||||
Args:
|
||||
job_id: Job ID to check
|
||||
worker_name: Worker name (audio, lyrics, screens, render, video)
|
||||
|
||||
Returns:
|
||||
WorkerResponse if should skip (already running/complete), None to proceed
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for {worker_name} worker")
|
||||
return WorkerResponse(
|
||||
status="not_found",
|
||||
job_id=job_id,
|
||||
message=f"Job {job_id} not found"
|
||||
)
|
||||
|
||||
# Check worker-specific progress in state_data
|
||||
progress_key = f"{worker_name}_progress"
|
||||
worker_progress = job.state_data.get(progress_key, {})
|
||||
stage = worker_progress.get('stage')
|
||||
|
||||
if stage == 'running':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already running, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_running",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already in progress"
|
||||
)
|
||||
|
||||
if stage == 'complete':
|
||||
logger.info(f"[job:{job_id}] {worker_name.capitalize()} worker already complete, skipping")
|
||||
return WorkerResponse(
|
||||
status="already_complete",
|
||||
job_id=job_id,
|
||||
message=f"{worker_name.capitalize()} worker already completed"
|
||||
)
|
||||
|
||||
# Mark as running before starting (for idempotency on next retry)
|
||||
job_manager.update_state_data(job_id, progress_key, {'stage': 'running'})
|
||||
return None
|
||||
|
||||
|
||||
@router.post("/workers/audio", response_model=WorkerResponse)
|
||||
async def trigger_audio_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger audio separation worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the audio processing track (parallel with lyrics processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request (propagated via Cloud Tasks)
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=audio")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "audio")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "audio")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
# This allows the HTTP response to return immediately
|
||||
# while the worker continues processing
|
||||
background_tasks.add_task(process_audio_separation, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Audio separation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/lyrics", response_model=WorkerResponse)
|
||||
async def trigger_lyrics_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger lyrics transcription worker for a job.
|
||||
|
||||
This endpoint is called internally after job creation to start
|
||||
the lyrics processing track (parallel with audio processing).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
The worker runs in the background and updates job state as it progresses.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=lyrics")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "lyrics")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "lyrics")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_lyrics_transcription, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Lyrics transcription worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/screens", response_model=WorkerResponse)
|
||||
async def trigger_screens_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger title/end screen generation worker.
|
||||
|
||||
This is called automatically when both audio and lyrics are complete.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=screens")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "screens")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "screens")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_screens, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Screens generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/video", response_model=WorkerResponse)
|
||||
async def trigger_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger final video generation and encoding worker.
|
||||
|
||||
This is called after user selects their preferred instrumental.
|
||||
This is the longest-running stage (15-20 minutes).
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "video")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(generate_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Video generation worker started"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/workers/render-video", response_model=WorkerResponse)
|
||||
async def trigger_render_video_worker(
|
||||
request: WorkerRequest,
|
||||
http_request: Request,
|
||||
background_tasks: BackgroundTasks,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Trigger render video worker (post-review).
|
||||
|
||||
This is called after human review is complete.
|
||||
Uses OutputGenerator from LyricsTranscriber to generate the karaoke video
|
||||
with the corrected lyrics.
|
||||
|
||||
Idempotency: If worker is already running or complete, returns early.
|
||||
|
||||
Output: with_vocals.mkv in GCS
|
||||
Next state: INSTRUMENTAL_SELECTED (instrumental is now selected during review)
|
||||
"""
|
||||
job_id = request.job_id
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] WORKER_TRIGGER worker=render-video")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("worker", "render-video")
|
||||
|
||||
# Idempotency check
|
||||
skip_response = _check_worker_idempotency(job_id, "render")
|
||||
if skip_response:
|
||||
add_span_event("worker_skipped", {"reason": skip_response.status})
|
||||
return skip_response
|
||||
|
||||
# Add task to background tasks
|
||||
background_tasks.add_task(process_render_video, job_id)
|
||||
|
||||
add_span_event("worker_started")
|
||||
return WorkerResponse(
|
||||
status="started",
|
||||
job_id=job_id,
|
||||
message="Render video worker started (post-review)"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/jobs/{job_id}/check-idle-reminder")
|
||||
async def check_idle_reminder(
|
||||
job_id: str,
|
||||
http_request: Request,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Check if a job needs an idle reminder email.
|
||||
|
||||
This endpoint is called by a Cloud Tasks scheduled task 5 minutes after
|
||||
a job enters AWAITING_REVIEW state.
|
||||
|
||||
If the job is still in the blocking state and no reminder has been sent yet,
|
||||
sends a reminder email to the user.
|
||||
|
||||
Idempotency: Only one reminder per job (tracked via reminder_sent flag).
|
||||
"""
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_notification_service import get_job_notification_service
|
||||
|
||||
# Extract trace context from incoming request
|
||||
trace_context = extract_trace_context(dict(http_request.headers))
|
||||
|
||||
logger.info(f"[job:{job_id}] IDLE_REMINDER_CHECK starting")
|
||||
add_span_attribute("job_id", job_id)
|
||||
add_span_attribute("operation", "idle_reminder_check")
|
||||
|
||||
job_manager = JobManager()
|
||||
job = job_manager.get_job(job_id)
|
||||
|
||||
if not job:
|
||||
logger.warning(f"[job:{job_id}] Job not found for idle reminder check")
|
||||
add_span_event("job_not_found")
|
||||
return {"status": "not_found", "job_id": job_id, "message": "Job not found"}
|
||||
|
||||
# Check if job is still in a blocking state
|
||||
# Note: AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs only
|
||||
blocking_states = [JobStatus.AWAITING_REVIEW, JobStatus.AWAITING_INSTRUMENTAL_SELECTION]
|
||||
if job.status not in [s.value for s in blocking_states]:
|
||||
logger.info(f"[job:{job_id}] Job no longer in blocking state ({job.status}), skipping reminder")
|
||||
add_span_event("not_blocking", {"current_status": job.status})
|
||||
return {
|
||||
"status": "skipped",
|
||||
"job_id": job_id,
|
||||
"message": f"Job not in blocking state (current: {job.status})"
|
||||
}
|
||||
|
||||
# Normalize state_data to prevent None errors
|
||||
state_data = job.state_data or {}
|
||||
|
||||
# Check if reminder was already sent (idempotency)
|
||||
if state_data.get('reminder_sent'):
|
||||
logger.info(f"[job:{job_id}] Reminder already sent, skipping")
|
||||
add_span_event("already_sent")
|
||||
return {"status": "already_sent", "job_id": job_id, "message": "Reminder already sent"}
|
||||
|
||||
# Skip reminders for made-for-you jobs (admin handles these directly, no intermediate customer emails)
|
||||
if getattr(job, 'made_for_you', False):
|
||||
logger.info(f"[job:{job_id}] Made-for-you job, skipping customer reminder (admin handles)")
|
||||
add_span_event("made_for_you_skip")
|
||||
return {"status": "skipped", "job_id": job_id, "message": "Made-for-you job - admin handles directly"}
|
||||
|
||||
# Check if user has an email
|
||||
if not job.user_email:
|
||||
logger.warning(f"[job:{job_id}] No user email, cannot send reminder")
|
||||
add_span_event("no_email")
|
||||
return {"status": "no_email", "job_id": job_id, "message": "No user email configured"}
|
||||
|
||||
# Determine action type
|
||||
action_type = state_data.get('blocking_action_type')
|
||||
if not action_type:
|
||||
action_type = "lyrics" if job.status == JobStatus.AWAITING_REVIEW.value else "instrumental"
|
||||
|
||||
# Send the reminder email
|
||||
try:
|
||||
notification_service = get_job_notification_service()
|
||||
|
||||
success = await notification_service.send_action_reminder_email(
|
||||
job_id=job.job_id,
|
||||
user_email=job.user_email,
|
||||
action_type=action_type,
|
||||
user_name=None, # Could fetch from user service if needed
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
audio_hash=job.audio_hash,
|
||||
review_token=job.review_token,
|
||||
instrumental_token=job.instrumental_token,
|
||||
)
|
||||
|
||||
if success:
|
||||
# Mark reminder as sent (prevents duplicate sends)
|
||||
job_manager.firestore.update_job(job_id, {
|
||||
'state_data': {**state_data, 'reminder_sent': True}
|
||||
})
|
||||
logger.info(f"[job:{job_id}] Sent {action_type} reminder email to {job.user_email}")
|
||||
add_span_event("reminder_sent", {"action_type": action_type})
|
||||
return {
|
||||
"status": "sent",
|
||||
"job_id": job_id,
|
||||
"message": f"Sent {action_type} reminder to {job.user_email}"
|
||||
}
|
||||
else:
|
||||
logger.error(f"[job:{job_id}] Failed to send reminder email")
|
||||
add_span_event("send_failed")
|
||||
return {"status": "failed", "job_id": job_id, "message": "Failed to send reminder"}
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"[job:{job_id}] Error sending reminder: {e}")
|
||||
add_span_event("error", {"error": str(e)})
|
||||
return {"status": "error", "job_id": job_id, "message": str(e)}
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def internal_health(
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Internal health check endpoint.
|
||||
|
||||
Used to verify the internal API is responsive.
|
||||
Requires admin authentication.
|
||||
"""
|
||||
return {"status": "healthy", "service": "karaoke-backend-internal"}
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Test Webhook Endpoint (for E2E testing)
|
||||
# =============================================================================
|
||||
|
||||
class TestWebhookRequest(BaseModel):
|
||||
"""
|
||||
Request to simulate a Stripe webhook event for E2E testing.
|
||||
|
||||
This allows E2E tests to trigger payment flow logic without requiring
|
||||
actual Stripe checkout sessions or valid webhook signatures.
|
||||
"""
|
||||
event_type: str # e.g., "checkout.session.completed"
|
||||
session_id: str # Must start with "e2e-test-" prefix
|
||||
customer_email: str
|
||||
metadata: dict # order_type, package_id, credits, artist, title, etc.
|
||||
|
||||
|
||||
class TestWebhookResponse(BaseModel):
|
||||
"""Response from test webhook processing."""
|
||||
status: str # "processed", "already_processed", "error"
|
||||
job_id: Optional[str] = None # For made-for-you orders
|
||||
credits_added: Optional[int] = None # For credit purchases
|
||||
new_balance: Optional[int] = None # For credit purchases
|
||||
message: str
|
||||
|
||||
|
||||
@router.post("/test-webhook", response_model=TestWebhookResponse)
|
||||
async def test_webhook(
|
||||
request: TestWebhookRequest,
|
||||
auth_data: Tuple[str, UserType, int] = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Test endpoint that simulates Stripe webhook events for E2E testing.
|
||||
|
||||
SECURITY:
|
||||
- Protected by admin authentication (X-Admin-Token header)
|
||||
- Session IDs must start with "e2e-test-" prefix to prevent collision
|
||||
with real Stripe sessions
|
||||
- Only for E2E testing - bypasses Stripe signature verification
|
||||
|
||||
This endpoint reuses the same handler logic as the real webhook endpoint,
|
||||
ensuring E2E tests validate actual business logic.
|
||||
|
||||
Supported event types:
|
||||
- checkout.session.completed: Handles credit purchases and made-for-you orders
|
||||
|
||||
For credit purchases, metadata must include:
|
||||
- package_id: e.g., "1_credit"
|
||||
- credits: e.g., "1"
|
||||
- user_email: Email of user to credit
|
||||
|
||||
For made-for-you orders, metadata must include:
|
||||
- order_type: "made_for_you"
|
||||
- customer_email: Customer email for delivery
|
||||
- artist: Song artist
|
||||
- title: Song title
|
||||
- source_type: "search" or "youtube"
|
||||
- youtube_url: (optional) If source_type is "youtube"
|
||||
- notes: (optional) Customer notes
|
||||
"""
|
||||
from backend.services.user_service import get_user_service
|
||||
from backend.services.email_service import get_email_service
|
||||
from backend.services.stripe_service import get_stripe_service
|
||||
from backend.api.routes.users import _handle_made_for_you_order
|
||||
|
||||
# Validate session_id prefix for safety
|
||||
if not request.session_id.startswith("e2e-test-"):
|
||||
logger.warning(f"Test webhook rejected: session_id '{request.session_id}' missing required prefix")
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="Session ID must start with 'e2e-test-' prefix for test webhooks"
|
||||
)
|
||||
|
||||
logger.info(f"TEST_WEBHOOK event_type={request.event_type} session_id={request.session_id}")
|
||||
add_span_attribute("event_type", request.event_type)
|
||||
add_span_attribute("session_id", request.session_id)
|
||||
add_span_attribute("is_test_webhook", True)
|
||||
|
||||
user_service = get_user_service()
|
||||
email_service = get_email_service()
|
||||
stripe_service = get_stripe_service()
|
||||
|
||||
if request.event_type == "checkout.session.completed":
|
||||
session_id = request.session_id
|
||||
metadata = request.metadata
|
||||
|
||||
# Idempotency check: Skip if this session was already processed
|
||||
if user_service.is_stripe_session_processed(session_id):
|
||||
logger.info(f"Test webhook: session {session_id} already processed")
|
||||
return TestWebhookResponse(
|
||||
status="already_processed",
|
||||
message=f"Session {session_id} was already processed"
|
||||
)
|
||||
|
||||
# Check if this is a made-for-you order
|
||||
if metadata.get("order_type") == "made_for_you":
|
||||
try:
|
||||
# Call the same handler used by the real webhook
|
||||
await _handle_made_for_you_order(
|
||||
session_id=session_id,
|
||||
metadata=metadata,
|
||||
user_service=user_service,
|
||||
email_service=email_service,
|
||||
)
|
||||
|
||||
# Get the job ID from the most recent job for this customer
|
||||
# The handler creates a job, so we need to find it
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
db = user_service.db
|
||||
# Look for the job by session_id pattern in state_data or by customer_email
|
||||
# Since the job was just created, query by customer_email and made_for_you flag
|
||||
customer_email = metadata.get("customer_email", "")
|
||||
jobs_query = db.collection("jobs").where(
|
||||
filter=FieldFilter("customer_email", "==", customer_email)
|
||||
).where(
|
||||
filter=FieldFilter("made_for_you", "==", True)
|
||||
).order_by("created_at", direction=firestore.Query.DESCENDING).limit(1)
|
||||
|
||||
jobs = list(jobs_query.stream())
|
||||
job_id = jobs[0].to_dict().get("job_id") if jobs else None
|
||||
|
||||
logger.info(f"Test webhook: made-for-you order processed, job_id={job_id}")
|
||||
return TestWebhookResponse(
|
||||
status="processed",
|
||||
job_id=job_id,
|
||||
message=f"Made-for-you order created successfully"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.exception(f"Test webhook: error processing made-for-you order: {e}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Error processing made-for-you order: {str(e)}"
|
||||
)
|
||||
else:
|
||||
# Handle regular credit purchase
|
||||
# Build a synthetic session object that matches Stripe's format
|
||||
synthetic_session = {
|
||||
"id": session_id,
|
||||
"customer_email": request.customer_email,
|
||||
"metadata": metadata,
|
||||
}
|
||||
|
||||
success, user_email, credits, msg = stripe_service.handle_checkout_completed(
|
||||
synthetic_session
|
||||
)
|
||||
|
||||
if not success:
|
||||
logger.warning(f"Test webhook: credit purchase validation failed: {msg}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=msg
|
||||
)
|
||||
|
||||
if user_email and credits > 0:
|
||||
# Add credits to user account
|
||||
ok, new_balance, credit_msg = user_service.add_credits(
|
||||
email=user_email,
|
||||
amount=credits,
|
||||
reason="stripe_purchase",
|
||||
stripe_session_id=session_id,
|
||||
)
|
||||
|
||||
if ok:
|
||||
# Send confirmation email (same as real webhook)
|
||||
email_service.send_credits_added(user_email, credits, new_balance)
|
||||
logger.info(f"Test webhook: added {credits} credits to {user_email}, new balance: {new_balance}")
|
||||
return TestWebhookResponse(
|
||||
status="processed",
|
||||
credits_added=credits,
|
||||
new_balance=new_balance,
|
||||
message=f"Added {credits} credits to {user_email}"
|
||||
)
|
||||
else:
|
||||
logger.error(f"Test webhook: failed to add credits: {credit_msg}")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Failed to add credits: {credit_msg}"
|
||||
)
|
||||
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message="Invalid credit purchase data"
|
||||
)
|
||||
else:
|
||||
# Unsupported event type
|
||||
logger.warning(f"Test webhook: unsupported event type '{request.event_type}'")
|
||||
return TestWebhookResponse(
|
||||
status="error",
|
||||
message=f"Unsupported event type: {request.event_type}"
|
||||
)
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,238 +0,0 @@
|
||||
"""
|
||||
Push Notification API routes.
|
||||
|
||||
Provides endpoints for managing Web Push notification subscriptions:
|
||||
- GET /api/push/vapid-public-key: Get VAPID public key for client-side subscription
|
||||
- POST /api/push/subscribe: Register a push subscription
|
||||
- POST /api/push/unsubscribe: Remove a push subscription
|
||||
- GET /api/push/subscriptions: List user's subscriptions
|
||||
- POST /api/push/test: Send a test notification (admin only)
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, List
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
from backend.config import get_settings
|
||||
from backend.api.dependencies import require_auth, require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.push_notification_service import get_push_notification_service
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/push", tags=["push"])
|
||||
|
||||
|
||||
# Request/Response Models
|
||||
|
||||
class VapidPublicKeyResponse(BaseModel):
|
||||
"""Response containing VAPID public key."""
|
||||
enabled: bool
|
||||
vapid_public_key: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeRequest(BaseModel):
|
||||
"""Request to subscribe to push notifications."""
|
||||
endpoint: str
|
||||
keys: Dict[str, str] # p256dh and auth
|
||||
device_name: Optional[str] = None
|
||||
|
||||
|
||||
class SubscribeResponse(BaseModel):
|
||||
"""Response after subscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class UnsubscribeRequest(BaseModel):
|
||||
"""Request to unsubscribe from push notifications."""
|
||||
endpoint: str
|
||||
|
||||
|
||||
class UnsubscribeResponse(BaseModel):
|
||||
"""Response after unsubscribing."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class SubscriptionInfo(BaseModel):
|
||||
"""Information about a push subscription."""
|
||||
endpoint: str
|
||||
device_name: Optional[str] = None
|
||||
created_at: Optional[str] = None
|
||||
last_used_at: Optional[str] = None
|
||||
|
||||
|
||||
class SubscriptionsListResponse(BaseModel):
|
||||
"""Response containing user's subscriptions."""
|
||||
subscriptions: List[SubscriptionInfo]
|
||||
count: int
|
||||
|
||||
|
||||
class TestNotificationRequest(BaseModel):
|
||||
"""Request to send a test notification."""
|
||||
title: Optional[str] = "Test Notification"
|
||||
body: Optional[str] = "This is a test push notification from Karaoke Generator"
|
||||
|
||||
|
||||
class TestNotificationResponse(BaseModel):
|
||||
"""Response after sending test notification."""
|
||||
status: str
|
||||
sent_count: int
|
||||
message: str
|
||||
|
||||
|
||||
# Routes
|
||||
|
||||
@router.get("/vapid-public-key", response_model=VapidPublicKeyResponse)
|
||||
async def get_vapid_public_key():
|
||||
"""
|
||||
Get the VAPID public key for push subscription.
|
||||
|
||||
This endpoint is public - no authentication required.
|
||||
Returns the public key needed for client-side PushManager.subscribe().
|
||||
"""
|
||||
settings = get_settings()
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
if not settings.enable_push_notifications:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
public_key = push_service.get_public_key()
|
||||
if not public_key:
|
||||
return VapidPublicKeyResponse(enabled=False)
|
||||
|
||||
return VapidPublicKeyResponse(
|
||||
enabled=True,
|
||||
vapid_public_key=public_key
|
||||
)
|
||||
|
||||
|
||||
@router.post("/subscribe", response_model=SubscribeResponse)
|
||||
async def subscribe_push(
|
||||
request: SubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Register a push notification subscription for the current user.
|
||||
|
||||
Requires authentication. Users can have up to 5 subscriptions
|
||||
(configurable via MAX_PUSH_SUBSCRIPTIONS_PER_USER).
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
# Validate keys
|
||||
if "p256dh" not in request.keys or "auth" not in request.keys:
|
||||
raise HTTPException(status_code=400, detail="Missing required keys (p256dh, auth)")
|
||||
|
||||
success = await push_service.add_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint,
|
||||
keys=request.keys,
|
||||
device_name=request.device_name
|
||||
)
|
||||
|
||||
if not success:
|
||||
raise HTTPException(status_code=500, detail="Failed to save subscription")
|
||||
|
||||
return SubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription registered successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/unsubscribe", response_model=UnsubscribeResponse)
|
||||
async def unsubscribe_push(
|
||||
request: UnsubscribeRequest,
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
Remove a push notification subscription.
|
||||
|
||||
Requires authentication. Users can only remove their own subscriptions.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
success = await push_service.remove_subscription(
|
||||
user_email=auth_result.user_email,
|
||||
endpoint=request.endpoint
|
||||
)
|
||||
|
||||
if not success:
|
||||
# Don't error if subscription wasn't found - might already be removed
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Subscription removed (or was not found)"
|
||||
)
|
||||
|
||||
return UnsubscribeResponse(
|
||||
status="success",
|
||||
message="Push subscription removed successfully"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/subscriptions", response_model=SubscriptionsListResponse)
|
||||
async def list_subscriptions(
|
||||
auth_result: AuthResult = Depends(require_auth)
|
||||
):
|
||||
"""
|
||||
List all push notification subscriptions for the current user.
|
||||
|
||||
Requires authentication.
|
||||
"""
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
subscriptions = await push_service.list_subscriptions(auth_result.user_email)
|
||||
|
||||
return SubscriptionsListResponse(
|
||||
subscriptions=[SubscriptionInfo(**s) for s in subscriptions],
|
||||
count=len(subscriptions)
|
||||
)
|
||||
|
||||
|
||||
@router.post("/test", response_model=TestNotificationResponse)
|
||||
async def send_test_notification(
|
||||
request: TestNotificationRequest,
|
||||
auth_result: AuthResult = Depends(require_admin)
|
||||
):
|
||||
"""
|
||||
Send a test push notification to the current user's devices.
|
||||
|
||||
Admin only. Useful for testing push notification setup.
|
||||
"""
|
||||
settings = get_settings()
|
||||
if not settings.enable_push_notifications:
|
||||
raise HTTPException(status_code=503, detail="Push notifications are not enabled")
|
||||
|
||||
if not auth_result.user_email:
|
||||
raise HTTPException(status_code=401, detail="User email not available")
|
||||
|
||||
push_service = get_push_notification_service()
|
||||
|
||||
sent_count = await push_service.send_push(
|
||||
user_email=auth_result.user_email,
|
||||
title=request.title or "Test Notification",
|
||||
body=request.body or "This is a test push notification",
|
||||
url="/app/",
|
||||
tag="test"
|
||||
)
|
||||
|
||||
return TestNotificationResponse(
|
||||
status="success",
|
||||
sent_count=sent_count,
|
||||
message=f"Test notification sent to {sent_count} device(s)"
|
||||
)
|
||||
@@ -1,428 +0,0 @@
|
||||
"""
|
||||
Admin API routes for rate limit management.
|
||||
|
||||
Handles:
|
||||
- Rate limit statistics and monitoring
|
||||
- Blocklist management (disposable domains, blocked emails, blocked IPs)
|
||||
- User override management (whitelist/bypass permissions)
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from pydantic import BaseModel, EmailStr
|
||||
|
||||
from backend.api.dependencies import require_admin
|
||||
from backend.services.auth_service import AuthResult
|
||||
from backend.services.rate_limit_service import get_rate_limit_service, RateLimitService
|
||||
from backend.services.email_validation_service import get_email_validation_service, EmailValidationService
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/admin/rate-limits", tags=["admin", "rate-limits"])
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Response Models
|
||||
# =============================================================================
|
||||
|
||||
class RateLimitStatsResponse(BaseModel):
|
||||
"""Current rate limit statistics."""
|
||||
# Configuration
|
||||
jobs_per_day_limit: int
|
||||
youtube_uploads_per_day_limit: int
|
||||
beta_ip_per_day_limit: int
|
||||
rate_limiting_enabled: bool
|
||||
|
||||
# Current usage
|
||||
youtube_uploads_today: int
|
||||
youtube_uploads_remaining: int
|
||||
|
||||
# Blocklist stats
|
||||
disposable_domains_count: int
|
||||
blocked_emails_count: int
|
||||
blocked_ips_count: int
|
||||
|
||||
# User override stats
|
||||
total_overrides: int
|
||||
|
||||
|
||||
class UserRateLimitStatusResponse(BaseModel):
|
||||
"""Rate limit status for a specific user."""
|
||||
email: str
|
||||
jobs_today: int
|
||||
jobs_limit: int
|
||||
jobs_remaining: int
|
||||
has_bypass: bool
|
||||
custom_limit: Optional[int]
|
||||
bypass_reason: Optional[str]
|
||||
|
||||
|
||||
class BlocklistsResponse(BaseModel):
|
||||
"""All blocklist data."""
|
||||
disposable_domains: List[str]
|
||||
blocked_emails: List[str]
|
||||
blocked_ips: List[str]
|
||||
updated_at: Optional[datetime]
|
||||
updated_by: Optional[str]
|
||||
|
||||
|
||||
class DomainRequest(BaseModel):
|
||||
"""Request to add/remove a domain."""
|
||||
domain: str
|
||||
|
||||
|
||||
class EmailRequest(BaseModel):
|
||||
"""Request to add/remove an email."""
|
||||
email: str
|
||||
|
||||
|
||||
class IPRequest(BaseModel):
|
||||
"""Request to add/remove an IP."""
|
||||
ip_address: str
|
||||
|
||||
|
||||
class UserOverride(BaseModel):
|
||||
"""User override configuration."""
|
||||
email: str
|
||||
bypass_job_limit: bool
|
||||
custom_daily_job_limit: Optional[int]
|
||||
reason: str
|
||||
created_by: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class UserOverrideRequest(BaseModel):
|
||||
"""Request to set/update a user override."""
|
||||
bypass_job_limit: bool = False
|
||||
custom_daily_job_limit: Optional[int] = None
|
||||
reason: str
|
||||
|
||||
|
||||
class UserOverridesListResponse(BaseModel):
|
||||
"""List of all user overrides."""
|
||||
overrides: List[UserOverride]
|
||||
total: int
|
||||
|
||||
|
||||
class SuccessResponse(BaseModel):
|
||||
"""Generic success response."""
|
||||
success: bool
|
||||
message: str
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Statistics Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/stats", response_model=RateLimitStatsResponse)
|
||||
async def get_rate_limit_stats(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get current rate limit statistics.
|
||||
|
||||
Returns configuration values, current usage counts, and blocklist stats.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Get YouTube uploads today
|
||||
youtube_today = rate_limit_service.get_youtube_uploads_today()
|
||||
youtube_limit = settings.rate_limit_youtube_uploads_per_day
|
||||
youtube_remaining = max(0, youtube_limit - youtube_today) if youtube_limit > 0 else -1
|
||||
|
||||
# Get blocklist stats
|
||||
blocklist_stats = email_validation.get_blocklist_stats()
|
||||
|
||||
# Get override count
|
||||
overrides = rate_limit_service.get_all_overrides()
|
||||
|
||||
return RateLimitStatsResponse(
|
||||
jobs_per_day_limit=settings.rate_limit_jobs_per_day,
|
||||
youtube_uploads_per_day_limit=settings.rate_limit_youtube_uploads_per_day,
|
||||
beta_ip_per_day_limit=settings.rate_limit_beta_ip_per_day,
|
||||
rate_limiting_enabled=settings.enable_rate_limiting,
|
||||
youtube_uploads_today=youtube_today,
|
||||
youtube_uploads_remaining=youtube_remaining,
|
||||
disposable_domains_count=blocklist_stats["disposable_domains_count"],
|
||||
blocked_emails_count=blocklist_stats["blocked_emails_count"],
|
||||
blocked_ips_count=blocklist_stats["blocked_ips_count"],
|
||||
total_overrides=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.get("/users/{email}", response_model=UserRateLimitStatusResponse)
|
||||
async def get_user_rate_limit_status(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get rate limit status for a specific user.
|
||||
|
||||
Returns their current usage and any override settings.
|
||||
"""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
jobs_today = rate_limit_service.get_user_job_count_today(email)
|
||||
override = rate_limit_service.get_user_override(email)
|
||||
|
||||
if override:
|
||||
has_bypass = override.get("bypass_job_limit", False)
|
||||
custom_limit = override.get("custom_daily_job_limit")
|
||||
bypass_reason = override.get("reason")
|
||||
jobs_limit = custom_limit if custom_limit is not None else settings.rate_limit_jobs_per_day
|
||||
else:
|
||||
has_bypass = False
|
||||
custom_limit = None
|
||||
bypass_reason = None
|
||||
jobs_limit = settings.rate_limit_jobs_per_day
|
||||
|
||||
jobs_remaining = max(0, jobs_limit - jobs_today) if not has_bypass else -1
|
||||
|
||||
return UserRateLimitStatusResponse(
|
||||
email=email,
|
||||
jobs_today=jobs_today,
|
||||
jobs_limit=jobs_limit,
|
||||
jobs_remaining=jobs_remaining,
|
||||
has_bypass=has_bypass,
|
||||
custom_limit=custom_limit,
|
||||
bypass_reason=bypass_reason,
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Blocklist Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/blocklists", response_model=BlocklistsResponse)
|
||||
async def get_blocklists(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""
|
||||
Get all blocklist data.
|
||||
|
||||
Returns disposable domains, blocked emails, and blocked IPs.
|
||||
"""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
# Force refresh to get latest data
|
||||
config = email_validation.get_blocklist_config(force_refresh=True)
|
||||
|
||||
# Get the raw document for metadata
|
||||
from google.cloud import firestore
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
from backend.services.email_validation_service import BLOCKLISTS_COLLECTION, BLOCKLIST_CONFIG_DOC
|
||||
|
||||
db = get_firestore_client()
|
||||
doc = db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC).get()
|
||||
|
||||
updated_at = None
|
||||
updated_by = None
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
updated_at = data.get("updated_at")
|
||||
updated_by = data.get("updated_by")
|
||||
|
||||
return BlocklistsResponse(
|
||||
disposable_domains=sorted(list(config["disposable_domains"])),
|
||||
blocked_emails=sorted(list(config["blocked_emails"])),
|
||||
blocked_ips=sorted(list(config["blocked_ips"])),
|
||||
updated_at=updated_at,
|
||||
updated_by=updated_by,
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/disposable-domains", response_model=SuccessResponse)
|
||||
async def add_disposable_domain(
|
||||
request: DomainRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = request.domain.lower().strip()
|
||||
if not domain or "." not in domain:
|
||||
raise HTTPException(status_code=400, detail="Invalid domain format")
|
||||
|
||||
email_validation.add_disposable_domain(domain, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' added to disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/disposable-domains/{domain}", response_model=SuccessResponse)
|
||||
async def remove_disposable_domain(
|
||||
domain: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
domain = domain.lower().strip()
|
||||
if not email_validation.remove_disposable_domain(domain, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Domain not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Domain '{domain}' removed from disposable domains blocklist"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-emails", response_model=SuccessResponse)
|
||||
async def add_blocked_email(
|
||||
request: EmailRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an email to the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = request.email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
email_validation.add_blocked_email(email, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' added to blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-emails/{email}", response_model=SuccessResponse)
|
||||
async def remove_blocked_email(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email_validation.remove_blocked_email(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Email not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Email '{email}' removed from blocked emails list"
|
||||
)
|
||||
|
||||
|
||||
@router.post("/blocklists/blocked-ips", response_model=SuccessResponse)
|
||||
async def add_blocked_ip(
|
||||
request: IPRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = request.ip_address.strip()
|
||||
if not ip_address:
|
||||
raise HTTPException(status_code=400, detail="Invalid IP address")
|
||||
|
||||
email_validation.add_blocked_ip(ip_address, auth_result.user_email)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' added to blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/blocklists/blocked-ips/{ip_address}", response_model=SuccessResponse)
|
||||
async def remove_blocked_ip(
|
||||
ip_address: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
email_validation = get_email_validation_service()
|
||||
|
||||
ip_address = ip_address.strip()
|
||||
if not email_validation.remove_blocked_ip(ip_address, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="IP not found in blocklist")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"IP '{ip_address}' removed from blocked IPs list"
|
||||
)
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# User Override Management Endpoints
|
||||
# =============================================================================
|
||||
|
||||
@router.get("/overrides", response_model=UserOverridesListResponse)
|
||||
async def get_all_overrides(
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Get all user overrides."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
overrides_data = rate_limit_service.get_all_overrides()
|
||||
|
||||
overrides = [
|
||||
UserOverride(
|
||||
email=email,
|
||||
bypass_job_limit=data.get("bypass_job_limit", False),
|
||||
custom_daily_job_limit=data.get("custom_daily_job_limit"),
|
||||
reason=data.get("reason", ""),
|
||||
created_by=data.get("created_by", "unknown"),
|
||||
created_at=data.get("created_at", datetime.now(timezone.utc)),
|
||||
)
|
||||
for email, data in overrides_data.items()
|
||||
]
|
||||
|
||||
return UserOverridesListResponse(
|
||||
overrides=overrides,
|
||||
total=len(overrides),
|
||||
)
|
||||
|
||||
|
||||
@router.put("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def set_user_override(
|
||||
email: str,
|
||||
request: UserOverrideRequest,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Set or update a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not email or "@" not in email:
|
||||
raise HTTPException(status_code=400, detail="Invalid email format")
|
||||
|
||||
if not request.reason or len(request.reason.strip()) < 3:
|
||||
raise HTTPException(status_code=400, detail="Reason is required (min 3 characters)")
|
||||
|
||||
rate_limit_service.set_user_override(
|
||||
user_email=email,
|
||||
bypass_job_limit=request.bypass_job_limit,
|
||||
custom_daily_job_limit=request.custom_daily_job_limit,
|
||||
reason=request.reason,
|
||||
admin_email=auth_result.user_email,
|
||||
)
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override set for user '{email}'"
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/overrides/{email}", response_model=SuccessResponse)
|
||||
async def remove_user_override(
|
||||
email: str,
|
||||
auth_result: AuthResult = Depends(require_admin),
|
||||
):
|
||||
"""Remove a user override."""
|
||||
rate_limit_service = get_rate_limit_service()
|
||||
|
||||
email = email.lower().strip()
|
||||
if not rate_limit_service.remove_user_override(email, auth_result.user_email):
|
||||
raise HTTPException(status_code=404, detail="Override not found for user")
|
||||
|
||||
return SuccessResponse(
|
||||
success=True,
|
||||
message=f"Override removed for user '{email}'"
|
||||
)
|
||||
@@ -1,873 +0,0 @@
|
||||
"""
|
||||
Review API routes - Combined lyrics + instrumental review.
|
||||
|
||||
These endpoints support the combined review flow where users review lyrics
|
||||
AND select their instrumental in a single session.
|
||||
|
||||
The correction-data endpoint now includes instrumental options and backing
|
||||
vocals analysis, and the complete endpoint requires instrumental selection.
|
||||
|
||||
Usage:
|
||||
Frontend URL: http://localhost:5173/?baseApiUrl=http://localhost:8000/api/review/{job_id}
|
||||
|
||||
The baseApiUrl includes the job_id, and all endpoints are relative to that.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Set, Tuple
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, Depends
|
||||
from fastapi.responses import FileResponse, StreamingResponse
|
||||
from starlette.background import BackgroundTask
|
||||
|
||||
from backend.models.job import JobStatus
|
||||
from backend.services.job_manager import JobManager
|
||||
from backend.services.storage_service import StorageService
|
||||
from backend.services.job_logging import job_log_context, JobLogger
|
||||
from backend.services.tracing import create_span, add_span_attribute, add_span_event
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
from backend.api.dependencies import require_auth, require_review_auth
|
||||
from backend.services.auth_service import UserType
|
||||
from backend.config import get_settings
|
||||
|
||||
# LyricsTranscriber imports for preview generation
|
||||
from karaoke_gen.lyrics_transcriber.types import CorrectionResult
|
||||
from karaoke_gen.lyrics_transcriber.core.config import OutputConfig
|
||||
from karaoke_gen.lyrics_transcriber.correction.operations import CorrectionOperations
|
||||
|
||||
# Import from the unified style loader
|
||||
from karaoke_gen.style_loader import load_styles_from_gcs
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
router = APIRouter(prefix="/review", tags=["review"])
|
||||
|
||||
# Store job context for the session
|
||||
# In production, this would be handled differently (e.g., session tokens)
|
||||
_job_contexts: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# Store preview video paths for serving
|
||||
_preview_videos: Dict[str, Dict[str, str]] = {}
|
||||
|
||||
# Keep references to background tasks to prevent garbage collection
|
||||
_background_tasks: Set[asyncio.Task] = set()
|
||||
|
||||
|
||||
def _get_audio_hash(job_id: str) -> str:
|
||||
"""Generate a consistent audio hash for a job."""
|
||||
return hashlib.md5(job_id.encode()).hexdigest()
|
||||
|
||||
|
||||
@router.get("/{job_id}/ping")
|
||||
async def ping(job_id: str):
|
||||
"""Health check endpoint expected by frontend."""
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/correction-data")
|
||||
async def get_correction_data(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get correction data for the combined review interface.
|
||||
|
||||
Returns the CorrectionResult data plus instrumental options and analysis
|
||||
for the combined lyrics + instrumental review UI.
|
||||
|
||||
Response includes:
|
||||
- All lyrics correction data (segments, reference lyrics, anchors, etc.)
|
||||
- instrumental_options: List of available instrumental tracks with audio URLs
|
||||
- backing_vocals_analysis: Analysis data to help users choose instrumental
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not ready for review (current status: {job.status})"
|
||||
)
|
||||
|
||||
# Check for updated corrections first (from previous review sessions where user edited lyrics)
|
||||
# This mirrors the logic in render_video_worker.py to ensure consistency
|
||||
corrections_updated_gcs = job.file_urls.get('lyrics', {}).get('corrections_updated')
|
||||
if not corrections_updated_gcs:
|
||||
# Try direct path for updated corrections
|
||||
corrections_updated_gcs = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
|
||||
if corrections_updated_gcs and storage.file_exists(corrections_updated_gcs):
|
||||
corrections_gcs = corrections_updated_gcs
|
||||
logger.info(f"Job {job_id}: Using updated corrections from previous review")
|
||||
else:
|
||||
# Fall back to original corrections
|
||||
corrections_gcs = job.file_urls.get('lyrics', {}).get('corrections')
|
||||
if not corrections_gcs:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
if not storage.file_exists(corrections_gcs):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail="Corrections data not found. Lyrics processing may not be complete."
|
||||
)
|
||||
|
||||
# Download and return corrections data
|
||||
try:
|
||||
corrections_data = storage.download_json(corrections_gcs)
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if 'metadata' not in corrections_data:
|
||||
corrections_data['metadata'] = {}
|
||||
corrections_data['metadata']['audio_hash'] = audio_hash
|
||||
corrections_data['metadata']['artist'] = job.artist
|
||||
corrections_data['metadata']['title'] = job.title
|
||||
|
||||
# Store context for audio serving
|
||||
_job_contexts[job_id] = {
|
||||
'audio_hash': audio_hash,
|
||||
'audio_gcs_path': job.input_media_gcs_path
|
||||
}
|
||||
|
||||
# === Add instrumental data for combined review ===
|
||||
|
||||
# Get instrumental stem URLs
|
||||
stems = job.file_urls.get('stems', {})
|
||||
clean_url = stems.get('instrumental_clean')
|
||||
backing_url = stems.get('instrumental_with_backing')
|
||||
|
||||
# Build instrumental options with signed URLs
|
||||
instrumental_options = []
|
||||
if clean_url:
|
||||
instrumental_options.append({
|
||||
"id": "clean",
|
||||
"label": "Clean Instrumental",
|
||||
"description": "No backing vocals - just the music",
|
||||
"audio_url": storage.generate_signed_url(clean_url, expiration_minutes=120),
|
||||
})
|
||||
if backing_url:
|
||||
instrumental_options.append({
|
||||
"id": "with_backing",
|
||||
"label": "Instrumental with Backing Vocals",
|
||||
"description": "Includes harmonies and background vocals",
|
||||
"audio_url": storage.generate_signed_url(backing_url, expiration_minutes=120),
|
||||
})
|
||||
|
||||
corrections_data['instrumental_options'] = instrumental_options
|
||||
|
||||
# Get backing vocals analysis from state_data (populated by screens_worker)
|
||||
backing_vocals_analysis = job.state_data.get('backing_vocals_analysis', {})
|
||||
corrections_data['backing_vocals_analysis'] = backing_vocals_analysis
|
||||
|
||||
# Get waveform URL if available
|
||||
analysis_files = job.file_urls.get('analysis', {})
|
||||
waveform_url = analysis_files.get('backing_vocals_waveform')
|
||||
if waveform_url:
|
||||
corrections_data['backing_vocals_waveform_url'] = storage.generate_signed_url(
|
||||
waveform_url, expiration_minutes=120
|
||||
)
|
||||
|
||||
# Transition to IN_REVIEW if not already
|
||||
if job.status == JobStatus.AWAITING_REVIEW:
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.IN_REVIEW,
|
||||
message="User opened combined review interface"
|
||||
)
|
||||
|
||||
logger.info(f"Job {job_id}: Serving correction data with instrumental options for combined review")
|
||||
return corrections_data
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error loading corrections: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error loading corrections: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/{audio_hash}")
|
||||
async def get_audio_with_hash(
|
||||
job_id: str,
|
||||
audio_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (with hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
@router.get("/{job_id}/audio/")
|
||||
@router.get("/{job_id}/audio")
|
||||
async def get_audio_no_hash(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the audio file for playback (without hash parameter)."""
|
||||
return await _stream_audio(job_id)
|
||||
|
||||
|
||||
async def _stream_audio(job_id: str):
|
||||
"""
|
||||
Stream the audio file for playback in the review interface.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
audio_gcs_path = job.input_media_gcs_path
|
||||
if not audio_gcs_path:
|
||||
raise HTTPException(status_code=404, detail="Audio file not found")
|
||||
|
||||
# Download to temp file and stream
|
||||
try:
|
||||
# Create temp file
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".flac") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(audio_gcs_path, tmp_path)
|
||||
|
||||
# Determine content type
|
||||
if audio_gcs_path.endswith('.flac'):
|
||||
media_type = "audio/flac"
|
||||
elif audio_gcs_path.endswith('.wav'):
|
||||
media_type = "audio/wav"
|
||||
elif audio_gcs_path.endswith('.mp3'):
|
||||
media_type = "audio/mpeg"
|
||||
else:
|
||||
media_type = "audio/mpeg"
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming audio for review")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type=media_type,
|
||||
filename=os.path.basename(audio_gcs_path),
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming audio: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming audio: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/complete")
|
||||
async def complete_review(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Complete the combined review - save corrected lyrics AND instrumental selection.
|
||||
|
||||
This endpoint receives:
|
||||
- Updated correction data (lyrics corrections)
|
||||
- instrumental_selection: "clean" or "with_backing" (REQUIRED)
|
||||
|
||||
After saving, triggers the render video worker which will use the
|
||||
pre-selected instrumental (no separate instrumental selection step).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
# === Require instrumental selection ===
|
||||
instrumental_selection = updated_data.get("instrumental_selection")
|
||||
if not instrumental_selection:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail="instrumental_selection is required. Must be 'clean' or 'with_backing'."
|
||||
)
|
||||
|
||||
valid_selections = ["clean", "with_backing"]
|
||||
# Also allow "custom" for user-provided instrumentals
|
||||
if job.existing_instrumental_gcs_path:
|
||||
valid_selections.append("custom")
|
||||
|
||||
if instrumental_selection not in valid_selections:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Invalid instrumental_selection. Must be one of: {valid_selections}"
|
||||
)
|
||||
|
||||
try:
|
||||
# Remove instrumental_selection from updated_data before saving to corrections
|
||||
# (it's stored separately in state_data)
|
||||
corrections_to_save = {k: v for k, v in updated_data.items() if k != "instrumental_selection"}
|
||||
|
||||
# Save updated corrections to GCS
|
||||
corrections_gcs_path = f"jobs/{job_id}/lyrics/corrections_updated.json"
|
||||
storage.upload_json(corrections_gcs_path, corrections_to_save)
|
||||
job_manager.update_file_url(job_id, 'lyrics', 'corrections_updated', corrections_gcs_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Saved updated corrections")
|
||||
|
||||
# === Store instrumental selection in state_data ===
|
||||
# This will be used by render_video_worker to skip AWAITING_INSTRUMENTAL_SELECTION
|
||||
job_manager.update_state_data(job_id, 'instrumental_selection', instrumental_selection)
|
||||
logger.info(f"Job {job_id}: Stored instrumental selection: {instrumental_selection}")
|
||||
|
||||
# Clear worker progress keys to ensure workers will run fresh (not skip due to idempotency)
|
||||
# This handles cases where a job is re-reviewed after completion or where state was inconsistent.
|
||||
# The idempotency check in workers looks at {worker}_progress.stage == 'complete' to skip,
|
||||
# so we must clear these before triggering the render-video worker.
|
||||
progress_keys = ['render_progress', 'screens_progress', 'video_progress', 'encoding_progress']
|
||||
cleared = job_manager.delete_state_data_keys(job_id, progress_keys)
|
||||
if cleared:
|
||||
logger.info(f"Job {job_id}: Cleared worker progress keys for re-processing: {cleared}")
|
||||
|
||||
# Transition to REVIEW_COMPLETE
|
||||
job_manager.transition_to_state(
|
||||
job_id=job_id,
|
||||
new_status=JobStatus.REVIEW_COMPLETE,
|
||||
progress=70,
|
||||
message=f"Review complete (instrumental: {instrumental_selection}), rendering video"
|
||||
)
|
||||
|
||||
# Trigger render video worker
|
||||
from backend.services.worker_service import get_worker_service
|
||||
worker_service = get_worker_service()
|
||||
|
||||
# Run in background, keep reference to prevent garbage collection
|
||||
task = asyncio.create_task(worker_service.trigger_render_video_worker(job_id))
|
||||
_background_tasks.add(task)
|
||||
task.add_done_callback(_background_tasks.discard)
|
||||
|
||||
logger.info(f"Job {job_id}: Combined review complete, triggered render video worker")
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"instrumental_selection": instrumental_selection,
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error completing review: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error completing review: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/handlers")
|
||||
async def update_handlers(
|
||||
job_id: str,
|
||||
enabled_handlers: list,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Update enabled correction handlers (optional feature).
|
||||
|
||||
For now, just acknowledge the request - full handler support
|
||||
would require re-running correction.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Handler update requested (not implemented)")
|
||||
return {"status": "success", "message": "Handler updates not yet implemented"}
|
||||
|
||||
|
||||
@router.post("/{job_id}/add-lyrics")
|
||||
async def add_lyrics(
|
||||
job_id: str,
|
||||
data: Dict[str, str],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Add custom lyrics source and rerun correction.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to add a new lyrics source
|
||||
and regenerate corrections with the new source included.
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to add lyrics
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
source = data.get("source", "").strip()
|
||||
lyrics_text = data.get("lyrics", "").strip()
|
||||
|
||||
logger.info(f"Job {job_id}: Adding lyrics source '{source}' with {len(lyrics_text)} characters")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("add-lyrics", {"job_id": job_id, "source": source, "lyrics_length": len(lyrics_text)}) as span:
|
||||
with job_log_context(job_id, worker="add-lyrics"):
|
||||
try:
|
||||
# Create temp directory for this operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download current corrections.json
|
||||
with create_span("download-corrections") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
download_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# Load as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded", {
|
||||
"segments": len(correction_result.corrected_segments) if correction_result.corrected_segments else 0,
|
||||
"reference_sources": len(correction_result.reference_lyrics) if correction_result.reference_lyrics else 0,
|
||||
})
|
||||
|
||||
# Set up cache directory
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
# Add lyrics source using CorrectionOperations (this is the heavy operation)
|
||||
with create_span("correction-operations-add-lyrics") as correction_span:
|
||||
correction_span.set_attribute("source_name", source)
|
||||
updated_result = CorrectionOperations.add_lyrics_source(
|
||||
correction_result=correction_result,
|
||||
source=source,
|
||||
lyrics_text=lyrics_text,
|
||||
cache_dir=cache_dir,
|
||||
logger=logger
|
||||
)
|
||||
add_span_event("correction_complete", {
|
||||
"new_segments": len(updated_result.corrected_segments) if updated_result.corrected_segments else 0,
|
||||
})
|
||||
|
||||
# Add audio hash for the frontend
|
||||
audio_hash = _get_audio_hash(job_id)
|
||||
if not updated_result.metadata:
|
||||
updated_result.metadata = {}
|
||||
updated_result.metadata['audio_hash'] = audio_hash
|
||||
updated_result.metadata['artist'] = job.artist
|
||||
updated_result.metadata['title'] = job.title
|
||||
|
||||
# Upload updated corrections back to GCS
|
||||
with create_span("upload-corrections") as upload_span:
|
||||
updated_data = updated_result.to_dict()
|
||||
storage.upload_json(corrections_gcs, updated_data)
|
||||
upload_span.set_attribute("gcs_path", corrections_gcs)
|
||||
|
||||
logger.info(f"Job {job_id}: Successfully added lyrics source '{source}'")
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "data": updated_data}
|
||||
|
||||
except ValueError as e:
|
||||
# ValueError from CorrectionOperations (e.g., duplicate source name)
|
||||
logger.warning(f"Job {job_id}: Invalid add lyrics request: {e}")
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to add lyrics: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(status_code=500, detail=f"Failed to add lyrics: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/preview-video")
|
||||
async def generate_preview_video(
|
||||
job_id: str,
|
||||
updated_data: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Generate a preview video with the current corrections.
|
||||
|
||||
Uses the LyricsTranscriber's CorrectionOperations to generate a 360p preview
|
||||
video with the user's current corrections applied.
|
||||
|
||||
When USE_GCE_PREVIEW_ENCODING is enabled, video encoding is offloaded to
|
||||
the high-performance GCE worker for faster generation (15-20s vs 60+s).
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
settings = get_settings()
|
||||
encoding_service = get_encoding_service()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Job must be in review state to generate preview
|
||||
if job.status not in [JobStatus.AWAITING_REVIEW, JobStatus.IN_REVIEW]:
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=f"Job not in review state (current status: {job.status})"
|
||||
)
|
||||
|
||||
# Check if GCE preview encoding is enabled
|
||||
use_gce_preview = encoding_service.is_preview_enabled
|
||||
|
||||
# Check if user wants theme background image (slower) or black background (faster, default)
|
||||
use_background_image = updated_data.get("use_background_image", False)
|
||||
logger.info(f"Job {job_id}: Generating preview video (GCE: {use_gce_preview}, background image: {use_background_image})")
|
||||
|
||||
# Use tracing and job_log_context for full observability
|
||||
with create_span("generate-preview-video", {"job_id": job_id, "use_gce": use_gce_preview}) as span:
|
||||
with job_log_context(job_id, worker="preview"):
|
||||
try:
|
||||
# Create temp directory for this preview operation
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# 1. Download original corrections.json (has full structure)
|
||||
with create_span("download-corrections-and-audio") as download_span:
|
||||
corrections_gcs = f"jobs/{job_id}/lyrics/corrections.json"
|
||||
corrections_path = os.path.join(temp_dir, "corrections.json")
|
||||
storage.download_file(corrections_gcs, corrections_path)
|
||||
|
||||
with open(corrections_path, 'r', encoding='utf-8') as f:
|
||||
original_data = json.load(f)
|
||||
|
||||
# 2. Download input audio
|
||||
audio_path = os.path.join(temp_dir, "audio.flac")
|
||||
storage.download_file(job.input_media_gcs_path, audio_path)
|
||||
download_span.set_attribute("audio_gcs_path", job.input_media_gcs_path)
|
||||
|
||||
# 3. Load original as CorrectionResult
|
||||
correction_result = CorrectionResult.from_dict(original_data)
|
||||
add_span_event("corrections_loaded")
|
||||
|
||||
# 4. Get or create styles file for preview using unified style loader
|
||||
with create_span("load-styles") as styles_span:
|
||||
styles_path, _ = load_styles_from_gcs(
|
||||
style_params_gcs_path=job.style_params_gcs_path,
|
||||
style_assets=job.style_assets,
|
||||
temp_dir=temp_dir,
|
||||
download_func=storage.download_file,
|
||||
logger=logger,
|
||||
)
|
||||
styles_span.set_attribute("styles_path", styles_path)
|
||||
|
||||
# 5. Set up output config for preview
|
||||
output_dir = os.path.join(temp_dir, "output")
|
||||
cache_dir = os.path.join(temp_dir, "cache")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
|
||||
output_config = OutputConfig(
|
||||
output_styles_json=styles_path,
|
||||
output_dir=output_dir,
|
||||
cache_dir=cache_dir,
|
||||
video_resolution="360p",
|
||||
)
|
||||
|
||||
# 6. Generate preview (ASS-only if using GCE, or full video if local)
|
||||
preview_gcs_path = None
|
||||
|
||||
if use_gce_preview:
|
||||
# GCE path: Generate ASS only, then offload encoding to GCE
|
||||
try:
|
||||
with create_span("generate-ass-subtitles") as ass_span:
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=True, # Only generate ASS, skip video encoding
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
ass_path = result["ass_path"]
|
||||
ass_span.set_attribute("ass_path", ass_path)
|
||||
add_span_event("ass_generated")
|
||||
|
||||
# Upload ASS to GCS
|
||||
with create_span("upload-ass-to-gcs") as upload_ass_span:
|
||||
ass_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.ass"
|
||||
storage.upload_file(ass_path, ass_gcs_path)
|
||||
upload_ass_span.set_attribute("ass_gcs_path", ass_gcs_path)
|
||||
|
||||
# Call GCE encoding service
|
||||
with create_span("gce-preview-encoding") as gce_span:
|
||||
bucket_name = settings.gcs_bucket_name
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
|
||||
# Get background image and font from style assets if available
|
||||
style_assets = job.style_assets or {}
|
||||
|
||||
# Only use background image if user explicitly requested it
|
||||
# Default is black background for faster preview generation (~10s vs ~30-60s)
|
||||
background_image_gcs_path = None
|
||||
if use_background_image:
|
||||
for key in ["karaoke_background", "style_karaoke_background"]:
|
||||
if key in style_assets:
|
||||
background_image_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("background_image", background_image_gcs_path)
|
||||
break
|
||||
gce_span.set_attribute("use_background_image", use_background_image)
|
||||
|
||||
font_gcs_path = None
|
||||
for key in ["font", "style_font"]:
|
||||
if key in style_assets:
|
||||
font_gcs_path = f"gs://{bucket_name}/{style_assets[key]}"
|
||||
gce_span.set_attribute("font", font_gcs_path)
|
||||
break
|
||||
|
||||
gce_result = await encoding_service.encode_preview_video(
|
||||
job_id=f"preview_{job_id}_{preview_hash}",
|
||||
ass_gcs_path=f"gs://{bucket_name}/{ass_gcs_path}",
|
||||
audio_gcs_path=f"gs://{bucket_name}/{job.input_media_gcs_path}",
|
||||
output_gcs_path=f"gs://{bucket_name}/{preview_gcs_path}",
|
||||
background_color="black",
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
gce_span.set_attribute("gce_status", gce_result.get("status"))
|
||||
add_span_event("gce_encoding_complete")
|
||||
|
||||
logger.info(f"Job {job_id}: Preview generated via GCE: {preview_hash}")
|
||||
|
||||
except Exception as gce_error:
|
||||
# Fall back to local encoding if GCE fails
|
||||
logger.warning(
|
||||
f"Job {job_id}: GCE preview encoding failed, falling back to local: {gce_error}"
|
||||
)
|
||||
span.set_attribute("gce_fallback", True)
|
||||
use_gce_preview = False # Fall through to local encoding below
|
||||
|
||||
if not use_gce_preview:
|
||||
# Local path: Generate full preview video locally
|
||||
with create_span("render-preview-video-local") as render_span:
|
||||
render_span.set_attribute("resolution", "360p")
|
||||
result = CorrectionOperations.generate_preview_video(
|
||||
correction_result=correction_result,
|
||||
updated_data=updated_data,
|
||||
output_config=output_config,
|
||||
audio_filepath=audio_path,
|
||||
artist=job.artist,
|
||||
title=job.title,
|
||||
logger=logger,
|
||||
ass_only=False, # Generate full video locally
|
||||
)
|
||||
preview_hash = result["preview_hash"]
|
||||
video_path = result["video_path"]
|
||||
add_span_event("render_complete")
|
||||
|
||||
# Upload preview video to GCS
|
||||
with create_span("upload-preview-video") as upload_span:
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
storage.upload_file(video_path, preview_gcs_path)
|
||||
upload_span.set_attribute("gcs_path", preview_gcs_path)
|
||||
|
||||
# Store the GCS path for serving
|
||||
if job_id not in _preview_videos:
|
||||
_preview_videos[job_id] = {}
|
||||
_preview_videos[job_id][preview_hash] = preview_gcs_path
|
||||
|
||||
logger.info(f"Job {job_id}: Preview video generated: {preview_hash}")
|
||||
span.set_attribute("preview_hash", preview_hash)
|
||||
span.set_attribute("success", True)
|
||||
|
||||
return {"status": "success", "preview_hash": preview_hash}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Failed to generate preview video: {e}", exc_info=True)
|
||||
span.set_attribute("error", str(e))
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to generate preview video: {e}"
|
||||
) from e
|
||||
|
||||
|
||||
@router.get("/{job_id}/preview-video/{preview_hash}")
|
||||
async def get_preview_video(
|
||||
job_id: str,
|
||||
preview_hash: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Stream the generated preview video."""
|
||||
storage = StorageService()
|
||||
|
||||
# Check in-memory cache first
|
||||
preview_gcs_path = None
|
||||
if job_id in _preview_videos and preview_hash in _preview_videos[job_id]:
|
||||
preview_gcs_path = _preview_videos[job_id][preview_hash]
|
||||
else:
|
||||
# Try standard path
|
||||
preview_gcs_path = f"jobs/{job_id}/previews/{preview_hash}.mp4"
|
||||
if not storage.file_exists(preview_gcs_path):
|
||||
raise HTTPException(status_code=404, detail="Preview video not found")
|
||||
|
||||
try:
|
||||
# Download to temp file and stream
|
||||
with tempfile.NamedTemporaryFile(delete=False, suffix=".mp4") as tmp:
|
||||
tmp_path = tmp.name
|
||||
|
||||
storage.download_file(preview_gcs_path, tmp_path)
|
||||
|
||||
logger.info(f"Job {job_id}: Streaming preview video {preview_hash}")
|
||||
|
||||
return FileResponse(
|
||||
tmp_path,
|
||||
media_type="video/mp4",
|
||||
filename=f"preview_{preview_hash}.mp4",
|
||||
headers={
|
||||
"Accept-Ranges": "bytes",
|
||||
"Content-Disposition": "inline",
|
||||
"Cache-Control": "no-cache",
|
||||
},
|
||||
background=BackgroundTask(os.unlink, tmp_path),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error streaming preview video: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error streaming preview video: {str(e)}")
|
||||
|
||||
|
||||
@router.post("/{job_id}/v1/annotations")
|
||||
async def submit_annotation(
|
||||
job_id: str,
|
||||
annotation: Dict[str, Any],
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Submit a correction annotation for ML training data.
|
||||
|
||||
For now, just log and acknowledge - full annotation support
|
||||
would require a database.
|
||||
"""
|
||||
logger.info(f"Job {job_id}: Annotation submitted (logged but not stored)")
|
||||
return {"status": "success", "annotation_id": "stub"}
|
||||
|
||||
|
||||
@router.get("/{job_id}/v1/annotations/stats")
|
||||
async def get_annotation_stats(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""Get annotation statistics."""
|
||||
return {
|
||||
"total_annotations": 0,
|
||||
"by_type": {},
|
||||
"message": "Annotation stats not yet implemented"
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{job_id}/instrumental-analysis")
|
||||
async def get_instrumental_analysis(
|
||||
job_id: str,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get instrumental analysis data for the review interface.
|
||||
|
||||
Returns analysis data in the format expected by InstrumentalSelector component:
|
||||
- analysis: backing vocals analysis with audible segments
|
||||
- audio_urls: signed URLs for instrumental stems
|
||||
- has_original: whether original audio is available
|
||||
"""
|
||||
job_manager = JobManager()
|
||||
storage = StorageService()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Get backing vocals analysis from state_data
|
||||
backing_analysis = job.state_data.get('backing_vocals_analysis', {})
|
||||
|
||||
# Get stem URLs
|
||||
stems = job.file_urls.get('stems', {})
|
||||
clean_url = stems.get('instrumental_clean')
|
||||
backing_url = stems.get('instrumental_with_backing')
|
||||
|
||||
# Build audio URLs with signed access
|
||||
audio_urls = {}
|
||||
if clean_url:
|
||||
audio_urls['clean'] = storage.generate_signed_url(clean_url, expiration_minutes=120)
|
||||
if backing_url:
|
||||
audio_urls['with_backing'] = storage.generate_signed_url(backing_url, expiration_minutes=120)
|
||||
if job.input_media_gcs_path:
|
||||
audio_urls['original'] = storage.generate_signed_url(job.input_media_gcs_path, expiration_minutes=120)
|
||||
|
||||
# Check for backing vocals stem for playback
|
||||
backing_vocals_path = stems.get('backing_vocals')
|
||||
if backing_vocals_path:
|
||||
audio_urls['backing_vocals'] = storage.generate_signed_url(backing_vocals_path, expiration_minutes=120)
|
||||
|
||||
# Format response to match InstrumentalAnalysis type
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"artist": job.artist,
|
||||
"title": job.title,
|
||||
"duration_seconds": backing_analysis.get('total_duration_seconds'),
|
||||
"analysis": {
|
||||
"has_audible_content": backing_analysis.get('has_audible_content', False),
|
||||
"total_duration_seconds": backing_analysis.get('total_duration_seconds', 0),
|
||||
"audible_segments": backing_analysis.get('audible_segments', []),
|
||||
"recommended_selection": backing_analysis.get('recommended_selection', 'with_backing'),
|
||||
"total_audible_duration_seconds": backing_analysis.get('total_audible_duration_seconds', 0),
|
||||
"audible_percentage": backing_analysis.get('audible_percentage', 0),
|
||||
"silence_threshold_db": backing_analysis.get('silence_threshold_db', -40),
|
||||
},
|
||||
"audio_urls": audio_urls,
|
||||
"has_original": bool(job.input_media_gcs_path),
|
||||
"has_uploaded_instrumental": bool(job.existing_instrumental_gcs_path),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{job_id}/waveform-data")
|
||||
async def get_waveform_data(
|
||||
job_id: str,
|
||||
num_points: int = 1000,
|
||||
auth_info: Tuple[str, str] = Depends(require_review_auth)
|
||||
):
|
||||
"""
|
||||
Get waveform amplitude data for client-side rendering.
|
||||
|
||||
Returns amplitude values for drawing the waveform in the frontend.
|
||||
"""
|
||||
from backend.services.audio_analysis_service import AudioAnalysisService
|
||||
|
||||
job_manager = JobManager()
|
||||
|
||||
job = job_manager.get_job(job_id)
|
||||
if not job:
|
||||
raise HTTPException(status_code=404, detail="Job not found")
|
||||
|
||||
# Get backing vocals path
|
||||
stems = job.file_urls.get('stems', {})
|
||||
backing_vocals_path = stems.get('backing_vocals')
|
||||
|
||||
if not backing_vocals_path:
|
||||
# Fallback to input audio if no backing vocals
|
||||
backing_vocals_path = job.input_media_gcs_path
|
||||
|
||||
if not backing_vocals_path:
|
||||
raise HTTPException(status_code=404, detail="No audio available for waveform")
|
||||
|
||||
try:
|
||||
analysis_service = AudioAnalysisService()
|
||||
amplitudes, duration = analysis_service.get_waveform_data(
|
||||
gcs_audio_path=backing_vocals_path,
|
||||
job_id=job_id,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
return {
|
||||
"amplitudes": amplitudes,
|
||||
"duration_seconds": duration,
|
||||
"duration": duration, # Legacy field name
|
||||
"sample_rate": num_points,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Job {job_id}: Error generating waveform data: {e}", exc_info=True)
|
||||
raise HTTPException(status_code=500, detail=f"Error generating waveform: {str(e)}")
|
||||
@@ -1,120 +0,0 @@
|
||||
"""
|
||||
Tenant API routes for white-label portal configuration.
|
||||
|
||||
These endpoints are public (no auth required) since the frontend needs
|
||||
to fetch tenant branding before the user logs in.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Query, Request
|
||||
|
||||
from backend.models.tenant import TenantConfigResponse, TenantPublicConfig
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/tenant", tags=["tenant"])
|
||||
|
||||
|
||||
@router.get("/config", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config(
|
||||
request: Request,
|
||||
tenant: Optional[str] = Query(
|
||||
None,
|
||||
description="Tenant ID override (for development). If not provided, detected from subdomain.",
|
||||
),
|
||||
):
|
||||
"""
|
||||
Get tenant configuration for the current portal.
|
||||
|
||||
This endpoint detects the tenant from:
|
||||
1. Query parameter `tenant` (for development/testing)
|
||||
2. X-Tenant-ID header (set by frontend)
|
||||
3. Host header subdomain
|
||||
|
||||
Returns the public tenant configuration (branding, features, defaults)
|
||||
or indicates this is the default Nomad Karaoke portal.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
|
||||
# Priority 1: Query parameter (for dev/testing)
|
||||
tenant_id = tenant
|
||||
|
||||
# Priority 2: X-Tenant-ID header
|
||||
if not tenant_id:
|
||||
tenant_id = request.headers.get("X-Tenant-ID")
|
||||
|
||||
# Priority 3: Detect from Host header
|
||||
if not tenant_id:
|
||||
host = request.headers.get("Host", "")
|
||||
# Check if this is a tenant subdomain
|
||||
# Pattern: {tenant}.nomadkaraoke.com or {tenant}.gen.nomadkaraoke.com
|
||||
if host and "nomadkaraoke.com" in host.lower():
|
||||
parts = host.lower().split(".")
|
||||
# Skip known non-tenant subdomains
|
||||
if parts[0] not in ["gen", "api", "www", "buy", "admin"]:
|
||||
potential_tenant = parts[0]
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
tenant_id = potential_tenant
|
||||
|
||||
# If no tenant detected, return default config
|
||||
if not tenant_id:
|
||||
logger.debug("No tenant detected, returning default config")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Load tenant config
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
logger.warning(f"Tenant not found: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
logger.info(f"Returning config for tenant: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/config/{tenant_id}", response_model=TenantConfigResponse)
|
||||
async def get_tenant_config_by_id(tenant_id: str):
|
||||
"""
|
||||
Get tenant configuration by explicit tenant ID.
|
||||
|
||||
This is useful for admin tools or debugging.
|
||||
"""
|
||||
tenant_service = get_tenant_service()
|
||||
public_config = tenant_service.get_public_config(tenant_id)
|
||||
|
||||
if not public_config:
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
# Check if tenant is active
|
||||
if not public_config.is_active:
|
||||
logger.warning(f"Tenant is inactive: {tenant_id}")
|
||||
return TenantConfigResponse(tenant=None, is_default=True)
|
||||
|
||||
return TenantConfigResponse(tenant=public_config, is_default=False)
|
||||
|
||||
|
||||
@router.get("/asset/{tenant_id}/{asset_name}")
|
||||
async def get_tenant_asset(tenant_id: str, asset_name: str):
|
||||
"""
|
||||
Get a signed URL for a tenant asset (logo, favicon, etc.).
|
||||
|
||||
This redirects to the signed GCS URL for the asset.
|
||||
"""
|
||||
from fastapi.responses import RedirectResponse
|
||||
|
||||
tenant_service = get_tenant_service()
|
||||
url = tenant_service.get_asset_url(tenant_id, asset_name)
|
||||
|
||||
if not url:
|
||||
from fastapi import HTTPException
|
||||
|
||||
raise HTTPException(status_code=404, detail="Asset not found")
|
||||
|
||||
return RedirectResponse(url=url)
|
||||
@@ -1,162 +0,0 @@
|
||||
"""
|
||||
Theme API routes for listing and retrieving theme information.
|
||||
|
||||
Themes are pre-created style configurations that users can select when creating jobs.
|
||||
These endpoints are public (no authentication required) to allow theme selection
|
||||
before job creation.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
|
||||
from backend.models.theme import ThemeDetailResponse, ThemesListResponse
|
||||
from backend.services.theme_service import get_theme_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/themes", tags=["themes"])
|
||||
|
||||
|
||||
@router.get("", response_model=ThemesListResponse)
|
||||
async def list_themes() -> ThemesListResponse:
|
||||
"""
|
||||
List all available themes.
|
||||
|
||||
Returns a list of themes with their metadata and preview image URLs.
|
||||
Preview URLs are signed and valid for 60 minutes.
|
||||
|
||||
This endpoint is public and does not require authentication.
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
themes = theme_service.list_themes()
|
||||
return ThemesListResponse(themes=themes)
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing themes: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load themes"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}", response_model=ThemeDetailResponse)
|
||||
async def get_theme(theme_id: str) -> ThemeDetailResponse:
|
||||
"""
|
||||
Get detailed information about a specific theme.
|
||||
|
||||
Returns the full theme details including style parameters.
|
||||
This can be used to preview theme settings or inspect configuration.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
ThemeDetailResponse with full theme details
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
|
||||
if theme is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
return ThemeDetailResponse(theme=theme)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/preview")
|
||||
async def get_theme_preview(theme_id: str) -> dict:
|
||||
"""
|
||||
Get a signed URL for the theme's preview image.
|
||||
|
||||
This endpoint is useful when you need just the preview URL
|
||||
without loading the full theme details.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with preview_url field
|
||||
|
||||
Raises:
|
||||
404: Theme not found or no preview available
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
theme = theme_service.get_theme(theme_id)
|
||||
if theme is None or theme.preview_url is None:
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"No preview available for theme: {theme_id}"
|
||||
)
|
||||
|
||||
return {"preview_url": theme.preview_url}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting preview for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load theme preview"
|
||||
)
|
||||
|
||||
|
||||
@router.get("/{theme_id}/youtube-description")
|
||||
async def get_theme_youtube_description(theme_id: str) -> dict:
|
||||
"""
|
||||
Get the YouTube description template for a theme.
|
||||
|
||||
Some themes include a default YouTube video description template
|
||||
that can be used when uploading videos.
|
||||
|
||||
Args:
|
||||
theme_id: The unique identifier of the theme
|
||||
|
||||
Returns:
|
||||
Object with description field containing the template text,
|
||||
or null if no template is available
|
||||
|
||||
Raises:
|
||||
404: Theme not found
|
||||
"""
|
||||
try:
|
||||
theme_service = get_theme_service()
|
||||
|
||||
if not theme_service.theme_exists(theme_id):
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"Theme not found: {theme_id}"
|
||||
)
|
||||
|
||||
description = theme_service.get_youtube_description(theme_id)
|
||||
return {"description": description}
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting YouTube description for theme {theme_id}: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail="Failed to load YouTube description"
|
||||
)
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,207 +0,0 @@
|
||||
"""
|
||||
Configuration management for the karaoke generation backend.
|
||||
"""
|
||||
import os
|
||||
import logging
|
||||
from typing import Optional, Dict
|
||||
from pydantic_settings import BaseSettings
|
||||
from google.cloud import secretmanager
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
"""Application settings."""
|
||||
|
||||
# Google Cloud
|
||||
google_cloud_project: str = os.getenv("GOOGLE_CLOUD_PROJECT", "")
|
||||
gcs_bucket_name: str = os.getenv("GCS_BUCKET_NAME", "karaoke-gen-storage")
|
||||
gcs_temp_bucket: str = os.getenv("GCS_TEMP_BUCKET", "karaoke-gen-temp")
|
||||
gcs_output_bucket: str = os.getenv("GCS_OUTPUT_BUCKET", "karaoke-gen-outputs")
|
||||
firestore_collection: str = os.getenv("FIRESTORE_COLLECTION", "jobs")
|
||||
|
||||
# Audio Separator API (for GPU processing)
|
||||
audio_separator_api_url: Optional[str] = os.getenv("AUDIO_SEPARATOR_API_URL")
|
||||
|
||||
# External APIs (can be set via env or Secret Manager)
|
||||
audioshake_api_key: Optional[str] = os.getenv("AUDIOSHAKE_API_KEY")
|
||||
genius_api_key: Optional[str] = os.getenv("GENIUS_API_KEY")
|
||||
spotify_cookie: Optional[str] = os.getenv("SPOTIFY_COOKIE_SP_DC")
|
||||
rapidapi_key: Optional[str] = os.getenv("RAPIDAPI_KEY")
|
||||
|
||||
# Authentication
|
||||
admin_tokens: Optional[str] = os.getenv("ADMIN_TOKENS") # Comma-separated list
|
||||
|
||||
# Application
|
||||
environment: str = os.getenv("ENVIRONMENT", "development")
|
||||
log_level: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
|
||||
# Processing
|
||||
max_concurrent_jobs: int = int(os.getenv("MAX_CONCURRENT_JOBS", "5"))
|
||||
job_timeout_seconds: int = int(os.getenv("JOB_TIMEOUT_SECONDS", "3600"))
|
||||
|
||||
# Lyrics Correction Settings
|
||||
# Skip all auto-correction (both agentic AI and heuristic handlers).
|
||||
# When true, raw transcription goes directly to human review without any automatic fixes.
|
||||
# This is the default because auto-correction currently creates more work for reviewers.
|
||||
# Set SKIP_CORRECTION=false to re-enable auto-correction if quality improves.
|
||||
skip_correction: bool = os.getenv("SKIP_CORRECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Agentic AI Correction (for lyrics correction via LLM)
|
||||
# Only used when skip_correction=false
|
||||
# When enabled, uses Gemini via Vertex AI for intelligent lyrics correction
|
||||
use_agentic_ai: bool = os.getenv("USE_AGENTIC_AI", "true").lower() in ("true", "1", "yes")
|
||||
agentic_ai_model: str = os.getenv("AGENTIC_AI_MODEL", "vertexai/gemini-3-flash-preview")
|
||||
# Timeout for agentic correction in seconds. If correction takes longer, abort and
|
||||
# use uncorrected transcription - human review will fix any issues.
|
||||
agentic_correction_timeout_seconds: int = int(os.getenv("AGENTIC_CORRECTION_TIMEOUT_SECONDS", "180"))
|
||||
|
||||
# Cloud Tasks (for scalable worker coordination)
|
||||
# When enabled, workers are triggered via Cloud Tasks for guaranteed delivery
|
||||
# When disabled (default), workers are triggered via direct HTTP (for development)
|
||||
enable_cloud_tasks: bool = os.getenv("ENABLE_CLOUD_TASKS", "false").lower() in ("true", "1", "yes")
|
||||
gcp_region: str = os.getenv("GCP_REGION", "us-central1")
|
||||
|
||||
# Cloud Run Jobs (for long-running video encoding)
|
||||
# When enabled AND enable_cloud_tasks is true, video worker uses Cloud Run Jobs
|
||||
# instead of Cloud Tasks. This supports encoding times >30 minutes (up to 24 hours).
|
||||
# Default is false - Cloud Tasks is sufficient for most videos (15-20 min).
|
||||
use_cloud_run_jobs_for_video: bool = os.getenv("USE_CLOUD_RUN_JOBS_FOR_VIDEO", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# GCE Encoding Worker (for high-performance video encoding)
|
||||
# When enabled, video encoding is offloaded to a dedicated C4 GCE instance
|
||||
# with faster CPU (Intel Granite Rapids 3.9 GHz) instead of Cloud Run.
|
||||
# This provides 2-3x faster encoding times for CPU-bound FFmpeg libx264 encoding.
|
||||
use_gce_encoding: bool = os.getenv("USE_GCE_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
encoding_worker_url: Optional[str] = os.getenv("ENCODING_WORKER_URL") # e.g., http://136.119.50.148:8080
|
||||
encoding_worker_api_key: Optional[str] = os.getenv("ENCODING_WORKER_API_KEY")
|
||||
|
||||
# GCE Preview Encoding (for faster preview video generation)
|
||||
# When enabled, preview video encoding during lyrics review is offloaded to the GCE worker.
|
||||
# This reduces preview generation time from 60+ seconds to ~15-20 seconds.
|
||||
# Requires use_gce_encoding to be enabled and the GCE worker to support /encode-preview endpoint.
|
||||
use_gce_preview_encoding: bool = os.getenv("USE_GCE_PREVIEW_ENCODING", "false").lower() in ("true", "1", "yes")
|
||||
|
||||
# Storage paths
|
||||
temp_dir: str = os.getenv("TEMP_DIR", "/tmp/karaoke-gen")
|
||||
|
||||
# Worker logs storage mode
|
||||
# When enabled, worker logs are stored in a Firestore subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array. This avoids the 1MB document size limit.
|
||||
# Default is true for new deployments.
|
||||
use_log_subcollection: bool = os.getenv("USE_LOG_SUBCOLLECTION", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Flacfetch remote service (for torrent downloads)
|
||||
# When configured, audio search uses the remote flacfetch HTTP API instead of local flacfetch.
|
||||
# This is required for torrent downloads since Cloud Run doesn't support BitTorrent.
|
||||
flacfetch_api_url: Optional[str] = os.getenv("FLACFETCH_API_URL") # e.g., http://10.0.0.5:8080
|
||||
flacfetch_api_key: Optional[str] = os.getenv("FLACFETCH_API_KEY")
|
||||
|
||||
# Default distribution settings (can be overridden per-request)
|
||||
default_dropbox_path: Optional[str] = os.getenv("DEFAULT_DROPBOX_PATH")
|
||||
default_gdrive_folder_id: Optional[str] = os.getenv("DEFAULT_GDRIVE_FOLDER_ID")
|
||||
# Strip whitespace/newlines from webhook URL - common issue when env vars are set with trailing newlines
|
||||
default_discord_webhook_url: Optional[str] = (
|
||||
os.getenv("DEFAULT_DISCORD_WEBHOOK_URL", "").strip() or None
|
||||
)
|
||||
|
||||
# Default values for web service jobs (YouTube/Dropbox distribution)
|
||||
default_enable_youtube_upload: bool = os.getenv("DEFAULT_ENABLE_YOUTUBE_UPLOAD", "false").lower() in ("true", "1", "yes")
|
||||
default_brand_prefix: Optional[str] = os.getenv("DEFAULT_BRAND_PREFIX")
|
||||
|
||||
# Rate Limiting Configuration
|
||||
# Enable/disable rate limiting system-wide (useful for development)
|
||||
enable_rate_limiting: bool = os.getenv("ENABLE_RATE_LIMITING", "true").lower() in ("true", "1", "yes")
|
||||
# Maximum jobs a user can create per day (0 = unlimited)
|
||||
rate_limit_jobs_per_day: int = int(os.getenv("RATE_LIMIT_JOBS_PER_DAY", "5"))
|
||||
# Maximum YouTube uploads system-wide per day (0 = unlimited)
|
||||
rate_limit_youtube_uploads_per_day: int = int(os.getenv("RATE_LIMIT_YOUTUBE_UPLOADS_PER_DAY", "10"))
|
||||
# Maximum beta enrollments from same IP per day (0 = unlimited)
|
||||
rate_limit_beta_ip_per_day: int = int(os.getenv("RATE_LIMIT_BETA_IP_PER_DAY", "1"))
|
||||
|
||||
# E2E test bypass key for rate limiting (set via secret in production)
|
||||
e2e_bypass_key: str = os.getenv("E2E_BYPASS_KEY", "")
|
||||
default_youtube_description: str = os.getenv(
|
||||
"DEFAULT_YOUTUBE_DESCRIPTION",
|
||||
"Karaoke video created with Nomad Karaoke (https://nomadkaraoke.com)\n\n"
|
||||
"AI-powered vocal separation and synchronized lyrics.\n\n"
|
||||
"#karaoke #music #singing #instrumental #lyrics"
|
||||
)
|
||||
|
||||
# Default CDG/TXT generation settings
|
||||
# When True, CDG and TXT packages are generated by default (when a theme is set)
|
||||
# These can be overridden per-request via explicit enable_cdg/enable_txt parameters
|
||||
default_enable_cdg: bool = os.getenv("DEFAULT_ENABLE_CDG", "true").lower() in ("true", "1", "yes")
|
||||
default_enable_txt: bool = os.getenv("DEFAULT_ENABLE_TXT", "true").lower() in ("true", "1", "yes")
|
||||
|
||||
# Push Notifications Configuration
|
||||
# When enabled, users can subscribe to push notifications for job status updates
|
||||
enable_push_notifications: bool = os.getenv("ENABLE_PUSH_NOTIFICATIONS", "false").lower() in ("true", "1", "yes")
|
||||
# Maximum number of push subscriptions per user (oldest removed when exceeded)
|
||||
max_push_subscriptions_per_user: int = int(os.getenv("MAX_PUSH_SUBSCRIPTIONS_PER_USER", "5"))
|
||||
# VAPID subject (email or URL for push service to contact)
|
||||
vapid_subject: str = os.getenv("VAPID_SUBJECT", "mailto:gen@nomadkaraoke.com")
|
||||
|
||||
# Secret Manager cache
|
||||
_secret_cache: Dict[str, str] = {}
|
||||
|
||||
class Config:
|
||||
env_file = ".env"
|
||||
case_sensitive = False
|
||||
|
||||
def get_secret(self, secret_id: str) -> Optional[str]:
|
||||
"""
|
||||
Get a secret from Google Secret Manager.
|
||||
|
||||
Caches secrets in memory to avoid repeated API calls.
|
||||
Falls back to environment variables if Secret Manager unavailable.
|
||||
|
||||
Args:
|
||||
secret_id: Secret name (e.g., "audioshake-api-key")
|
||||
|
||||
Returns:
|
||||
Secret value or None if not found
|
||||
"""
|
||||
# Check cache first
|
||||
if secret_id in self._secret_cache:
|
||||
return self._secret_cache[secret_id]
|
||||
|
||||
# Check environment variable (development mode)
|
||||
env_var = secret_id.upper().replace('-', '_')
|
||||
env_value = os.getenv(env_var)
|
||||
if env_value:
|
||||
logger.debug(f"Using {secret_id} from environment variable")
|
||||
self._secret_cache[secret_id] = env_value
|
||||
return env_value
|
||||
|
||||
# Try Secret Manager (production mode)
|
||||
if not self.google_cloud_project:
|
||||
logger.warning(f"Cannot fetch secret {secret_id}: No GCP project configured")
|
||||
return None
|
||||
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
name = f"projects/{self.google_cloud_project}/secrets/{secret_id}/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
# Strip whitespace/newlines - common issue when secrets are created with trailing newlines
|
||||
secret_value = response.payload.data.decode('UTF-8').strip()
|
||||
|
||||
# Cache it
|
||||
self._secret_cache[secret_id] = secret_value
|
||||
logger.info(f"Loaded secret {secret_id} from Secret Manager")
|
||||
return secret_value
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load secret {secret_id} from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Global settings instance
|
||||
settings = Settings()
|
||||
|
||||
|
||||
def get_settings() -> Settings:
|
||||
"""Get the global settings instance."""
|
||||
return settings
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
"""
|
||||
Custom exceptions for the karaoke generation backend.
|
||||
|
||||
These exceptions are used for structured error handling across the application,
|
||||
particularly for rate limiting and validation errors.
|
||||
"""
|
||||
|
||||
|
||||
class RateLimitExceededError(Exception):
|
||||
"""
|
||||
Raised when a rate limit is exceeded.
|
||||
|
||||
Includes information about when the limit will reset for client retry logic.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
limit_type: Type of limit exceeded (e.g., "jobs_per_day", "youtube_uploads")
|
||||
remaining_seconds: Seconds until the limit resets (for Retry-After header)
|
||||
current_count: Current usage count
|
||||
limit_value: The limit that was exceeded
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
limit_type: str = "unknown",
|
||||
remaining_seconds: int = 0,
|
||||
current_count: int = 0,
|
||||
limit_value: int = 0
|
||||
):
|
||||
self.message = message
|
||||
self.limit_type = limit_type
|
||||
self.remaining_seconds = remaining_seconds
|
||||
self.current_count = current_count
|
||||
self.limit_value = limit_value
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class EmailValidationError(Exception):
|
||||
"""
|
||||
Raised when email validation fails.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
reason: Specific reason for failure (e.g., "disposable", "blocked", "invalid")
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, reason: str = "invalid"):
|
||||
self.message = message
|
||||
self.reason = reason
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
class IPBlockedError(Exception):
|
||||
"""
|
||||
Raised when a request comes from a blocked IP address.
|
||||
|
||||
Attributes:
|
||||
message: Human-readable error message
|
||||
ip_address: The blocked IP address (may be partially masked)
|
||||
"""
|
||||
|
||||
def __init__(self, message: str, ip_address: str = ""):
|
||||
self.message = message
|
||||
self.ip_address = ip_address
|
||||
super().__init__(message)
|
||||
-203
@@ -1,203 +0,0 @@
|
||||
"""
|
||||
FastAPI application entry point for karaoke generation backend.
|
||||
"""
|
||||
import logging
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from backend.config import settings
|
||||
from backend.api.routes import health, jobs, internal, file_upload, review, auth, audio_search, themes, users, admin, tenant, rate_limits, push
|
||||
from backend.services.tracing import setup_tracing, instrument_app, get_current_trace_id
|
||||
from backend.services.structured_logging import setup_structured_logging
|
||||
from backend.services.spacy_preloader import preload_spacy_model
|
||||
from backend.services.nltk_preloader import preload_all_nltk_resources
|
||||
from backend.services.langfuse_preloader import preload_langfuse_handler
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware
|
||||
from backend.workers.registry import worker_registry
|
||||
|
||||
|
||||
from backend.version import VERSION
|
||||
|
||||
|
||||
# Configure structured logging (JSON in Cloud Run, human-readable locally)
|
||||
# This must happen before any logging calls
|
||||
setup_structured_logging()
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Initialize OpenTelemetry tracing (must happen before app creation)
|
||||
tracing_enabled = setup_tracing(
|
||||
service_name="karaoke-backend",
|
||||
service_version=VERSION,
|
||||
enable_in_dev=False, # Set to True to enable tracing locally
|
||||
)
|
||||
|
||||
|
||||
async def validate_credentials_on_startup():
|
||||
"""Validate OAuth credentials on startup and send alerts if needed."""
|
||||
try:
|
||||
from backend.services.credential_manager import get_credential_manager, CredentialStatus
|
||||
|
||||
manager = get_credential_manager()
|
||||
results = manager.check_all_credentials()
|
||||
|
||||
invalid_services = [
|
||||
result for result in results.values()
|
||||
if result.status in (CredentialStatus.INVALID, CredentialStatus.EXPIRED)
|
||||
]
|
||||
|
||||
if invalid_services:
|
||||
logger.warning(f"Some OAuth credentials need attention:")
|
||||
for result in invalid_services:
|
||||
logger.warning(f" - {result.service}: {result.message}")
|
||||
|
||||
# Try to send Discord alert
|
||||
discord_url = settings.get_secret("discord-alert-webhook") if hasattr(settings, 'get_secret') else None
|
||||
if discord_url:
|
||||
manager.send_credential_alert(invalid_services, discord_url)
|
||||
logger.info("Sent credential alert to Discord")
|
||||
else:
|
||||
logger.info("All OAuth credentials validated successfully")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to validate credentials on startup: {e}")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Lifespan event handler for startup and shutdown."""
|
||||
# Startup
|
||||
logger.info("Starting karaoke generation backend")
|
||||
logger.info(f"Environment: {settings.environment}")
|
||||
logger.info(f"GCS Bucket: {settings.gcs_bucket_name}")
|
||||
logger.info(f"Tracing enabled: {tracing_enabled}")
|
||||
|
||||
# Preload NLP models and resources to avoid cold start delays
|
||||
# See docs/archive/2026-01-08-performance-investigation.md for background
|
||||
|
||||
# 1. SpaCy model (60+ second delay without preload)
|
||||
try:
|
||||
preload_spacy_model("en_core_web_sm")
|
||||
except Exception as e:
|
||||
logger.warning(f"SpaCy preload failed (will load lazily): {e}")
|
||||
|
||||
# 2. NLTK cmudict (50-100+ second delay without preload)
|
||||
try:
|
||||
preload_all_nltk_resources()
|
||||
except Exception as e:
|
||||
logger.warning(f"NLTK preload failed (will load lazily): {e}")
|
||||
|
||||
# 3. Langfuse callback handler (200+ second delay without preload)
|
||||
try:
|
||||
preload_langfuse_handler()
|
||||
except Exception as e:
|
||||
logger.warning(f"Langfuse preload failed (will initialize lazily): {e}")
|
||||
|
||||
# Validate OAuth credentials (non-blocking)
|
||||
try:
|
||||
await validate_credentials_on_startup()
|
||||
except Exception as e:
|
||||
logger.error(f"Credential validation failed: {e}")
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown - wait for any active workers to complete before terminating
|
||||
# This prevents Cloud Run from killing workers mid-processing
|
||||
logger.info("Shutdown requested, checking for active workers...")
|
||||
if worker_registry.has_active_workers():
|
||||
active = worker_registry.get_active_workers()
|
||||
logger.info(f"Active workers found: {active}")
|
||||
logger.info("Waiting for workers to complete (timeout: 600s)...")
|
||||
completed = await worker_registry.wait_for_completion(timeout=600) # 10 min max
|
||||
if not completed:
|
||||
logger.error(
|
||||
"Shutdown timeout - some workers may not have completed cleanly. "
|
||||
f"Remaining workers: {worker_registry.get_active_workers()}"
|
||||
)
|
||||
else:
|
||||
logger.info("No active workers, proceeding with shutdown")
|
||||
|
||||
logger.info("Shutting down karaoke generation backend")
|
||||
|
||||
|
||||
# Create FastAPI app
|
||||
app = FastAPI(
|
||||
title="Karaoke Generator API",
|
||||
description="Backend API for web-based karaoke video generation",
|
||||
version=VERSION,
|
||||
lifespan=lifespan
|
||||
)
|
||||
|
||||
# Instrument FastAPI with OpenTelemetry (adds automatic spans for all requests)
|
||||
if tracing_enabled:
|
||||
instrument_app(app)
|
||||
|
||||
# Configure CORS
|
||||
app.add_middleware(
|
||||
CORSMiddleware,
|
||||
allow_origins=["*"], # Configure this properly for production
|
||||
allow_credentials=True,
|
||||
allow_methods=["*"],
|
||||
allow_headers=["*"],
|
||||
)
|
||||
|
||||
# Add audit logging middleware (captures all requests with request_id for correlation)
|
||||
app.add_middleware(AuditLoggingMiddleware)
|
||||
|
||||
# Add tenant detection middleware (extracts tenant from subdomain/headers)
|
||||
app.add_middleware(TenantMiddleware)
|
||||
|
||||
# Include routers
|
||||
app.include_router(health.router, prefix="/api")
|
||||
app.include_router(jobs.router, prefix="/api")
|
||||
app.include_router(file_upload.router, prefix="/api") # File upload endpoint
|
||||
app.include_router(internal.router, prefix="/api") # Internal worker endpoints
|
||||
app.include_router(review.router, prefix="/api") # Review UI compatibility endpoints
|
||||
app.include_router(auth.router, prefix="/api") # OAuth credential management
|
||||
app.include_router(audio_search.router, prefix="/api") # Audio search (artist+title mode)
|
||||
app.include_router(themes.router, prefix="/api") # Theme selection for styles
|
||||
app.include_router(users.router, prefix="/api") # User auth, credits, and Stripe webhooks
|
||||
app.include_router(admin.router, prefix="/api") # Admin dashboard and management
|
||||
app.include_router(rate_limits.router, prefix="/api") # Rate limits admin management
|
||||
app.include_router(push.router, prefix="/api") # Push notification subscription management
|
||||
app.include_router(tenant.router) # Tenant/white-label configuration (no /api prefix, router has it)
|
||||
|
||||
|
||||
# Exception handler for rate limiting
|
||||
from fastapi import Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from backend.exceptions import RateLimitExceededError
|
||||
|
||||
|
||||
@app.exception_handler(RateLimitExceededError)
|
||||
async def rate_limit_exception_handler(request: Request, exc: RateLimitExceededError):
|
||||
"""Handle rate limit exceeded errors with 429 status."""
|
||||
return JSONResponse(
|
||||
status_code=429,
|
||||
content={
|
||||
"detail": exc.message,
|
||||
"limit_type": exc.limit_type,
|
||||
"current_count": exc.current_count,
|
||||
"limit_value": exc.limit_value,
|
||||
},
|
||||
headers={
|
||||
"Retry-After": str(exc.remaining_seconds),
|
||||
} if exc.remaining_seconds > 0 else None,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/")
|
||||
async def root():
|
||||
"""Root endpoint."""
|
||||
return {
|
||||
"service": "karaoke-gen-backend",
|
||||
"version": VERSION,
|
||||
"status": "running"
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
uvicorn.run(app, host="0.0.0.0", port=8080)
|
||||
|
||||
@@ -1,11 +0,0 @@
|
||||
"""Middleware package for FastAPI application."""
|
||||
|
||||
from backend.middleware.audit_logging import AuditLoggingMiddleware
|
||||
from backend.middleware.tenant import TenantMiddleware, get_tenant_from_request, get_tenant_config_from_request
|
||||
|
||||
__all__ = [
|
||||
"AuditLoggingMiddleware",
|
||||
"TenantMiddleware",
|
||||
"get_tenant_from_request",
|
||||
"get_tenant_config_from_request",
|
||||
]
|
||||
@@ -1,124 +0,0 @@
|
||||
"""
|
||||
Audit logging middleware for tracking all HTTP requests.
|
||||
|
||||
This middleware captures request metadata and logs it to Cloud Logging
|
||||
for audit trail and user activity investigation purposes.
|
||||
"""
|
||||
import logging
|
||||
import time
|
||||
import uuid
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
|
||||
logger = logging.getLogger("audit")
|
||||
|
||||
# Endpoints to exclude from audit logging (high-frequency health checks)
|
||||
EXCLUDED_PATHS = {
|
||||
"/",
|
||||
"/api/health",
|
||||
"/api/health/detailed",
|
||||
"/api/readiness",
|
||||
"/healthz",
|
||||
"/ready",
|
||||
}
|
||||
|
||||
|
||||
class AuditLoggingMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that logs all HTTP requests for audit purposes.
|
||||
|
||||
Captures:
|
||||
- request_id: UUID for correlating with auth logs
|
||||
- method: HTTP method
|
||||
- path: Request path
|
||||
- status_code: Response status
|
||||
- latency_ms: Request duration
|
||||
- client_ip: Client IP (from X-Forwarded-For for proxied requests)
|
||||
- user_agent: Browser/client identifier
|
||||
|
||||
The request_id is stored in request.state and added to response headers,
|
||||
allowing correlation with auth logs that capture user_email.
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Skip excluded paths (health checks)
|
||||
if request.url.path in EXCLUDED_PATHS:
|
||||
return await call_next(request)
|
||||
|
||||
# Generate unique request ID for correlation
|
||||
request_id = str(uuid.uuid4())
|
||||
request.state.request_id = request_id
|
||||
|
||||
# Capture start time
|
||||
start_time = time.time()
|
||||
|
||||
# Extract client info
|
||||
client_ip = self._get_client_ip(request)
|
||||
user_agent = request.headers.get("user-agent", "")
|
||||
|
||||
# Process request
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except Exception:
|
||||
# Log failed requests too (exception() auto-includes stack trace)
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
logger.exception(
|
||||
"request_audit_error",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
raise
|
||||
|
||||
# Calculate latency
|
||||
latency_ms = int((time.time() - start_time) * 1000)
|
||||
|
||||
# Log audit entry (INFO level for successful requests)
|
||||
log_level = logging.WARNING if response.status_code >= 400 else logging.INFO
|
||||
logger.log(
|
||||
log_level,
|
||||
"request_audit",
|
||||
extra={
|
||||
"request_id": request_id,
|
||||
"method": request.method,
|
||||
"path": request.url.path,
|
||||
"query_string": str(request.query_params) if request.query_params else None,
|
||||
"status_code": response.status_code,
|
||||
"latency_ms": latency_ms,
|
||||
"client_ip": client_ip,
|
||||
"user_agent": user_agent[:200] if user_agent else None,
|
||||
"audit_type": "http_request",
|
||||
}
|
||||
)
|
||||
|
||||
# Add request_id to response headers for debugging/correlation
|
||||
response.headers["X-Request-ID"] = request_id
|
||||
|
||||
return response
|
||||
|
||||
def _get_client_ip(self, request: Request) -> str:
|
||||
"""
|
||||
Extract client IP address, handling proxy scenarios.
|
||||
|
||||
Cloud Run and other proxies set X-Forwarded-For header.
|
||||
"""
|
||||
# Check X-Forwarded-For for proxy scenarios (Cloud Run, load balancers)
|
||||
forwarded = request.headers.get("x-forwarded-for", "")
|
||||
if forwarded:
|
||||
# First IP is the original client
|
||||
return forwarded.split(",")[0].strip()
|
||||
|
||||
# Fall back to direct connection
|
||||
if request.client:
|
||||
return request.client.host
|
||||
|
||||
return "unknown"
|
||||
@@ -1,192 +0,0 @@
|
||||
"""
|
||||
Tenant detection middleware for white-label B2B portals.
|
||||
|
||||
This middleware extracts the tenant ID from incoming requests and attaches
|
||||
the tenant configuration to the request state for use by downstream handlers.
|
||||
|
||||
Tenant detection priority:
|
||||
1. X-Tenant-ID header (explicitly set by frontend)
|
||||
2. `tenant` query parameter (for development/testing)
|
||||
3. Host header subdomain detection (production flow)
|
||||
|
||||
The middleware is non-blocking - if no tenant is detected, the request
|
||||
proceeds as a default Nomad Karaoke request (tenant_id = None).
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from typing import Optional
|
||||
|
||||
from starlette.middleware.base import BaseHTTPMiddleware
|
||||
from starlette.requests import Request
|
||||
from starlette.responses import Response
|
||||
|
||||
from backend.services.tenant_service import get_tenant_service
|
||||
from backend.models.tenant import TenantConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Only allow query param tenant override in non-production environments
|
||||
IS_PRODUCTION = os.environ.get("ENV", "").lower() == "production" or \
|
||||
os.environ.get("ENVIRONMENT", "").lower() == "production"
|
||||
|
||||
|
||||
# Known non-tenant subdomains that should be treated as default Nomad Karaoke
|
||||
NON_TENANT_SUBDOMAINS = {"gen", "api", "www", "buy", "admin", "app", "beta"}
|
||||
|
||||
|
||||
class TenantMiddleware(BaseHTTPMiddleware):
|
||||
"""
|
||||
Middleware that detects tenant from request and attaches config to state.
|
||||
|
||||
After this middleware runs, routes can access:
|
||||
- request.state.tenant_id: str or None
|
||||
- request.state.tenant_config: TenantConfig or None
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = getattr(request.state, "tenant_id", None)
|
||||
tenant_config = getattr(request.state, "tenant_config", None)
|
||||
```
|
||||
"""
|
||||
|
||||
async def dispatch(self, request: Request, call_next) -> Response:
|
||||
# Extract tenant ID from request
|
||||
tenant_id = self._extract_tenant_id(request)
|
||||
|
||||
# Load tenant config if tenant detected
|
||||
tenant_config: Optional[TenantConfig] = None
|
||||
if tenant_id:
|
||||
tenant_service = get_tenant_service()
|
||||
tenant_config = tenant_service.get_tenant_config(tenant_id)
|
||||
|
||||
if tenant_config and not tenant_config.is_active:
|
||||
# Tenant exists but is inactive - treat as default
|
||||
logger.warning(f"Inactive tenant requested: {tenant_id}")
|
||||
tenant_id = None
|
||||
tenant_config = None
|
||||
|
||||
# Attach to request state
|
||||
request.state.tenant_id = tenant_id
|
||||
request.state.tenant_config = tenant_config
|
||||
|
||||
# Log tenant detection for debugging
|
||||
if tenant_id:
|
||||
logger.debug(f"Request tenant: {tenant_id} (path: {request.url.path})")
|
||||
|
||||
# Process request
|
||||
response = await call_next(request)
|
||||
|
||||
# Add tenant ID to response headers for debugging
|
||||
if tenant_id:
|
||||
response.headers["X-Tenant-ID"] = tenant_id
|
||||
|
||||
return response
|
||||
|
||||
def _extract_tenant_id(self, request: Request) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from request using priority-based detection.
|
||||
|
||||
Returns:
|
||||
Tenant ID if detected, None for default Nomad Karaoke
|
||||
"""
|
||||
# Priority 1: X-Tenant-ID header (explicitly set by frontend)
|
||||
header_tenant = request.headers.get("X-Tenant-ID")
|
||||
if header_tenant:
|
||||
return header_tenant.lower().strip()
|
||||
|
||||
# Priority 2: Query parameter (for development/testing only)
|
||||
# Disabled in production to prevent tenant spoofing
|
||||
if not IS_PRODUCTION:
|
||||
query_tenant = request.query_params.get("tenant")
|
||||
if query_tenant:
|
||||
return query_tenant.lower().strip()
|
||||
|
||||
# Priority 3: Host header subdomain detection
|
||||
host = request.headers.get("Host", "")
|
||||
return self._extract_tenant_from_host(host)
|
||||
|
||||
def _extract_tenant_from_host(self, host: str) -> Optional[str]:
|
||||
"""
|
||||
Extract tenant ID from Host header subdomain.
|
||||
|
||||
Patterns handled:
|
||||
- vocalstar.nomadkaraoke.com -> vocalstar
|
||||
- vocalstar.gen.nomadkaraoke.com -> vocalstar
|
||||
- localhost:3000 -> None (local dev)
|
||||
- gen.nomadkaraoke.com -> None (main app)
|
||||
- api.nomadkaraoke.com -> None (API)
|
||||
|
||||
Returns:
|
||||
Tenant ID if subdomain matches tenant, None otherwise
|
||||
"""
|
||||
if not host:
|
||||
return None
|
||||
|
||||
# Normalize
|
||||
host_lower = host.lower()
|
||||
|
||||
# Remove port if present
|
||||
if ":" in host_lower:
|
||||
host_lower = host_lower.split(":")[0]
|
||||
|
||||
# Check if this is a nomadkaraoke.com domain
|
||||
if "nomadkaraoke.com" not in host_lower:
|
||||
return None
|
||||
|
||||
# Split into parts
|
||||
parts = host_lower.split(".")
|
||||
|
||||
# Need at least 3 parts: subdomain.nomadkaraoke.com
|
||||
if len(parts) < 3:
|
||||
return None
|
||||
|
||||
# First part is the potential tenant ID
|
||||
potential_tenant = parts[0]
|
||||
|
||||
# Skip known non-tenant subdomains
|
||||
if potential_tenant in NON_TENANT_SUBDOMAINS:
|
||||
return None
|
||||
|
||||
# Verify tenant exists in GCS
|
||||
tenant_service = get_tenant_service()
|
||||
if tenant_service.tenant_exists(potential_tenant):
|
||||
return potential_tenant
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def get_tenant_from_request(request: Request) -> Optional[str]:
|
||||
"""
|
||||
Helper function to get tenant ID from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
tenant_id = get_tenant_from_request(request)
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_id", None)
|
||||
|
||||
|
||||
def get_tenant_config_from_request(request: Request) -> Optional[TenantConfig]:
|
||||
"""
|
||||
Helper function to get tenant config from request state.
|
||||
|
||||
Usage in route handlers:
|
||||
```python
|
||||
from backend.middleware.tenant import get_tenant_config_from_request
|
||||
|
||||
@router.get("/something")
|
||||
async def handler(request: Request):
|
||||
config = get_tenant_config_from_request(request)
|
||||
if config and not config.features.audio_search:
|
||||
raise HTTPException(403, "Audio search not available")
|
||||
```
|
||||
"""
|
||||
return getattr(request.state, "tenant_config", None)
|
||||
Whitespace-only changes.
@@ -1,546 +0,0 @@
|
||||
"""
|
||||
Job data models for karaoke generation.
|
||||
|
||||
This module defines the complete state machine for karaoke generation jobs,
|
||||
mirroring the 8-stage CLI workflow with human-in-the-loop interaction points.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, Field, validator
|
||||
|
||||
from karaoke_gen.utils import normalize_text
|
||||
|
||||
|
||||
class JobStatus(str, Enum):
|
||||
"""
|
||||
Job status enumeration - Complete state machine.
|
||||
|
||||
The workflow has 7 main stages with 2 human interaction points:
|
||||
1. Input & Setup (may include audio source selection)
|
||||
2. Parallel Processing (audio + lyrics)
|
||||
3. Title/End Screen Generation + Backing Vocals Analysis
|
||||
4. Countdown Padding Synchronization
|
||||
5. Combined Human Review (BLOCKING) - lyrics review + instrumental selection
|
||||
6. Video Finalization
|
||||
7. Distribution
|
||||
|
||||
Note: As of 2026-01, lyrics review and instrumental selection are combined
|
||||
into a single human review step. The user selects their instrumental
|
||||
during the lyrics review phase, before the video is rendered.
|
||||
"""
|
||||
# Initial states
|
||||
PENDING = "pending" # Job created, queued for processing
|
||||
|
||||
# Audio search states (for artist+title search mode)
|
||||
SEARCHING_AUDIO = "searching_audio" # Searching for audio sources via flacfetch
|
||||
AWAITING_AUDIO_SELECTION = "awaiting_audio_selection" # ⚠️ WAITING FOR USER - select audio source
|
||||
DOWNLOADING_AUDIO = "downloading_audio" # Downloading selected audio from source
|
||||
|
||||
DOWNLOADING = "downloading" # Downloading from URL or processing upload
|
||||
|
||||
# Stage 2a: Audio separation (parallel track 1)
|
||||
SEPARATING_STAGE1 = "separating_stage1" # Clean instrumental separation (Modal API)
|
||||
SEPARATING_STAGE2 = "separating_stage2" # Backing vocals separation (Modal API)
|
||||
AUDIO_COMPLETE = "audio_complete" # All audio stems ready
|
||||
|
||||
# Stage 2b: Lyrics processing (parallel track 2)
|
||||
TRANSCRIBING = "transcribing" # AudioShake API transcription
|
||||
CORRECTING = "correcting" # Automatic lyrics correction
|
||||
LYRICS_COMPLETE = "lyrics_complete" # Corrections JSON ready
|
||||
|
||||
# Stage 3: Title/End screens + Backing Vocals Analysis
|
||||
GENERATING_SCREENS = "generating_screens" # Creating title and end screen videos
|
||||
|
||||
# Stage 4: Countdown padding (automatic)
|
||||
APPLYING_PADDING = "applying_padding" # Synchronizing countdown padding
|
||||
|
||||
# Stage 5: Combined Human Review (BLOCKING) - lyrics + instrumental selection
|
||||
AWAITING_REVIEW = "awaiting_review" # ⚠️ WAITING FOR USER - combined review needed
|
||||
IN_REVIEW = "in_review" # User is actively reviewing
|
||||
REVIEW_COMPLETE = "review_complete" # User submitted lyrics + instrumental selection
|
||||
|
||||
# Stage 5.5: Render video with corrected lyrics (post-review)
|
||||
RENDERING_VIDEO = "rendering_video" # Using OutputGenerator to create with_vocals.mkv
|
||||
|
||||
# Stage 6: Instrumental already selected (during combined review)
|
||||
# Note: AWAITING_INSTRUMENTAL_SELECTION kept for DB compatibility with historical jobs only
|
||||
AWAITING_INSTRUMENTAL_SELECTION = "awaiting_instrumental_selection" # LEGACY - no longer used
|
||||
INSTRUMENTAL_SELECTED = "instrumental_selected" # Instrumental was selected during combined review
|
||||
|
||||
# Stage 7: Video generation and finalization
|
||||
GENERATING_VIDEO = "generating_video" # Creating initial karaoke video
|
||||
ENCODING = "encoding" # Multi-format video encoding (Cloud Build)
|
||||
PACKAGING = "packaging" # CDG/TXT ZIP generation
|
||||
|
||||
# Stage 8: Distribution (optional)
|
||||
UPLOADING = "uploading" # YouTube/Dropbox upload
|
||||
NOTIFYING = "notifying" # Discord/Email notifications
|
||||
|
||||
# Terminal states
|
||||
COMPLETE = "complete" # All processing finished successfully
|
||||
PREP_COMPLETE = "prep_complete" # Prep-only job completed (stops after review)
|
||||
FAILED = "failed" # Unrecoverable error occurred
|
||||
CANCELLED = "cancelled" # User cancelled the job
|
||||
|
||||
# Legacy compatibility (will be removed)
|
||||
QUEUED = "queued" # Deprecated: use PENDING
|
||||
PROCESSING = "processing" # Deprecated: use specific states
|
||||
READY_FOR_FINALIZATION = "ready_for_finalization" # Deprecated
|
||||
FINALIZING = "finalizing" # Deprecated: use ENCODING/PACKAGING
|
||||
ERROR = "error" # Deprecated: use FAILED
|
||||
|
||||
|
||||
# Valid state transitions
|
||||
STATE_TRANSITIONS = {
|
||||
# PENDING can go to DOWNLOADING (file upload) or SEARCHING_AUDIO (artist+title search)
|
||||
JobStatus.PENDING: [JobStatus.DOWNLOADING, JobStatus.SEARCHING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
|
||||
# Audio search flow (for artist+title search mode)
|
||||
JobStatus.SEARCHING_AUDIO: [JobStatus.AWAITING_AUDIO_SELECTION, JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED],
|
||||
JobStatus.AWAITING_AUDIO_SELECTION: [JobStatus.DOWNLOADING_AUDIO, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.DOWNLOADING_AUDIO: [JobStatus.DOWNLOADING, JobStatus.FAILED],
|
||||
|
||||
# DOWNLOADING allows parallel processing (audio + lyrics) and then screens when both complete
|
||||
JobStatus.DOWNLOADING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING, JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Audio separation flow
|
||||
JobStatus.SEPARATING_STAGE1: [JobStatus.SEPARATING_STAGE2, JobStatus.FAILED],
|
||||
JobStatus.SEPARATING_STAGE2: [JobStatus.AUDIO_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.AUDIO_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Lyrics flow
|
||||
JobStatus.TRANSCRIBING: [JobStatus.CORRECTING, JobStatus.FAILED],
|
||||
JobStatus.CORRECTING: [JobStatus.LYRICS_COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.LYRICS_COMPLETE: [JobStatus.GENERATING_SCREENS, JobStatus.FAILED],
|
||||
|
||||
# Post-parallel processing (screens + backing vocals analysis)
|
||||
JobStatus.GENERATING_SCREENS: [JobStatus.APPLYING_PADDING, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.APPLYING_PADDING: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
|
||||
# Combined human review flow (lyrics + instrumental selection)
|
||||
# AWAITING_REVIEW can go directly to REVIEW_COMPLETE (quick review) or to IN_REVIEW (editing)
|
||||
JobStatus.AWAITING_REVIEW: [JobStatus.IN_REVIEW, JobStatus.REVIEW_COMPLETE, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.IN_REVIEW: [JobStatus.REVIEW_COMPLETE, JobStatus.AWAITING_REVIEW, JobStatus.FAILED],
|
||||
JobStatus.REVIEW_COMPLETE: [JobStatus.RENDERING_VIDEO, JobStatus.PREP_COMPLETE, JobStatus.FAILED], # PREP_COMPLETE for prep-only jobs
|
||||
|
||||
# Video rendering (post-review) - instrumental was already selected during combined review
|
||||
JobStatus.RENDERING_VIDEO: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.PREP_COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# AWAITING_INSTRUMENTAL_SELECTION is LEGACY - kept for historical jobs in DB
|
||||
# New jobs never enter this state; they go directly to INSTRUMENTAL_SELECTED after render
|
||||
JobStatus.AWAITING_INSTRUMENTAL_SELECTION: [JobStatus.INSTRUMENTAL_SELECTED, JobStatus.FAILED, JobStatus.CANCELLED],
|
||||
JobStatus.INSTRUMENTAL_SELECTED: [JobStatus.GENERATING_VIDEO, JobStatus.FAILED],
|
||||
|
||||
# Video generation flow
|
||||
JobStatus.GENERATING_VIDEO: [JobStatus.ENCODING, JobStatus.FAILED],
|
||||
JobStatus.ENCODING: [JobStatus.PACKAGING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.PACKAGING: [JobStatus.UPLOADING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Distribution flow
|
||||
JobStatus.UPLOADING: [JobStatus.NOTIFYING, JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
JobStatus.NOTIFYING: [JobStatus.COMPLETE, JobStatus.FAILED],
|
||||
|
||||
# Terminal states - COMPLETE, PREP_COMPLETE have no transitions
|
||||
# FAILED and CANCELLED allow retry transitions to resume from checkpoints
|
||||
# PREP_COMPLETE allows continuation from combined review
|
||||
JobStatus.COMPLETE: [],
|
||||
JobStatus.PREP_COMPLETE: [JobStatus.AWAITING_REVIEW, JobStatus.FAILED], # Continue from combined review
|
||||
JobStatus.FAILED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
JobStatus.AWAITING_REVIEW, # Retry from combined review
|
||||
],
|
||||
JobStatus.CANCELLED: [
|
||||
JobStatus.DOWNLOADING, # Retry from beginning (if input audio exists)
|
||||
JobStatus.INSTRUMENTAL_SELECTED, # Retry from video generation
|
||||
JobStatus.REVIEW_COMPLETE, # Retry from render stage
|
||||
JobStatus.LYRICS_COMPLETE, # Retry from screens generation
|
||||
JobStatus.AWAITING_REVIEW, # Retry from combined review
|
||||
],
|
||||
|
||||
# Legacy states (for backward compatibility)
|
||||
JobStatus.QUEUED: [JobStatus.PENDING],
|
||||
JobStatus.PROCESSING: [JobStatus.SEPARATING_STAGE1, JobStatus.TRANSCRIBING],
|
||||
JobStatus.READY_FOR_FINALIZATION: [JobStatus.GENERATING_VIDEO],
|
||||
JobStatus.FINALIZING: [JobStatus.ENCODING],
|
||||
JobStatus.ERROR: [JobStatus.FAILED],
|
||||
}
|
||||
|
||||
|
||||
class TimelineEvent(BaseModel):
|
||||
"""Timeline event for job progress tracking."""
|
||||
status: str
|
||||
timestamp: str
|
||||
progress: Optional[int] = None
|
||||
message: Optional[str] = None
|
||||
|
||||
|
||||
class LogEntry(BaseModel):
|
||||
"""Worker log entry for debugging and monitoring."""
|
||||
timestamp: str
|
||||
level: str # DEBUG, INFO, WARNING, ERROR
|
||||
worker: str # audio, lyrics, screens, video, render
|
||||
message: str
|
||||
|
||||
|
||||
class Job(BaseModel):
|
||||
"""
|
||||
Complete job data model.
|
||||
|
||||
Tracks the full lifecycle of a karaoke generation job from submission
|
||||
through all 8 stages to completion or failure.
|
||||
"""
|
||||
job_id: str
|
||||
status: JobStatus
|
||||
progress: int = 0 # 0-100 percentage for UI display
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
# Input data
|
||||
url: Optional[str] = None # YouTube URL (if provided)
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
input_media_gcs_path: Optional[str] = None # GCS path to uploaded file
|
||||
|
||||
# User preferences
|
||||
enable_cdg: bool = False # Generate CDG+MP3 package (requires style config)
|
||||
enable_txt: bool = False # Generate TXT+MP3 package (requires style config)
|
||||
enable_youtube_upload: bool = False # Upload to YouTube
|
||||
youtube_description: Optional[str] = None # YouTube video description
|
||||
webhook_url: Optional[str] = None # Webhook for notifications
|
||||
user_email: Optional[str] = None # Email for notifications
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None # Tenant ID for white-label portal scoping
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (uploaded files - used when theme_id is not set)
|
||||
style_params_gcs_path: Optional[str] = None # GCS path to style_params.json
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
GCS paths to style asset files:
|
||||
{
|
||||
"intro_background": "gs://bucket/jobs/{job_id}/style/intro_bg.png",
|
||||
"karaoke_background": "gs://bucket/jobs/{job_id}/style/karaoke_bg.png",
|
||||
"end_background": "gs://bucket/jobs/{job_id}/style/end_bg.png",
|
||||
"font": "gs://bucket/jobs/{job_id}/style/font.ttf",
|
||||
"cdg_instrumental_background": "gs://bucket/jobs/{job_id}/style/cdg_instr.png",
|
||||
"cdg_title_background": "gs://bucket/jobs/{job_id}/style/cdg_title.png",
|
||||
"cdg_outro_background": "gs://bucket/jobs/{job_id}/style/cdg_outro.png"
|
||||
}
|
||||
"""
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None # Brand code prefix (e.g., "NOMAD")
|
||||
discord_webhook_url: Optional[str] = None # Discord notification webhook
|
||||
youtube_description_template: Optional[str] = None # YouTube description template text
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output (e.g., "/Karaoke/Tracks-Organized")
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy distribution configuration (rclone - for local CLI backward compat)
|
||||
organised_dir_rclone_root: Optional[str] = None # Deprecated: use dropbox_path instead
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation (requires uploaded prep outputs)
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Review authentication (Batch 7)
|
||||
review_token: Optional[str] = None # Job-scoped token for lyrics review UI access (generated when entering AWAITING_REVIEW)
|
||||
review_token_expires_at: Optional[datetime] = None # Token expiry time (optional, for extra security)
|
||||
instrumental_token: Optional[str] = None # Job-scoped token for instrumental review UI access (generated when entering AWAITING_INSTRUMENTAL_SELECTION)
|
||||
instrumental_token_expires_at: Optional[datetime] = None # Token expiry time
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Output deletion tracking (for admin cleanup without deleting job)
|
||||
outputs_deleted_at: Optional[datetime] = None # Timestamp when outputs were deleted by admin
|
||||
outputs_deleted_by: Optional[str] = None # Admin email who deleted outputs
|
||||
|
||||
# Processing state
|
||||
track_output_dir: Optional[str] = None # Local output directory (temp)
|
||||
audio_hash: Optional[str] = None # Hash for deduplication
|
||||
|
||||
# State-specific data (JSON field for stage-specific metadata)
|
||||
state_data: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Stage-specific metadata. Examples:
|
||||
- audio_complete: {"stems": {"clean": "gs://...", "backing": "gs://..."}}
|
||||
- lyrics_complete: {"corrections_url": "gs://...", "audio_url": "gs://..."}
|
||||
- review_complete: {"corrected_lyrics": {...}}
|
||||
- instrumental_selected: {"selection": "clean" | "with_backing"}
|
||||
- encoding: {"build_id": "...", "progress": 45}
|
||||
"""
|
||||
|
||||
# Timeline tracking
|
||||
timeline: List[TimelineEvent] = Field(default_factory=list)
|
||||
|
||||
# Worker logs for debugging (limited to last N entries to avoid document size issues)
|
||||
worker_logs: List[LogEntry] = Field(default_factory=list)
|
||||
|
||||
# File URLs (GCS storage)
|
||||
file_urls: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
File storage URLs. Structure:
|
||||
{
|
||||
"input": "gs://bucket/jobs/{job_id}/input.flac",
|
||||
"stems": {
|
||||
"instrumental_clean": "gs://...",
|
||||
"instrumental_with_backing": "gs://...",
|
||||
"vocals": "gs://...",
|
||||
"backing_vocals": "gs://...",
|
||||
"lead_vocals": "gs://...",
|
||||
"bass": "gs://...",
|
||||
"drums": "gs://...",
|
||||
"guitar": "gs://...",
|
||||
"piano": "gs://...",
|
||||
"other": "gs://..."
|
||||
},
|
||||
"lyrics": {
|
||||
"corrections": "gs://bucket/jobs/{job_id}/lyrics/corrections.json",
|
||||
"audio": "gs://bucket/jobs/{job_id}/lyrics/audio.flac",
|
||||
"lrc": "gs://...",
|
||||
"ass": "gs://..."
|
||||
},
|
||||
"screens": {
|
||||
"title": "gs://bucket/jobs/{job_id}/screens/title.mov",
|
||||
"end": "gs://bucket/jobs/{job_id}/screens/end.mov"
|
||||
},
|
||||
"videos": {
|
||||
"with_vocals": "gs://bucket/jobs/{job_id}/videos/with_vocals.mkv"
|
||||
},
|
||||
"finals": {
|
||||
"lossless_4k_mp4": "gs://...",
|
||||
"lossless_4k_mkv": "gs://...",
|
||||
"lossy_4k_mp4": "gs://...",
|
||||
"lossy_720p_mp4": "gs://..."
|
||||
},
|
||||
"packages": {
|
||||
"cdg_zip": "gs://...",
|
||||
"txt_zip": "gs://..."
|
||||
},
|
||||
"youtube": {
|
||||
"url": "https://youtube.com/watch?v=...",
|
||||
"video_id": "..."
|
||||
}
|
||||
}
|
||||
"""
|
||||
|
||||
# Results (for backward compatibility, will be deprecated)
|
||||
output_files: Dict[str, str] = Field(default_factory=dict)
|
||||
download_urls: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Error handling
|
||||
error_message: Optional[str] = None
|
||||
error_details: Optional[Dict[str, Any]] = None # Structured error information
|
||||
retry_count: int = 0 # Number of retry attempts
|
||||
|
||||
# Worker tracking
|
||||
worker_ids: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
IDs of background workers/jobs:
|
||||
{
|
||||
"audio_worker": "cloud-run-request-id",
|
||||
"lyrics_worker": "cloud-run-request-id",
|
||||
"video_encoder": "cloud-build-id",
|
||||
"distribution": "cloud-run-request-id"
|
||||
}
|
||||
"""
|
||||
|
||||
# Request metadata (captured at job creation for tracking and filtering)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Metadata captured from the original API request.
|
||||
Used for tracking, filtering, and operational management.
|
||||
|
||||
Standard fields:
|
||||
{
|
||||
"client_ip": "192.168.1.1", # IP address of the client
|
||||
"user_agent": "karaoke-gen-remote/0.71.0", # User-Agent header
|
||||
"environment": "test", # From X-Environment header (test/production/development)
|
||||
"client_id": "cli-user-123", # From X-Client-ID header (customer/user identifier)
|
||||
"server_version": "0.71.0", # Server version at job creation
|
||||
"created_from": "upload", # "upload" (file) or "url" (YouTube URL)
|
||||
"custom_headers": { # All X-* headers for extensibility
|
||||
"X-Environment": "test",
|
||||
"X-Client-ID": "cli-user-123",
|
||||
"X-Request-ID": "abc-123"
|
||||
}
|
||||
}
|
||||
|
||||
Use cases:
|
||||
- Filter test vs production jobs
|
||||
- Track jobs by customer/client
|
||||
- Debug issues with specific clients
|
||||
- Bulk cleanup of test jobs
|
||||
"""
|
||||
|
||||
# Note: Status transition validation is handled by JobManager.validate_state_transition()
|
||||
# which is called before status updates. The Job model does not validate transitions
|
||||
# because Firestore updates happen directly without reconstructing the model.
|
||||
|
||||
class Config:
|
||||
use_enum_values = True
|
||||
|
||||
|
||||
class JobCreate(BaseModel):
|
||||
"""
|
||||
Job creation request.
|
||||
|
||||
Either `url` OR file upload is required (file upload handled separately).
|
||||
Artist and title are optional - will be auto-detected from YouTube if not provided.
|
||||
"""
|
||||
url: Optional[str] = None
|
||||
artist: Optional[str] = None
|
||||
title: Optional[str] = None
|
||||
filename: Optional[str] = None # Original uploaded filename
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: bool = False
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
non_interactive: bool = False # Skip interactive steps (lyrics review, instrumental selection)
|
||||
|
||||
# Theme configuration (pre-made themes from GCS)
|
||||
theme_id: Optional[str] = None # Theme identifier (e.g., "nomad", "default")
|
||||
color_overrides: Dict[str, str] = Field(default_factory=dict)
|
||||
"""
|
||||
User color overrides applied on top of theme. Keys:
|
||||
- artist_color: Hex color for artist name (#RRGGBB)
|
||||
- title_color: Hex color for song title
|
||||
- sung_lyrics_color: Hex color for highlighted lyrics
|
||||
- unsung_lyrics_color: Hex color for unhighlighted lyrics
|
||||
"""
|
||||
|
||||
# Style configuration (will be populated after file upload, or from theme)
|
||||
style_params_gcs_path: Optional[str] = None
|
||||
style_assets: Dict[str, str] = Field(default_factory=dict)
|
||||
|
||||
# Finalisation configuration
|
||||
brand_prefix: Optional[str] = None
|
||||
discord_webhook_url: Optional[str] = None
|
||||
youtube_description_template: Optional[str] = None
|
||||
|
||||
# Distribution configuration (native API - for remote CLI)
|
||||
dropbox_path: Optional[str] = None # Dropbox folder path for organized output
|
||||
gdrive_folder_id: Optional[str] = None # Google Drive folder ID for public share uploads
|
||||
|
||||
# Legacy (rclone - deprecated, use dropbox_path instead)
|
||||
organised_dir_rclone_root: Optional[str] = None
|
||||
|
||||
# Lyrics configuration (overrides for search/transcription)
|
||||
lyrics_artist: Optional[str] = None # Override artist name for lyrics search
|
||||
lyrics_title: Optional[str] = None # Override title for lyrics search
|
||||
lyrics_file_gcs_path: Optional[str] = None # GCS path to user-provided lyrics file
|
||||
subtitle_offset_ms: int = 0 # Offset for subtitle timing (positive = delay)
|
||||
|
||||
# Audio separation model configuration
|
||||
clean_instrumental_model: Optional[str] = None # Model for clean instrumental separation
|
||||
backing_vocals_models: Optional[List[str]] = None # Models for backing vocals separation
|
||||
other_stems_models: Optional[List[str]] = None # Models for other stems (bass, drums, etc.)
|
||||
|
||||
# Existing instrumental configuration (Batch 3)
|
||||
existing_instrumental_gcs_path: Optional[str] = None # GCS path to user-provided instrumental file
|
||||
|
||||
# Audio search configuration (Batch 5 - artist+title search mode)
|
||||
audio_search_artist: Optional[str] = None # Artist name used for audio search
|
||||
audio_search_title: Optional[str] = None # Title used for audio search
|
||||
auto_download: bool = False # Auto-select best audio source (skip selection)
|
||||
|
||||
# Two-phase workflow configuration (Batch 6)
|
||||
prep_only: bool = False # Stop after review, don't run finalisation
|
||||
finalise_only: bool = False # Skip prep, run only finalisation
|
||||
keep_brand_code: Optional[str] = None # Preserve existing brand code instead of generating new one
|
||||
|
||||
# Made-for-you order tracking
|
||||
made_for_you: bool = False # Flag indicating this is a made-for-you customer order
|
||||
customer_email: Optional[str] = None # Customer email for final delivery (job owned by admin during processing)
|
||||
customer_notes: Optional[str] = None # Notes provided by customer with their order
|
||||
|
||||
# Request metadata (set by API endpoint from request headers)
|
||||
request_metadata: Dict[str, Any] = Field(default_factory=dict)
|
||||
"""
|
||||
Populated by the API endpoint with request context:
|
||||
- client_ip: Client IP address
|
||||
- user_agent: User-Agent header
|
||||
- environment: From X-Environment header (test/production/development)
|
||||
- client_id: From X-Client-ID header
|
||||
- server_version: Current server version
|
||||
- custom_headers: All X-* headers
|
||||
"""
|
||||
|
||||
# Tenant scoping for white-label portals
|
||||
tenant_id: Optional[str] = None # Tenant ID for job scoping
|
||||
|
||||
@validator('url')
|
||||
def validate_url(cls, v):
|
||||
"""Validate URL is not empty."""
|
||||
if v is not None and isinstance(v, str) and not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@validator('artist', 'title')
|
||||
def normalize_artist_title(cls, v):
|
||||
"""Normalize artist/title text to standardize Unicode characters.
|
||||
|
||||
This ensures consistent data storage by converting:
|
||||
- Curly quotes -> straight quotes
|
||||
- Various dashes -> hyphen
|
||||
- Unusual whitespace -> regular space
|
||||
"""
|
||||
if v is not None and isinstance(v, str):
|
||||
if not v.strip():
|
||||
raise ValueError("Field cannot be empty string")
|
||||
# normalize_text handles stripping and Unicode normalization
|
||||
return normalize_text(v)
|
||||
return v
|
||||
|
||||
|
||||
class JobResponse(BaseModel):
|
||||
"""Job response model."""
|
||||
status: str
|
||||
job_id: str
|
||||
message: str
|
||||
|
||||
@@ -1,143 +0,0 @@
|
||||
"""
|
||||
API request models for karaoke generation endpoints.
|
||||
"""
|
||||
from typing import Optional, Dict, Any, List
|
||||
from pydantic import BaseModel, HttpUrl, validator
|
||||
|
||||
|
||||
class URLSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from a URL (YouTube, etc.)."""
|
||||
url: HttpUrl
|
||||
artist: Optional[str] = None # Auto-detected if not provided
|
||||
title: Optional[str] = None # Auto-detected if not provided
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class UploadSubmissionRequest(BaseModel):
|
||||
"""Request to submit a job from an uploaded file."""
|
||||
artist: str
|
||||
title: str
|
||||
|
||||
# Optional preferences
|
||||
enable_cdg: bool = False # Requires style config
|
||||
enable_txt: bool = False # Requires style config
|
||||
enable_youtube_upload: Optional[bool] = None # None = use server default
|
||||
youtube_description: Optional[str] = None
|
||||
webhook_url: Optional[str] = None
|
||||
user_email: Optional[str] = None
|
||||
|
||||
|
||||
class CorrectionsSubmission(BaseModel):
|
||||
"""
|
||||
Request to submit corrected lyrics after human review.
|
||||
|
||||
This is the critical human-in-the-loop interaction point.
|
||||
The corrections data comes from the lyrics-transcriber review interface.
|
||||
"""
|
||||
corrections: Dict[str, Any] # Full corrections JSON from frontend
|
||||
user_notes: Optional[str] = None # Optional notes from reviewer
|
||||
|
||||
@validator('corrections')
|
||||
def validate_corrections_format(cls, v):
|
||||
"""Validate corrections has required fields."""
|
||||
required_fields = ['lines', 'metadata']
|
||||
for field in required_fields:
|
||||
if field not in v:
|
||||
raise ValueError(f"Corrections must include '{field}' field")
|
||||
return v
|
||||
|
||||
|
||||
class InstrumentalSelection(BaseModel):
|
||||
"""
|
||||
Request to select instrumental audio option.
|
||||
|
||||
This is the second critical human-in-the-loop interaction point.
|
||||
User chooses between clean instrumental, instrumental with backing vocals,
|
||||
or a custom instrumental (created via create-custom-instrumental endpoint).
|
||||
"""
|
||||
selection: str # "clean", "with_backing", or "custom"
|
||||
|
||||
@validator('selection')
|
||||
def validate_selection(cls, v):
|
||||
"""Validate selection is a valid option."""
|
||||
valid_options = ['clean', 'with_backing', 'custom']
|
||||
if v not in valid_options:
|
||||
raise ValueError(f"Selection must be one of: {valid_options}")
|
||||
return v
|
||||
|
||||
|
||||
class CompleteReviewRequest(BaseModel):
|
||||
"""
|
||||
Request to complete the review with optional instrumental selection.
|
||||
|
||||
For the combined review flow, this includes the instrumental selection.
|
||||
For legacy flows, the instrumental_selection can be omitted.
|
||||
"""
|
||||
instrumental_selection: Optional[str] = None # "clean", "with_backing", or "custom"
|
||||
|
||||
@validator('instrumental_selection')
|
||||
def validate_instrumental_selection(cls, v):
|
||||
"""Validate instrumental selection if provided."""
|
||||
if v is None:
|
||||
return v
|
||||
valid_options = ['clean', 'with_backing', 'custom']
|
||||
if v not in valid_options:
|
||||
raise ValueError(f"instrumental_selection must be one of: {valid_options}")
|
||||
return v
|
||||
|
||||
|
||||
class MuteRegionRequest(BaseModel):
|
||||
"""A region to mute in the backing vocals."""
|
||||
start_seconds: float
|
||||
end_seconds: float
|
||||
|
||||
@validator('start_seconds')
|
||||
def validate_start(cls, v):
|
||||
if v < 0:
|
||||
raise ValueError("start_seconds must be non-negative")
|
||||
return v
|
||||
|
||||
@validator('end_seconds')
|
||||
def validate_end(cls, v, values):
|
||||
if 'start_seconds' in values and v <= values['start_seconds']:
|
||||
raise ValueError("end_seconds must be greater than start_seconds")
|
||||
return v
|
||||
|
||||
|
||||
class CreateCustomInstrumentalRequest(BaseModel):
|
||||
"""
|
||||
Request to create a custom instrumental with muted backing vocal regions.
|
||||
|
||||
The mute_regions specify time ranges in the backing vocals track that
|
||||
should be silenced before mixing with the clean instrumental.
|
||||
"""
|
||||
mute_regions: List[MuteRegionRequest]
|
||||
|
||||
@validator('mute_regions')
|
||||
def validate_regions(cls, v):
|
||||
if not v:
|
||||
raise ValueError("At least one mute region is required")
|
||||
return v
|
||||
|
||||
|
||||
class StartReviewRequest(BaseModel):
|
||||
"""Request to mark job as in-review (user opened interface)."""
|
||||
pass # No body needed, just triggers state transition
|
||||
|
||||
|
||||
class CancelJobRequest(BaseModel):
|
||||
"""Request to cancel a job."""
|
||||
reason: Optional[str] = None
|
||||
|
||||
|
||||
class RetryJobRequest(BaseModel):
|
||||
"""Request to retry a failed job."""
|
||||
from_stage: Optional[str] = None # Optional: restart from specific stage
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
"""
|
||||
Tenant data models for white-label B2B portals.
|
||||
|
||||
Tenants are B2B customers who get their own branded portal with custom
|
||||
configuration. Each tenant has:
|
||||
- Custom branding (logo, colors, site title)
|
||||
- Feature flags (enable/disable audio search, distribution, etc.)
|
||||
- Default settings (theme, distribution mode)
|
||||
- Auth configuration (allowed email domains, fixed tokens)
|
||||
|
||||
Tenant configs are stored in GCS at tenants/{tenant_id}/config.json
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class TenantBranding(BaseModel):
|
||||
"""Branding configuration for a tenant's portal."""
|
||||
|
||||
logo_url: Optional[str] = Field(
|
||||
None, description="URL to tenant's logo image (PNG with transparency preferred)"
|
||||
)
|
||||
logo_height: int = Field(40, description="Logo height in pixels for header display")
|
||||
primary_color: str = Field(
|
||||
"#ff5bb8", description="Primary brand color (hex format)"
|
||||
)
|
||||
secondary_color: str = Field(
|
||||
"#8b5cf6", description="Secondary brand color (hex format)"
|
||||
)
|
||||
accent_color: Optional[str] = Field(
|
||||
None, description="Accent color for highlights (hex format)"
|
||||
)
|
||||
background_color: Optional[str] = Field(
|
||||
None, description="Custom background color (defaults to dark theme)"
|
||||
)
|
||||
favicon_url: Optional[str] = Field(None, description="URL to custom favicon")
|
||||
site_title: str = Field(
|
||||
"Karaoke Generator", description="Browser tab title"
|
||||
)
|
||||
tagline: Optional[str] = Field(
|
||||
None, description="Optional tagline shown below logo"
|
||||
)
|
||||
|
||||
|
||||
class TenantFeatures(BaseModel):
|
||||
"""Feature flags controlling what's available in the tenant's portal."""
|
||||
|
||||
# Audio input methods
|
||||
audio_search: bool = Field(
|
||||
True, description="Enable audio search (flacfetch integration)"
|
||||
)
|
||||
file_upload: bool = Field(True, description="Enable direct file upload")
|
||||
youtube_url: bool = Field(True, description="Enable YouTube URL input")
|
||||
|
||||
# Distribution options
|
||||
youtube_upload: bool = Field(True, description="Enable YouTube upload option")
|
||||
dropbox_upload: bool = Field(True, description="Enable Dropbox upload option")
|
||||
gdrive_upload: bool = Field(True, description="Enable Google Drive upload option")
|
||||
|
||||
# Customization options
|
||||
theme_selection: bool = Field(
|
||||
True, description="Allow user to select theme (false = use default)"
|
||||
)
|
||||
color_overrides: bool = Field(
|
||||
True, description="Allow user to customize colors"
|
||||
)
|
||||
|
||||
# Output formats
|
||||
enable_cdg: bool = Field(True, description="Generate CDG format output")
|
||||
enable_4k: bool = Field(True, description="Generate 4K video output")
|
||||
|
||||
# Advanced features
|
||||
admin_access: bool = Field(
|
||||
False, description="Allow access to admin dashboard"
|
||||
)
|
||||
|
||||
|
||||
class TenantDefaults(BaseModel):
|
||||
"""Default settings applied to all jobs for this tenant."""
|
||||
|
||||
theme_id: Optional[str] = Field(
|
||||
None, description="Default theme ID (required if theme_selection=false)"
|
||||
)
|
||||
locked_theme: Optional[str] = Field(
|
||||
None,
|
||||
description="If set, users cannot change theme - always uses this theme ID"
|
||||
)
|
||||
distribution_mode: str = Field(
|
||||
"all",
|
||||
description="Distribution mode: 'all', 'download_only', or 'cloud_only'"
|
||||
)
|
||||
brand_prefix: Optional[str] = Field(
|
||||
None, description="Prefix for output filenames (e.g., 'VSTAR')"
|
||||
)
|
||||
youtube_description_template: Optional[str] = Field(
|
||||
None, description="Default YouTube description template"
|
||||
)
|
||||
|
||||
|
||||
class TenantAuth(BaseModel):
|
||||
"""Authentication configuration for a tenant."""
|
||||
|
||||
allowed_email_domains: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="Email domains allowed for magic link auth (e.g., ['vocal-star.com'])"
|
||||
)
|
||||
require_email_domain: bool = Field(
|
||||
True,
|
||||
description="If true, only allowed domains can sign up. If false, domains get auto-approved."
|
||||
)
|
||||
fixed_token_ids: List[str] = Field(
|
||||
default_factory=list,
|
||||
description="IDs of fixed API tokens for this tenant (tokens stored in auth_tokens)"
|
||||
)
|
||||
sender_email: Optional[str] = Field(
|
||||
None,
|
||||
description="Email sender address for this tenant (e.g., 'vocalstar@nomadkaraoke.com')"
|
||||
)
|
||||
|
||||
|
||||
class TenantConfig(BaseModel):
|
||||
"""Complete configuration for a white-label tenant."""
|
||||
|
||||
id: str = Field(..., description="Unique tenant identifier (e.g., 'vocalstar')")
|
||||
name: str = Field(..., description="Display name (e.g., 'Vocal Star')")
|
||||
subdomain: str = Field(
|
||||
..., description="Full subdomain (e.g., 'vocalstar.nomadkaraoke.com')"
|
||||
)
|
||||
is_active: bool = Field(True, description="Whether tenant portal is active")
|
||||
|
||||
branding: TenantBranding = Field(default_factory=TenantBranding)
|
||||
features: TenantFeatures = Field(default_factory=TenantFeatures)
|
||||
defaults: TenantDefaults = Field(default_factory=TenantDefaults)
|
||||
auth: TenantAuth = Field(default_factory=TenantAuth)
|
||||
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
def get_sender_email(self) -> str:
|
||||
"""Get the email sender address for this tenant."""
|
||||
if self.auth.sender_email:
|
||||
return self.auth.sender_email
|
||||
# Default pattern: {tenant_id}@nomadkaraoke.com
|
||||
return f"{self.id}@nomadkaraoke.com"
|
||||
|
||||
def is_email_allowed(self, email: str) -> bool:
|
||||
"""Check if an email address is allowed for this tenant."""
|
||||
if not self.auth.allowed_email_domains:
|
||||
# No domain restrictions
|
||||
return True
|
||||
|
||||
email_lower = email.lower()
|
||||
for domain in self.auth.allowed_email_domains:
|
||||
if email_lower.endswith(f"@{domain.lower()}"):
|
||||
return True
|
||||
|
||||
return not self.auth.require_email_domain
|
||||
|
||||
|
||||
class TenantPublicConfig(BaseModel):
|
||||
"""
|
||||
Public tenant configuration returned to frontend.
|
||||
|
||||
This excludes sensitive auth details like token IDs.
|
||||
"""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
subdomain: str
|
||||
is_active: bool
|
||||
branding: TenantBranding
|
||||
features: TenantFeatures
|
||||
defaults: TenantDefaults
|
||||
# Auth info limited to what frontend needs
|
||||
allowed_email_domains: List[str] = Field(default_factory=list)
|
||||
|
||||
@classmethod
|
||||
def from_config(cls, config: TenantConfig) -> "TenantPublicConfig":
|
||||
"""Create public config from full config."""
|
||||
return cls(
|
||||
id=config.id,
|
||||
name=config.name,
|
||||
subdomain=config.subdomain,
|
||||
is_active=config.is_active,
|
||||
branding=config.branding,
|
||||
features=config.features,
|
||||
defaults=TenantDefaults(
|
||||
theme_id=config.defaults.theme_id,
|
||||
locked_theme=config.defaults.locked_theme,
|
||||
distribution_mode=config.defaults.distribution_mode,
|
||||
# Don't expose brand_prefix or youtube_description_template
|
||||
),
|
||||
allowed_email_domains=config.auth.allowed_email_domains,
|
||||
)
|
||||
|
||||
|
||||
class TenantConfigResponse(BaseModel):
|
||||
"""Response from GET /api/tenant/config endpoint."""
|
||||
|
||||
tenant: Optional[TenantPublicConfig] = Field(
|
||||
None, description="Tenant config if found, null for default Nomad Karaoke"
|
||||
)
|
||||
is_default: bool = Field(
|
||||
True, description="True if using default Nomad Karaoke config"
|
||||
)
|
||||
@@ -1,153 +0,0 @@
|
||||
"""
|
||||
Theme data models for karaoke-gen.
|
||||
|
||||
Themes are pre-created style configurations stored in GCS that users can select
|
||||
when creating karaoke jobs. Each theme includes:
|
||||
- Style parameters (fonts, colors, backgrounds, layouts)
|
||||
- Asset files (images, fonts)
|
||||
- Preview images
|
||||
- Optional YouTube description template
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ThemeSummary(BaseModel):
|
||||
"""Summary of a theme for listing purposes."""
|
||||
|
||||
id: str = Field(..., description="Unique theme identifier (e.g., 'nomad', 'default')")
|
||||
name: str = Field(..., description="Human-readable theme name")
|
||||
description: str = Field(..., description="Brief description of the theme style")
|
||||
preview_url: Optional[str] = Field(None, description="Signed URL for preview image")
|
||||
thumbnail_url: Optional[str] = Field(None, description="Signed URL for smaller thumbnail")
|
||||
is_default: bool = Field(False, description="Whether this is the default theme")
|
||||
|
||||
|
||||
class ThemeMetadata(BaseModel):
|
||||
"""Metadata for a single theme in the registry."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
is_default: bool = False
|
||||
created_at: Optional[datetime] = None
|
||||
|
||||
|
||||
class ThemeRegistry(BaseModel):
|
||||
"""Registry of all available themes, stored as _metadata.json in GCS."""
|
||||
|
||||
version: int = 1
|
||||
themes: List[ThemeMetadata] = Field(default_factory=list)
|
||||
|
||||
|
||||
class ThemeDetail(BaseModel):
|
||||
"""Full theme details including style parameters."""
|
||||
|
||||
id: str
|
||||
name: str
|
||||
description: str
|
||||
preview_url: Optional[str] = None
|
||||
is_default: bool = False
|
||||
style_params: Dict[str, Any] = Field(default_factory=dict)
|
||||
has_youtube_description: bool = False
|
||||
|
||||
|
||||
class ColorOverrides(BaseModel):
|
||||
"""
|
||||
User-customizable color overrides that can be applied on top of a theme.
|
||||
|
||||
All colors are hex format (#RRGGBB). When applied:
|
||||
- artist_color: Applied to intro.artist_color, end.artist_color, cdg.artist_color
|
||||
- title_color: Applied to intro.title_color, end.title_color, cdg.title_color
|
||||
- sung_lyrics_color: Applied to karaoke.primary_color (converted to RGBA), cdg.active_fill
|
||||
- unsung_lyrics_color: Applied to karaoke.secondary_color (converted to RGBA), cdg.inactive_fill
|
||||
"""
|
||||
|
||||
artist_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for artist name on intro/end screens",
|
||||
)
|
||||
title_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for song title on intro/end screens",
|
||||
)
|
||||
sung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics being sung (highlighted)",
|
||||
)
|
||||
unsung_lyrics_color: Optional[str] = Field(
|
||||
None,
|
||||
pattern=r"^#[0-9A-Fa-f]{6}$",
|
||||
description="Hex color for lyrics not yet sung",
|
||||
)
|
||||
|
||||
def has_overrides(self) -> bool:
|
||||
"""Check if any color overrides are set."""
|
||||
return any(
|
||||
[
|
||||
self.artist_color,
|
||||
self.title_color,
|
||||
self.sung_lyrics_color,
|
||||
self.unsung_lyrics_color,
|
||||
]
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, str]:
|
||||
"""Convert to dict, excluding None values."""
|
||||
return {k: v for k, v in self.model_dump().items() if v is not None}
|
||||
|
||||
|
||||
class ThemesListResponse(BaseModel):
|
||||
"""Response from GET /api/themes endpoint."""
|
||||
|
||||
themes: List[ThemeSummary]
|
||||
|
||||
|
||||
class ThemeDetailResponse(BaseModel):
|
||||
"""Response from GET /api/themes/{theme_id} endpoint."""
|
||||
|
||||
theme: ThemeDetail
|
||||
|
||||
|
||||
# Utility functions for color conversion
|
||||
|
||||
|
||||
def hex_to_rgba(hex_color: str, alpha: int = 255) -> str:
|
||||
"""
|
||||
Convert hex color (#RRGGBB) to RGBA format for ASS subtitles.
|
||||
|
||||
ASS subtitle format uses "R, G, B, A" string format.
|
||||
|
||||
Args:
|
||||
hex_color: Hex color string like "#7070F7"
|
||||
alpha: Alpha value 0-255 (default 255 = opaque)
|
||||
|
||||
Returns:
|
||||
RGBA string like "112, 112, 247, 255"
|
||||
"""
|
||||
hex_color = hex_color.lstrip("#")
|
||||
r = int(hex_color[0:2], 16)
|
||||
g = int(hex_color[2:4], 16)
|
||||
b = int(hex_color[4:6], 16)
|
||||
return f"{r}, {g}, {b}, {alpha}"
|
||||
|
||||
|
||||
def rgba_to_hex(rgba_str: str) -> str:
|
||||
"""
|
||||
Convert RGBA format to hex color.
|
||||
|
||||
Args:
|
||||
rgba_str: RGBA string like "112, 112, 247, 255"
|
||||
|
||||
Returns:
|
||||
Hex color string like "#7070F7"
|
||||
"""
|
||||
parts = [int(x.strip()) for x in rgba_str.split(",")]
|
||||
r, g, b = parts[0], parts[1], parts[2]
|
||||
return f"#{r:02x}{g:02x}{b:02x}"
|
||||
@@ -1,290 +0,0 @@
|
||||
"""
|
||||
User model for authentication and credits.
|
||||
|
||||
Supports:
|
||||
- Magic link authentication (email-based)
|
||||
- Credit system for karaoke generation
|
||||
- Role-based access control (user/admin)
|
||||
- Stripe integration for payments
|
||||
- Beta tester program with feedback collection
|
||||
"""
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
from typing import Optional, List, Dict
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class UserRole(str, Enum):
|
||||
"""User roles for access control."""
|
||||
USER = "user"
|
||||
ADMIN = "admin"
|
||||
|
||||
|
||||
class BetaTesterStatus(str, Enum):
|
||||
"""Status of beta tester participation."""
|
||||
ACTIVE = "active" # Enrolled, free credits available
|
||||
PENDING_FEEDBACK = "pending_feedback" # Job completed, awaiting feedback
|
||||
COMPLETED = "completed" # Feedback submitted
|
||||
EXPIRED = "expired" # 24hr deadline passed without feedback
|
||||
|
||||
|
||||
class CreditTransaction(BaseModel):
|
||||
"""Record of a credit transaction."""
|
||||
id: str
|
||||
amount: int # Positive = add, negative = deduct
|
||||
reason: str # e.g., "purchase", "refund", "admin_grant", "job_creation"
|
||||
job_id: Optional[str] = None
|
||||
stripe_session_id: Optional[str] = None
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
created_by: Optional[str] = None # Admin email if granted by admin
|
||||
|
||||
|
||||
class PushSubscription(BaseModel):
|
||||
"""
|
||||
Web Push subscription for a user's device.
|
||||
|
||||
Stores the push subscription endpoint and encryption keys needed
|
||||
to send push notifications to the user's browser/device.
|
||||
"""
|
||||
endpoint: str # Push service endpoint URL
|
||||
keys: Dict[str, str] # p256dh and auth keys for encryption
|
||||
device_name: Optional[str] = None # e.g., "iPhone", "Chrome on Windows"
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
last_used_at: Optional[datetime] = None # Last time a notification was sent
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
"""
|
||||
User model stored in Firestore.
|
||||
|
||||
Users are identified by email address. Authentication is via magic links
|
||||
(no passwords). Credits are consumed when creating karaoke jobs.
|
||||
|
||||
For multi-tenant white-label portals, users are scoped to a tenant.
|
||||
Users with tenant_id=None are default Nomad Karaoke users.
|
||||
"""
|
||||
email: str # Primary identifier
|
||||
role: UserRole = UserRole.USER
|
||||
credits: int = 0
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
# Stripe integration
|
||||
stripe_customer_id: Optional[str] = None
|
||||
|
||||
# Account state
|
||||
is_active: bool = True
|
||||
email_verified: bool = False
|
||||
|
||||
# Timestamps
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
updated_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
last_login_at: Optional[datetime] = None
|
||||
|
||||
# Credit history (last 100 transactions)
|
||||
credit_transactions: List[CreditTransaction] = Field(default_factory=list)
|
||||
|
||||
# Job tracking
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
|
||||
# Optional profile fields for future use
|
||||
display_name: Optional[str] = None
|
||||
|
||||
# Beta tester program
|
||||
is_beta_tester: bool = False
|
||||
beta_tester_status: Optional[BetaTesterStatus] = None
|
||||
beta_enrolled_at: Optional[datetime] = None
|
||||
beta_promise_text: Optional[str] = None # User's promise statement
|
||||
beta_feedback_due_at: Optional[datetime] = None # 24hr after job completion
|
||||
beta_feedback_email_sent: bool = False
|
||||
|
||||
# Push notification subscriptions (Web Push API)
|
||||
# Users can subscribe from multiple devices/browsers
|
||||
push_subscriptions: List[PushSubscription] = Field(default_factory=list)
|
||||
|
||||
|
||||
class MagicLinkToken(BaseModel):
|
||||
"""
|
||||
Magic link token for passwordless authentication.
|
||||
|
||||
Tokens are short-lived (15 minutes) and single-use.
|
||||
Stored in Firestore with automatic TTL.
|
||||
|
||||
For multi-tenant portals, the tenant_id determines which portal
|
||||
the user will be redirected to after verification.
|
||||
"""
|
||||
token: str # Secure random token
|
||||
email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
used: bool = False
|
||||
used_at: Optional[datetime] = None
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
"""
|
||||
User session for authenticated requests.
|
||||
|
||||
Sessions are created after successful magic link verification.
|
||||
Sessions expire after 7 days of inactivity or 30 days absolute.
|
||||
|
||||
For multi-tenant portals, sessions are scoped to a tenant.
|
||||
"""
|
||||
token: str # Secure random session token
|
||||
user_email: str
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
expires_at: datetime
|
||||
last_activity_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
ip_address: Optional[str] = None
|
||||
user_agent: Optional[str] = None
|
||||
is_active: bool = True
|
||||
|
||||
# Multi-tenant support (None = default Nomad Karaoke)
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
# Pydantic models for API requests/responses
|
||||
|
||||
class SendMagicLinkRequest(BaseModel):
|
||||
"""Request to send a magic link email."""
|
||||
email: str
|
||||
|
||||
|
||||
class SendMagicLinkResponse(BaseModel):
|
||||
"""Response after sending magic link."""
|
||||
status: str
|
||||
message: str
|
||||
|
||||
|
||||
class VerifyMagicLinkRequest(BaseModel):
|
||||
"""Request to verify a magic link token."""
|
||||
token: str
|
||||
|
||||
|
||||
class VerifyMagicLinkResponse(BaseModel):
|
||||
"""Response after verifying magic link."""
|
||||
status: str
|
||||
session_token: str
|
||||
user: "UserPublic"
|
||||
message: str
|
||||
|
||||
|
||||
class UserPublic(BaseModel):
|
||||
"""Public user information (safe to expose to frontend)."""
|
||||
email: str
|
||||
role: UserRole
|
||||
credits: int
|
||||
display_name: Optional[str] = None
|
||||
total_jobs_created: int = 0
|
||||
total_jobs_completed: int = 0
|
||||
tenant_id: Optional[str] = None
|
||||
|
||||
|
||||
class AddCreditsRequest(BaseModel):
|
||||
"""Admin request to add credits to a user."""
|
||||
email: str
|
||||
amount: int
|
||||
reason: str = "admin_grant"
|
||||
|
||||
|
||||
class AddCreditsResponse(BaseModel):
|
||||
"""Response after adding credits."""
|
||||
status: str
|
||||
email: str
|
||||
credits_added: int
|
||||
new_balance: int
|
||||
message: str
|
||||
|
||||
|
||||
class UserListResponse(BaseModel):
|
||||
"""Response for listing users (admin only)."""
|
||||
users: List[UserPublic]
|
||||
total: int
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Beta Tester Program Models
|
||||
# ============================================================================
|
||||
|
||||
class BetaTesterFeedback(BaseModel):
|
||||
"""
|
||||
Feedback from a beta tester after using the service.
|
||||
|
||||
Stored in Firestore 'beta_feedback' collection.
|
||||
"""
|
||||
id: str # Unique feedback ID
|
||||
user_email: str
|
||||
job_id: Optional[str] = None # The job they're providing feedback on
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
# Metadata
|
||||
created_at: datetime = Field(default_factory=datetime.utcnow)
|
||||
submitted_via: str = "web" # 'web' or 'email'
|
||||
|
||||
|
||||
class BetaTesterEnrollRequest(BaseModel):
|
||||
"""Request to enroll as a beta tester."""
|
||||
email: str
|
||||
promise_text: str = Field(
|
||||
min_length=10,
|
||||
description="User's promise to provide feedback (min 10 chars)"
|
||||
)
|
||||
accept_corrections_work: bool = Field(
|
||||
description="User accepts they may need to review/correct lyrics"
|
||||
)
|
||||
|
||||
|
||||
class BetaTesterEnrollResponse(BaseModel):
|
||||
"""Response after enrolling as beta tester."""
|
||||
status: str
|
||||
message: str
|
||||
credits_granted: int
|
||||
session_token: Optional[str] = None # If new user, provide session
|
||||
|
||||
|
||||
class BetaFeedbackRequest(BaseModel):
|
||||
"""Request to submit beta tester feedback."""
|
||||
job_id: Optional[str] = None
|
||||
|
||||
# Ratings (1-5 scale)
|
||||
overall_rating: int = Field(ge=1, le=5)
|
||||
ease_of_use_rating: int = Field(ge=1, le=5)
|
||||
lyrics_accuracy_rating: int = Field(ge=1, le=5)
|
||||
correction_experience_rating: int = Field(ge=1, le=5)
|
||||
|
||||
# Open-ended feedback
|
||||
what_went_well: Optional[str] = None
|
||||
what_could_improve: Optional[str] = None
|
||||
additional_comments: Optional[str] = None
|
||||
|
||||
# Would they recommend / use again?
|
||||
would_recommend: bool = True
|
||||
would_use_again: bool = True
|
||||
|
||||
|
||||
class BetaFeedbackResponse(BaseModel):
|
||||
"""Response after submitting feedback."""
|
||||
status: str
|
||||
message: str
|
||||
bonus_credits: int = 0 # Bonus credits for great feedback
|
||||
@@ -1,164 +0,0 @@
|
||||
"""
|
||||
Worker log entry model for subcollection storage.
|
||||
|
||||
This module defines the LogEntry model for storing worker logs in a Firestore
|
||||
subcollection instead of an embedded array. This avoids the 1MB document size
|
||||
limit that caused job 501258e1 to fail when logs reached 1.26 MB.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from typing import Optional, Dict, Any
|
||||
import uuid
|
||||
|
||||
|
||||
# Default TTL for log entries (30 days)
|
||||
DEFAULT_LOG_TTL_DAYS = 30
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkerLogEntry:
|
||||
"""
|
||||
Worker log entry for Firestore subcollection storage.
|
||||
|
||||
Stored at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This model is separate from the legacy LogEntry (in job.py) which is
|
||||
stored as an embedded array in the job document. This new model supports
|
||||
the subcollection approach with TTL and richer metadata.
|
||||
"""
|
||||
# Core fields (required)
|
||||
timestamp: datetime
|
||||
level: str # "DEBUG", "INFO", "WARNING", "ERROR"
|
||||
worker: str # "audio", "lyrics", "screens", "video", "render", "distribution"
|
||||
message: str
|
||||
|
||||
# Identifiers
|
||||
id: str = field(default_factory=lambda: str(uuid.uuid4()))
|
||||
job_id: str = "" # Set when writing to Firestore
|
||||
|
||||
# TTL for automatic cleanup
|
||||
ttl_expiry: datetime = field(
|
||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
)
|
||||
|
||||
# Optional metadata for debugging
|
||||
metadata: Optional[Dict[str, Any]] = None
|
||||
|
||||
@classmethod
|
||||
def create(
|
||||
cls,
|
||||
job_id: str,
|
||||
worker: str,
|
||||
level: str,
|
||||
message: str,
|
||||
metadata: Optional[Dict[str, Any]] = None,
|
||||
ttl_days: int = DEFAULT_LOG_TTL_DAYS
|
||||
) -> "WorkerLogEntry":
|
||||
"""
|
||||
Factory method to create a new log entry.
|
||||
|
||||
Args:
|
||||
job_id: Job ID this log belongs to
|
||||
worker: Worker name (audio, lyrics, screens, video, render, distribution)
|
||||
level: Log level (DEBUG, INFO, WARNING, ERROR)
|
||||
message: Log message (truncated to 1000 chars)
|
||||
metadata: Optional additional metadata
|
||||
ttl_days: Days until log expires (default 30)
|
||||
|
||||
Returns:
|
||||
New WorkerLogEntry instance
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
return cls(
|
||||
id=str(uuid.uuid4()),
|
||||
job_id=job_id,
|
||||
timestamp=now,
|
||||
level=level.upper(),
|
||||
worker=worker,
|
||||
message=message[:1000], # Truncate long messages
|
||||
metadata=metadata,
|
||||
ttl_expiry=now + timedelta(days=ttl_days)
|
||||
)
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to dictionary for Firestore storage.
|
||||
|
||||
Returns:
|
||||
Dictionary representation for Firestore
|
||||
"""
|
||||
result = {
|
||||
"id": self.id,
|
||||
"job_id": self.job_id,
|
||||
"timestamp": self.timestamp,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message,
|
||||
"ttl_expiry": self.ttl_expiry
|
||||
}
|
||||
if self.metadata:
|
||||
result["metadata"] = self.metadata
|
||||
return result
|
||||
|
||||
def to_legacy_dict(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert to legacy format for backward compatibility with API responses.
|
||||
|
||||
The existing API returns logs in this format:
|
||||
{"timestamp": "...", "level": "INFO", "worker": "audio", "message": "..."}
|
||||
|
||||
Returns:
|
||||
Dictionary in legacy format
|
||||
"""
|
||||
# Format timestamp: use "Z" suffix for UTC, isoformat() for others
|
||||
if self.timestamp.tzinfo is None:
|
||||
# Naive datetime - use as-is
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
elif self.timestamp.utcoffset() == timedelta(0):
|
||||
# UTC datetime - replace +00:00 with Z for cleaner format
|
||||
timestamp_str = self.timestamp.replace(tzinfo=None).isoformat() + "Z"
|
||||
else:
|
||||
# Non-UTC timezone-aware - use full isoformat with offset
|
||||
timestamp_str = self.timestamp.isoformat()
|
||||
|
||||
return {
|
||||
"timestamp": timestamp_str,
|
||||
"level": self.level,
|
||||
"worker": self.worker,
|
||||
"message": self.message
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, data: Dict[str, Any]) -> "WorkerLogEntry":
|
||||
"""
|
||||
Create from Firestore document data.
|
||||
|
||||
Args:
|
||||
data: Dictionary from Firestore document
|
||||
|
||||
Returns:
|
||||
WorkerLogEntry instance
|
||||
"""
|
||||
# Handle timestamp conversion (Firestore returns datetime objects)
|
||||
timestamp = data.get("timestamp")
|
||||
if isinstance(timestamp, str):
|
||||
# Parse ISO format string
|
||||
timestamp = datetime.fromisoformat(timestamp.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
|
||||
ttl_expiry = data.get("ttl_expiry")
|
||||
if isinstance(ttl_expiry, str):
|
||||
ttl_expiry = datetime.fromisoformat(ttl_expiry.rstrip("Z")).replace(tzinfo=timezone.utc)
|
||||
elif ttl_expiry is None:
|
||||
# Default TTL if not set
|
||||
ttl_expiry = datetime.now(timezone.utc) + timedelta(days=DEFAULT_LOG_TTL_DAYS)
|
||||
|
||||
return cls(
|
||||
id=data.get("id", str(uuid.uuid4())),
|
||||
job_id=data.get("job_id", ""),
|
||||
timestamp=timestamp or datetime.now(timezone.utc),
|
||||
level=data.get("level", "INFO"),
|
||||
worker=data.get("worker", "unknown"),
|
||||
message=data.get("message", ""),
|
||||
metadata=data.get("metadata"),
|
||||
ttl_expiry=ttl_expiry
|
||||
)
|
||||
@@ -1,29 +0,0 @@
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--tb=short",
|
||||
"--strict-markers",
|
||||
"-ra",
|
||||
]
|
||||
|
||||
markers = [
|
||||
"slow: marks tests as slow (deselect with '-m \"not slow\"')",
|
||||
"integration: marks tests as integration tests requiring live service",
|
||||
]
|
||||
|
||||
# Timeout for slow tests (in seconds)
|
||||
timeout = 300
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["backend"]
|
||||
omit = ["*/tests/*", "*/test_*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
precision = 2
|
||||
show_missing = true
|
||||
skip_covered = false
|
||||
|
||||
@@ -1,93 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Quick syntax and structure check (no dependencies required)
|
||||
# Run this before every deployment!
|
||||
|
||||
set -e
|
||||
|
||||
echo "🔍 Quick Backend Check (no dependencies required)"
|
||||
echo "=================================================="
|
||||
echo ""
|
||||
|
||||
# Change to backend directory
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
ERRORS=0
|
||||
|
||||
# Check 1: Python syntax
|
||||
echo "1. Checking Python syntax..."
|
||||
for file in $(find . -name "*.py" -not -path "./__pycache__/*" -not -path "./venv/*"); do
|
||||
if ! python3 -m py_compile "$file" 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Syntax error in $file${NC}"
|
||||
python3 -m py_compile "$file"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All Python files have valid syntax${NC}"
|
||||
fi
|
||||
|
||||
# Check 2: No missing imports (basic check)
|
||||
echo ""
|
||||
echo "2. Checking for obvious import issues..."
|
||||
|
||||
# Check that we don't import deleted modules
|
||||
if grep -r "from backend.services.processing_service import" --include="*.py" . 2>/dev/null; then
|
||||
echo -e " ${RED}❌ Found import of deleted processing_service${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
else
|
||||
echo -e " ${GREEN}✅ No imports of deleted modules${NC}"
|
||||
fi
|
||||
|
||||
# Check 3: Required files exist
|
||||
echo ""
|
||||
echo "3. Checking required files exist..."
|
||||
REQUIRED_FILES=(
|
||||
"main.py"
|
||||
"config.py"
|
||||
"requirements.txt"
|
||||
"Dockerfile"
|
||||
"api/routes/health.py"
|
||||
"api/routes/jobs.py"
|
||||
"api/routes/internal.py"
|
||||
"api/routes/file_upload.py"
|
||||
"services/job_manager.py"
|
||||
"services/firestore_service.py"
|
||||
"services/storage_service.py"
|
||||
"services/worker_service.py"
|
||||
"services/auth_service.py"
|
||||
"workers/audio_worker.py"
|
||||
"workers/lyrics_worker.py"
|
||||
"workers/screens_worker.py"
|
||||
"workers/video_worker.py"
|
||||
"models/job.py"
|
||||
"models/requests.py"
|
||||
)
|
||||
|
||||
for file in "${REQUIRED_FILES[@]}"; do
|
||||
if [ ! -f "$file" ]; then
|
||||
echo -e " ${RED}❌ Missing required file: $file${NC}"
|
||||
ERRORS=$((ERRORS+1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e " ${GREEN}✅ All required files exist${NC}"
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
echo "=================================================="
|
||||
if [ $ERRORS -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ All checks passed! Safe to deploy.${NC}"
|
||||
exit 0
|
||||
else
|
||||
echo -e "${RED}❌ $ERRORS error(s) found. Fix before deploying!${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# DEPRECATED: This file is no longer used for Docker builds or CI.
|
||||
# Docker and CI both use poetry/pyproject.toml directly.
|
||||
# This file is kept for reference only.
|
||||
|
||||
fastapi>=0.104.0,<1.0.0
|
||||
uvicorn[standard]>=0.24.0,<1.0.0
|
||||
pydantic>=2.5.0,<3.0.0
|
||||
pydantic-settings>=2.1.0,<3.0.0
|
||||
python-multipart>=0.0.6,<1.0.0
|
||||
httpx>=0.25.0,<1.0.0
|
||||
google-cloud-firestore>=2.14.0,<3.0.0
|
||||
google-cloud-storage>=2.14.0,<3.0.0
|
||||
google-cloud-secret-manager>=2.18.0,<3.0.0
|
||||
google-cloud-tasks>=2.14.0,<3.0.0
|
||||
google-cloud-run>=0.10.0,<1.0.0
|
||||
python-dotenv>=1.0.0,<2.0.0
|
||||
|
||||
# Audio processing (reuses karaoke_gen library)
|
||||
audio-separator[remote]>=0.18.0 # Remote Modal API client
|
||||
pydub>=0.25.1 # Audio manipulation (used by karaoke_gen)
|
||||
yt-dlp>=2024.0.0 # YouTube downloads (future use)
|
||||
|
||||
# Cloud storage native APIs
|
||||
dropbox>=11.0.0 # Dropbox Python SDK for native API access
|
||||
# google-api-python-client already installed via karaoke_gen for YouTube
|
||||
|
||||
# karaoke-gen dependencies (will be installed from parent dir in Dockerfile)
|
||||
# Using explicit dependencies instead of -e ../
|
||||
|
||||
@@ -1,60 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Backend Test Runner
|
||||
# Runs integration tests against the deployed Cloud Run service
|
||||
|
||||
set -e
|
||||
|
||||
echo "🧪 Backend Integration Test Suite"
|
||||
echo "=================================="
|
||||
echo ""
|
||||
|
||||
# Check prerequisites
|
||||
echo "Checking prerequisites..."
|
||||
if ! command -v gcloud &> /dev/null; then
|
||||
echo "❌ gcloud CLI not found. Please install Google Cloud SDK."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command -v pytest &> /dev/null; then
|
||||
echo "❌ pytest not found. Installing test dependencies..."
|
||||
pip install -r tests/requirements.txt
|
||||
fi
|
||||
|
||||
# Verify authentication
|
||||
echo "Verifying authentication..."
|
||||
if ! gcloud auth print-identity-token &> /dev/null; then
|
||||
echo "❌ Not authenticated with gcloud. Run: gcloud auth login"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ Prerequisites met"
|
||||
echo ""
|
||||
|
||||
# Run tests
|
||||
echo "Running integration tests..."
|
||||
echo ""
|
||||
|
||||
# Run fast tests (no slow marker)
|
||||
echo "📝 Running fast tests..."
|
||||
pytest tests/test_api_integration.py -v -m "not slow" --tb=short
|
||||
|
||||
echo ""
|
||||
echo "✅ Fast tests complete!"
|
||||
echo ""
|
||||
|
||||
# Ask if user wants to run slow tests
|
||||
read -p "Run slow/integration tests? These test actual job processing (5-10 min). [y/N] " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "🐌 Running slow integration tests..."
|
||||
pytest tests/test_api_integration.py -v -m "slow" --tb=short
|
||||
echo ""
|
||||
echo "✅ All tests complete!"
|
||||
else
|
||||
echo "⏭️ Skipping slow tests"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=================================="
|
||||
echo "Test run complete! 🎉"
|
||||
|
||||
Whitespace-only changes.
@@ -1,243 +0,0 @@
|
||||
"""
|
||||
Audio analysis service for analyzing backing vocals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio analysis capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioAnalyzer,
|
||||
AnalysisResult,
|
||||
WaveformGenerator,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioAnalysisService:
|
||||
"""
|
||||
Service for analyzing backing vocals audio files stored in GCS.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioAnalyzer
|
||||
and WaveformGenerator classes, handling GCS download/upload operations.
|
||||
|
||||
The actual analysis logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
silence_threshold_db: float = -40.0,
|
||||
min_segment_duration_ms: int = 100,
|
||||
):
|
||||
"""
|
||||
Initialize the audio analysis service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
silence_threshold_db: Threshold for considering audio as silent.
|
||||
Default is -40.0 dB.
|
||||
min_segment_duration_ms: Minimum duration for audible segments.
|
||||
Default is 100ms.
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.analyzer = AudioAnalyzer(
|
||||
silence_threshold_db=silence_threshold_db,
|
||||
min_segment_duration_ms=min_segment_duration_ms,
|
||||
)
|
||||
self.waveform_generator = WaveformGenerator()
|
||||
|
||||
def analyze_backing_vocals(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
) -> AnalysisResult:
|
||||
"""
|
||||
Analyze a backing vocals audio file from GCS.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS to a temp file
|
||||
2. Runs the analysis using the shared AudioAnalyzer
|
||||
3. Returns the analysis result
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
AnalysisResult containing analysis data
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing backing vocals: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis complete: "
|
||||
f"has_audible={result.has_audible_content}, "
|
||||
f"segments={result.segment_count}, "
|
||||
f"recommendation={result.recommended_selection.value}"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def analyze_and_generate_waveform(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
) -> tuple[AnalysisResult, str]:
|
||||
"""
|
||||
Analyze backing vocals and generate a waveform image.
|
||||
|
||||
This method:
|
||||
1. Downloads the audio file from GCS
|
||||
2. Runs analysis using AudioAnalyzer
|
||||
3. Generates waveform image using WaveformGenerator
|
||||
4. Uploads the waveform image to GCS
|
||||
5. Returns analysis result and waveform GCS path
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image in GCS
|
||||
|
||||
Returns:
|
||||
Tuple of (AnalysisResult, waveform_gcs_path)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Analyzing and generating waveform: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Run analysis
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
show_time_axis=True,
|
||||
silence_threshold_db=self.analyzer.silence_threshold_db,
|
||||
)
|
||||
|
||||
# Upload waveform to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Analysis and waveform generation complete. "
|
||||
f"Waveform uploaded to: {gcs_waveform_destination}"
|
||||
)
|
||||
|
||||
return result, gcs_waveform_destination
|
||||
|
||||
def get_waveform_data(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
num_points: int = 500,
|
||||
) -> tuple[list[float], float]:
|
||||
"""
|
||||
Get waveform data (amplitude envelope) for client-side rendering.
|
||||
|
||||
This is useful when the frontend wants to render the waveform
|
||||
itself using Canvas or SVG, rather than displaying a pre-generated
|
||||
image.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
num_points: Number of data points to return
|
||||
|
||||
Returns:
|
||||
Tuple of (amplitude_values, duration_seconds)
|
||||
"""
|
||||
logger.info(f"[{job_id}] Getting waveform data: {gcs_audio_path}")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# Generate waveform data
|
||||
amplitudes, duration = self.waveform_generator.generate_data_only(
|
||||
audio_path=local_audio_path,
|
||||
num_points=num_points,
|
||||
)
|
||||
|
||||
return amplitudes, duration
|
||||
|
||||
def generate_waveform_with_mutes(
|
||||
self,
|
||||
gcs_audio_path: str,
|
||||
job_id: str,
|
||||
gcs_waveform_destination: str,
|
||||
mute_regions: list,
|
||||
) -> str:
|
||||
"""
|
||||
Generate a waveform image with mute regions highlighted.
|
||||
|
||||
This is useful for showing the user which regions will be muted
|
||||
in the custom instrumental.
|
||||
|
||||
Args:
|
||||
gcs_audio_path: Path to the audio file in GCS
|
||||
job_id: Job ID for logging
|
||||
gcs_waveform_destination: Where to upload the waveform image
|
||||
mute_regions: List of MuteRegion objects to highlight
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded waveform image
|
||||
"""
|
||||
from karaoke_gen.instrumental_review import MuteRegion
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Generating waveform with {len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download audio file
|
||||
local_audio_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
self.storage_service.download_file(gcs_audio_path, local_audio_path)
|
||||
|
||||
# First run analysis to get segments
|
||||
result = self.analyzer.analyze(local_audio_path)
|
||||
|
||||
# Generate waveform with mute regions
|
||||
local_waveform_path = os.path.join(temp_dir, "waveform_with_mutes.png")
|
||||
self.waveform_generator.generate(
|
||||
audio_path=local_audio_path,
|
||||
output_path=local_waveform_path,
|
||||
segments=result.audible_segments,
|
||||
mute_regions=mute_regions,
|
||||
show_time_axis=True,
|
||||
)
|
||||
|
||||
# Upload to GCS
|
||||
self.storage_service.upload_file(
|
||||
local_waveform_path,
|
||||
gcs_waveform_destination
|
||||
)
|
||||
|
||||
return gcs_waveform_destination
|
||||
@@ -1,278 +0,0 @@
|
||||
"""
|
||||
Audio editing service for creating custom instrumentals.
|
||||
|
||||
This service wraps the shared karaoke_gen.instrumental_review module
|
||||
to provide GCS-integrated audio editing capabilities.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
from typing import List, Optional
|
||||
|
||||
from backend.services.storage_service import StorageService
|
||||
from karaoke_gen.instrumental_review import (
|
||||
AudioEditor,
|
||||
MuteRegion,
|
||||
)
|
||||
from karaoke_gen.instrumental_review.models import CustomInstrumentalResult
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AudioEditingService:
|
||||
"""
|
||||
Service for creating custom instrumental tracks with GCS integration.
|
||||
|
||||
This service acts as a thin wrapper around the shared AudioEditor
|
||||
class, handling GCS download/upload operations.
|
||||
|
||||
The actual editing logic is in the shared karaoke_gen.instrumental_review
|
||||
module, ensuring feature parity between local and remote workflows.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
storage_service: Optional[StorageService] = None,
|
||||
output_format: str = "flac",
|
||||
):
|
||||
"""
|
||||
Initialize the audio editing service.
|
||||
|
||||
Args:
|
||||
storage_service: GCS storage service. If not provided, a new
|
||||
instance will be created.
|
||||
output_format: Output audio format. Default is "flac".
|
||||
"""
|
||||
self.storage_service = storage_service or StorageService()
|
||||
self.editor = AudioEditor(output_format=output_format)
|
||||
|
||||
def create_custom_instrumental(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> CustomInstrumentalResult:
|
||||
"""
|
||||
Create a custom instrumental by muting regions of backing vocals.
|
||||
|
||||
This method:
|
||||
1. Downloads clean instrumental and backing vocals from GCS
|
||||
2. Applies mute regions to backing vocals
|
||||
3. Combines with clean instrumental
|
||||
4. Uploads result to GCS
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute in backing vocals
|
||||
gcs_output_path: Where to upload the custom instrumental in GCS
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
CustomInstrumentalResult with details about the created file
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Creating custom instrumental with "
|
||||
f"{len(mute_regions)} mute regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "custom_instrumental.flac")
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading clean instrumental")
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
|
||||
logger.debug(f"[{job_id}] Downloading backing vocals")
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Create custom instrumental using shared editor
|
||||
result = self.editor.create_custom_instrumental(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
# Upload result to GCS
|
||||
logger.debug(f"[{job_id}] Uploading custom instrumental to GCS")
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
# Update result with GCS path
|
||||
result.output_path = gcs_output_path
|
||||
|
||||
logger.info(
|
||||
f"[{job_id}] Custom instrumental created: {gcs_output_path}, "
|
||||
f"muted {result.total_muted_duration_seconds:.1f}s across "
|
||||
f"{len(result.mute_regions_applied)} regions"
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
def create_preview(
|
||||
self,
|
||||
gcs_clean_instrumental_path: str,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_preview_path: str,
|
||||
job_id: str,
|
||||
preview_duration_seconds: Optional[float] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Create a preview of the custom instrumental.
|
||||
|
||||
This is similar to create_custom_instrumental but optimized for
|
||||
quick preview generation (optionally truncated duration).
|
||||
|
||||
Args:
|
||||
gcs_clean_instrumental_path: Path to clean instrumental in GCS
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_preview_path: Where to upload the preview in GCS
|
||||
job_id: Job ID for logging
|
||||
preview_duration_seconds: Optional max duration for preview
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded preview
|
||||
"""
|
||||
logger.info(f"[{job_id}] Creating preview with {len(mute_regions)} mute regions")
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
# Download input files
|
||||
local_clean_path = os.path.join(temp_dir, "clean_instrumental.flac")
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_preview_path = os.path.join(temp_dir, "preview.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_clean_instrumental_path,
|
||||
local_clean_path
|
||||
)
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
# Generate preview
|
||||
from pydub import AudioSegment
|
||||
|
||||
preview = self.editor.preview_with_mutes(
|
||||
clean_instrumental_path=local_clean_path,
|
||||
backing_vocals_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
)
|
||||
|
||||
# Optionally truncate
|
||||
if preview_duration_seconds:
|
||||
max_ms = int(preview_duration_seconds * 1000)
|
||||
preview = preview[:max_ms]
|
||||
|
||||
# Export and upload
|
||||
preview.export(local_preview_path, format="flac")
|
||||
self.storage_service.upload_file(local_preview_path, gcs_preview_path)
|
||||
|
||||
logger.info(f"[{job_id}] Preview created: {gcs_preview_path}")
|
||||
|
||||
return gcs_preview_path
|
||||
|
||||
def mute_backing_vocals_only(
|
||||
self,
|
||||
gcs_backing_vocals_path: str,
|
||||
mute_regions: List[MuteRegion],
|
||||
gcs_output_path: str,
|
||||
job_id: str,
|
||||
) -> str:
|
||||
"""
|
||||
Apply mute regions to backing vocals without combining with instrumental.
|
||||
|
||||
This is useful for creating an edited backing vocals track that
|
||||
can be combined with the instrumental later.
|
||||
|
||||
Args:
|
||||
gcs_backing_vocals_path: Path to backing vocals in GCS
|
||||
mute_regions: List of regions to mute
|
||||
gcs_output_path: Where to upload the edited backing vocals
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
GCS path to the uploaded edited backing vocals
|
||||
"""
|
||||
logger.info(
|
||||
f"[{job_id}] Muting backing vocals with {len(mute_regions)} regions"
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
local_backing_path = os.path.join(temp_dir, "backing_vocals.flac")
|
||||
local_output_path = os.path.join(temp_dir, "muted_backing.flac")
|
||||
|
||||
self.storage_service.download_file(
|
||||
gcs_backing_vocals_path,
|
||||
local_backing_path
|
||||
)
|
||||
|
||||
self.editor.apply_mute_to_single_track(
|
||||
audio_path=local_backing_path,
|
||||
mute_regions=mute_regions,
|
||||
output_path=local_output_path,
|
||||
)
|
||||
|
||||
self.storage_service.upload_file(local_output_path, gcs_output_path)
|
||||
|
||||
logger.info(f"[{job_id}] Muted backing vocals: {gcs_output_path}")
|
||||
|
||||
return gcs_output_path
|
||||
|
||||
def validate_mute_regions(
|
||||
self,
|
||||
mute_regions: List[MuteRegion],
|
||||
total_duration_seconds: float,
|
||||
) -> List[str]:
|
||||
"""
|
||||
Validate mute regions for consistency.
|
||||
|
||||
Args:
|
||||
mute_regions: List of mute regions to validate
|
||||
total_duration_seconds: Total duration of the audio
|
||||
|
||||
Returns:
|
||||
List of validation error messages (empty if valid)
|
||||
"""
|
||||
errors = []
|
||||
|
||||
for i, region in enumerate(mute_regions):
|
||||
if region.start_seconds < 0:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) cannot be negative"
|
||||
)
|
||||
|
||||
if region.end_seconds <= region.start_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) must be after "
|
||||
f"start_seconds ({region.start_seconds})"
|
||||
)
|
||||
|
||||
if region.start_seconds > total_duration_seconds:
|
||||
errors.append(
|
||||
f"Region {i}: start_seconds ({region.start_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds})"
|
||||
)
|
||||
|
||||
if region.end_seconds > total_duration_seconds:
|
||||
# Not an error, but log a warning - the region will be clamped
|
||||
logger.warning(
|
||||
f"Region {i}: end_seconds ({region.end_seconds}) exceeds "
|
||||
f"audio duration ({total_duration_seconds}), will be clamped"
|
||||
)
|
||||
|
||||
return errors
|
||||
@@ -1,704 +0,0 @@
|
||||
"""
|
||||
Audio search service for finding and downloading audio files.
|
||||
|
||||
This service integrates with karaoke_gen.audio_fetcher (which wraps flacfetch)
|
||||
to search for audio from various sources (YouTube, music trackers, etc.)
|
||||
and download the selected audio file.
|
||||
|
||||
This is a thin wrapper that adds backend-specific functionality:
|
||||
- Caching raw results for API-based selection flow
|
||||
- Firestore-compatible serialization
|
||||
- Singleton pattern for service lifecycle
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This is required for torrent
|
||||
downloads since Cloud Run doesn't support BitTorrent peer connections.
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from typing import List, Optional
|
||||
|
||||
import nest_asyncio
|
||||
|
||||
# Import shared classes from karaoke_gen.audio_fetcher - single source of truth
|
||||
# Note: Import directly from audio_fetcher module, not karaoke_gen.__init__
|
||||
# to avoid pulling in heavier dependencies like lyrics_transcriber
|
||||
from karaoke_gen.audio_fetcher import (
|
||||
FlacFetcher,
|
||||
AudioSearchResult,
|
||||
AudioFetchResult,
|
||||
AudioFetcherError,
|
||||
NoResultsError,
|
||||
DownloadError,
|
||||
)
|
||||
|
||||
from .flacfetch_client import get_flacfetch_client, FlacfetchServiceError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Re-export exception under backend's naming convention for compatibility
|
||||
AudioSearchError = AudioFetcherError
|
||||
|
||||
|
||||
# Also alias AudioFetchResult as AudioDownloadResult for backwards compatibility
|
||||
AudioDownloadResult = AudioFetchResult
|
||||
|
||||
|
||||
class AudioSearchService:
|
||||
"""
|
||||
Service for searching and downloading audio files.
|
||||
|
||||
This is a thin wrapper around karaoke_gen.audio_fetcher.FlacFetcher that adds
|
||||
backend-specific functionality:
|
||||
- Caches raw results for API-based selection (client picks by index)
|
||||
- Provides Firestore-compatible serialization
|
||||
- Manages service lifecycle as singleton
|
||||
- Remote flacfetch service integration for torrent downloads
|
||||
|
||||
When FLACFETCH_API_URL is configured, this service uses the remote flacfetch
|
||||
HTTP API for search and download operations. This enables torrent downloads
|
||||
which are not possible in Cloud Run due to network restrictions.
|
||||
|
||||
The actual flacfetch integration is in karaoke_gen.audio_fetcher.FlacFetcher,
|
||||
which is shared between local CLI and cloud backend.
|
||||
"""
|
||||
|
||||
# Sentinel value to indicate "use environment variable"
|
||||
_USE_ENV = object()
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
red_api_key: Optional[str] = _USE_ENV,
|
||||
red_api_url: Optional[str] = _USE_ENV,
|
||||
ops_api_key: Optional[str] = _USE_ENV,
|
||||
ops_api_url: Optional[str] = _USE_ENV,
|
||||
):
|
||||
"""
|
||||
Initialize the audio search service.
|
||||
|
||||
Args:
|
||||
red_api_key: API key for RED tracker (optional, uses env if not provided)
|
||||
red_api_url: Base URL for RED tracker API (optional, uses env if not provided)
|
||||
ops_api_key: API key for OPS tracker (optional, uses env if not provided)
|
||||
ops_api_url: Base URL for OPS tracker API (optional, uses env if not provided)
|
||||
"""
|
||||
# Use environment variables if not explicitly provided
|
||||
if red_api_key is self._USE_ENV:
|
||||
red_api_key = os.environ.get("RED_API_KEY")
|
||||
if red_api_url is self._USE_ENV:
|
||||
red_api_url = os.environ.get("RED_API_URL")
|
||||
|
||||
if ops_api_key is self._USE_ENV:
|
||||
ops_api_key = os.environ.get("OPS_API_KEY")
|
||||
if ops_api_url is self._USE_ENV:
|
||||
ops_api_url = os.environ.get("OPS_API_URL")
|
||||
|
||||
# Check for remote flacfetch client
|
||||
self._remote_client = get_flacfetch_client()
|
||||
|
||||
# Log which mode we're using
|
||||
if self._remote_client:
|
||||
logger.info("AudioSearchService using REMOTE flacfetch service (torrent downloads enabled)")
|
||||
else:
|
||||
logger.info("AudioSearchService using LOCAL flacfetch (YouTube only in Cloud Run)")
|
||||
|
||||
# Delegate to shared FlacFetcher implementation (for local mode or fallback)
|
||||
self._fetcher = FlacFetcher(
|
||||
red_api_key=red_api_key,
|
||||
red_api_url=red_api_url,
|
||||
ops_api_key=ops_api_key,
|
||||
ops_api_url=ops_api_url,
|
||||
)
|
||||
|
||||
# Cache search results for API-based selection flow
|
||||
# Key: index, Value: AudioSearchResult (with raw_result)
|
||||
self._cached_results: List[AudioSearchResult] = []
|
||||
|
||||
# Cache for remote search results (search_id -> results)
|
||||
self._remote_search_id: Optional[str] = None
|
||||
|
||||
def search(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search for audio matching the given artist and title.
|
||||
|
||||
Results are cached internally for later download via download().
|
||||
|
||||
If a remote flacfetch service is configured (FLACFETCH_API_URL), uses that
|
||||
for better torrent download support. Otherwise falls back to local flacfetch.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
|
||||
Returns:
|
||||
List of AudioSearchResult objects
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
AudioSearchError: For other errors (e.g., flacfetch not installed)
|
||||
"""
|
||||
try:
|
||||
# Try remote flacfetch service first if configured
|
||||
if self._remote_client:
|
||||
return self._search_remote(artist, title)
|
||||
|
||||
# Fallback to local flacfetch
|
||||
results = self._fetcher.search(artist, title)
|
||||
|
||||
# Cache results for later download
|
||||
self._cached_results = results
|
||||
self._remote_search_id = None
|
||||
|
||||
logger.info(f"Found {len(results)} results for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except (NoResultsError, AudioFetcherError):
|
||||
# Re-raise these as-is
|
||||
raise
|
||||
except Exception as e:
|
||||
logger.error(f"Search failed: {e}")
|
||||
raise AudioSearchError(f"Search failed: {e}") from e
|
||||
|
||||
def _search_remote(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Search using the remote flacfetch service.
|
||||
|
||||
Runs async code in a sync context for compatibility with existing API.
|
||||
"""
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
# Apply to the current loop if one exists, otherwise it's a no-op
|
||||
try:
|
||||
loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
# Run async search in sync context
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
response = loop.run_until_complete(
|
||||
self._remote_client.search(artist, title)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Check for empty results
|
||||
if not response.get("results"):
|
||||
raise NoResultsError(f"No results found for: {artist} - {title}")
|
||||
|
||||
# Store search_id for download
|
||||
self._remote_search_id = response.get("search_id")
|
||||
|
||||
# Convert remote results to AudioSearchResult objects
|
||||
results = []
|
||||
for item in response.get("results", []):
|
||||
result = AudioSearchResult(
|
||||
title=item.get("title", "Unknown"),
|
||||
artist=item.get("artist", "Unknown"),
|
||||
url=item.get("download_url", ""), # May be empty for remote
|
||||
provider=item.get("provider", "Unknown"),
|
||||
duration=item.get("duration_seconds"),
|
||||
quality=item.get("quality", ""),
|
||||
# source_id: use explicit source_id field first (YouTube video ID, Spotify track ID),
|
||||
# fall back to info_hash for torrent sources (RED/OPS)
|
||||
source_id=item.get("source_id") or item.get("info_hash"),
|
||||
index=item.get("index", 0),
|
||||
seeders=item.get("seeders"),
|
||||
target_file=item.get("target_file"),
|
||||
# Store full remote data in raw_result for rich display
|
||||
raw_result=item,
|
||||
)
|
||||
results.append(result)
|
||||
|
||||
# Cache results
|
||||
self._cached_results = results
|
||||
|
||||
logger.info(f"Found {len(results)} results from remote flacfetch for: {artist} - {title}")
|
||||
return results
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
logger.warning(f"Remote flacfetch search failed, falling back to local: {e}")
|
||||
# Fallback to local search
|
||||
self._remote_search_id = None
|
||||
results = self._fetcher.search(artist, title)
|
||||
self._cached_results = results
|
||||
return results
|
||||
|
||||
def select_best(self, results: List[AudioSearchResult]) -> int:
|
||||
"""
|
||||
Select the best result from a list of search results.
|
||||
|
||||
Uses flacfetch's built-in quality ranking to select the best source.
|
||||
|
||||
Args:
|
||||
results: List of search results
|
||||
|
||||
Returns:
|
||||
Index of the best result
|
||||
"""
|
||||
return self._fetcher.select_best(results)
|
||||
|
||||
def download(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio from a cached search result.
|
||||
|
||||
This method uses the cached results from the last search() call.
|
||||
The API flow is:
|
||||
1. Client calls search() -> gets list of results
|
||||
2. Client picks an index
|
||||
3. Client calls download(index) -> gets downloaded file
|
||||
|
||||
If a remote flacfetch service is configured and the search was performed
|
||||
remotely (for torrent sources), uses the remote service for download.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download (from search results)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads (e.g., "uploads/job123/audio/")
|
||||
remote_search_id: Optional search_id for remote downloads. If provided,
|
||||
uses this instead of the cached _remote_search_id. This is important
|
||||
for concurrent requests where the service-level cache may be stale.
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails or no cached result for index
|
||||
"""
|
||||
# Use provided search_id or fall back to service-level cache
|
||||
effective_search_id = remote_search_id or self._remote_search_id
|
||||
|
||||
# Check if we have local cached results
|
||||
has_local_cache = result_index >= 0 and result_index < len(self._cached_results)
|
||||
|
||||
# If no local cache but we have remote search_id, use remote download
|
||||
# This handles multi-instance deployments where the cache doesn't persist
|
||||
if not has_local_cache:
|
||||
if effective_search_id and self._remote_client:
|
||||
logger.info(f"No local cache, using remote download with search_id={effective_search_id}, index={result_index}")
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
else:
|
||||
# Provide clear error message based on whether cache is empty or index out of bounds
|
||||
if len(self._cached_results) == 0:
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
"No cached results available. Run search() first."
|
||||
)
|
||||
raise DownloadError(
|
||||
f"No cached result for index {result_index}. "
|
||||
f"Available indices: 0-{len(self._cached_results) - 1}. "
|
||||
"Run search() first."
|
||||
)
|
||||
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
logger.info(f"Downloading: {result.artist} - {result.title} from {result.provider}")
|
||||
|
||||
# Check if this was a remote search (torrent sources need remote download)
|
||||
if effective_search_id and self._remote_client:
|
||||
# Check if this is a torrent source that needs remote handling
|
||||
is_torrent = result.provider in ["RED", "OPS"]
|
||||
|
||||
if is_torrent:
|
||||
return self._download_remote(result_index, output_dir, output_filename, gcs_path, effective_search_id)
|
||||
|
||||
# Delegate to shared FlacFetcher (local download)
|
||||
fetch_result = self._fetcher.download(result, output_dir, output_filename)
|
||||
|
||||
logger.info(f"Downloaded to: {fetch_result.filepath}")
|
||||
|
||||
return fetch_result
|
||||
|
||||
def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download audio directly by source ID (no prior search required).
|
||||
|
||||
This is the preferred method when you have stored the source_id from a
|
||||
previous search and want to download later without re-searching. This
|
||||
avoids unnecessary API calls to private trackers.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
target_file: For torrents, specific file to extract from the torrent
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
logger.info(f"Download by ID: {source_name} ID={source_id}")
|
||||
|
||||
# For torrent sources, must use remote client
|
||||
if source_name in ["RED", "OPS"]:
|
||||
if not self._remote_client:
|
||||
raise DownloadError(
|
||||
f"Cannot download from {source_name} without remote flacfetch service"
|
||||
)
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# For YouTube/Spotify, can use local flacfetch via FetchManager.download_by_id
|
||||
# But we currently only have remote support, so use remote if available
|
||||
if self._remote_client:
|
||||
return self._download_by_id_remote(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_dir=output_dir,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
|
||||
# Local download for YouTube/Spotify (fallback)
|
||||
# Use the local fetcher's download_by_id if available
|
||||
raise DownloadError(
|
||||
f"Local download_by_id not yet implemented for {source_name}. "
|
||||
"Configure FLACFETCH_API_URL for remote downloads."
|
||||
)
|
||||
|
||||
def _download_by_id_remote(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str, # Not used for remote downloads (files go to GCS or remote server)
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download by ID using the remote flacfetch service.
|
||||
|
||||
Note: output_dir is accepted for API compatibility but ignored for remote
|
||||
downloads. Remote downloads either go to GCS (if gcs_path is set) or to
|
||||
the remote server's download directory.
|
||||
"""
|
||||
# output_dir is intentionally unused - remote downloads don't use local paths
|
||||
_ = output_dir
|
||||
try:
|
||||
# Enable nested event loops
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download_by_id(
|
||||
source_name=source_name,
|
||||
source_id=source_id,
|
||||
output_filename=output_filename,
|
||||
target_file=target_file,
|
||||
download_url=download_url,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download by ID started: {download_id}")
|
||||
|
||||
# Wait for completion
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600,
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download by ID complete: {filepath}")
|
||||
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist="", # Not available without search
|
||||
title="", # Not available without search
|
||||
provider=source_name,
|
||||
quality="", # Not available without search
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download by ID error: {e}")
|
||||
raise DownloadError(f"Remote download by ID failed: {e}") from e
|
||||
|
||||
def _download_remote(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Download using the remote flacfetch service.
|
||||
|
||||
The remote service downloads via torrent and optionally uploads to GCS.
|
||||
|
||||
Args:
|
||||
result_index: Index of the result to download
|
||||
output_dir: Directory to save downloaded file
|
||||
output_filename: Optional filename
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
search_id: Remote search ID to use (overrides self._remote_search_id)
|
||||
"""
|
||||
effective_search_id = search_id or self._remote_search_id
|
||||
if not effective_search_id:
|
||||
raise DownloadError("No remote search ID - run search() first")
|
||||
|
||||
# Try to get local result for metadata, but it's optional
|
||||
# The remote service maintains its own cache by search_id
|
||||
result = None
|
||||
if result_index >= 0 and result_index < len(self._cached_results):
|
||||
result = self._cached_results[result_index]
|
||||
|
||||
try:
|
||||
# Enable nested event loops (needed when called from FastAPI async context)
|
||||
try:
|
||||
running_loop = asyncio.get_running_loop()
|
||||
nest_asyncio.apply(running_loop)
|
||||
except RuntimeError:
|
||||
pass # No running loop, nest_asyncio not needed
|
||||
|
||||
loop = asyncio.new_event_loop()
|
||||
asyncio.set_event_loop(loop)
|
||||
try:
|
||||
# Start download
|
||||
download_id = loop.run_until_complete(
|
||||
self._remote_client.download(
|
||||
search_id=effective_search_id,
|
||||
result_index=result_index,
|
||||
output_filename=output_filename,
|
||||
gcs_path=gcs_path,
|
||||
)
|
||||
)
|
||||
|
||||
logger.info(f"Remote download started: {download_id}")
|
||||
|
||||
# Wait for completion with progress logging
|
||||
def log_progress(status):
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
logger.debug(f"Download progress: {progress:.1f}% ({speed:.1f} KB/s)")
|
||||
|
||||
final_status = loop.run_until_complete(
|
||||
self._remote_client.wait_for_download(
|
||||
download_id,
|
||||
timeout=600, # 10 minute timeout
|
||||
progress_callback=log_progress,
|
||||
)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
# Determine file path
|
||||
filepath = final_status.get("gcs_path") or final_status.get("output_path")
|
||||
|
||||
if not filepath:
|
||||
raise DownloadError("Remote download completed but no file path returned")
|
||||
|
||||
logger.info(f"Remote download complete: {filepath}")
|
||||
|
||||
# Use local result metadata if available, otherwise use empty strings
|
||||
# (the actual download is handled by the remote service using search_id)
|
||||
return AudioDownloadResult(
|
||||
filepath=filepath,
|
||||
artist=result.artist if result else "",
|
||||
title=result.title if result else "",
|
||||
provider=result.provider if result else "remote",
|
||||
quality=result.quality if result else "",
|
||||
)
|
||||
|
||||
except FlacfetchServiceError as e:
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
except Exception as e:
|
||||
logger.error(f"Remote download error: {e}")
|
||||
raise DownloadError(f"Remote download failed: {e}") from e
|
||||
|
||||
def search_and_download_auto(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Search for audio and automatically download the best result.
|
||||
|
||||
This is a convenience method that combines search(), select_best(),
|
||||
and download() for automated/non-interactive usage.
|
||||
|
||||
Args:
|
||||
artist: The artist name to search for
|
||||
title: The track title to search for
|
||||
output_dir: Directory to save the downloaded file
|
||||
output_filename: Optional filename (without extension)
|
||||
gcs_path: Optional GCS path for remote uploads
|
||||
|
||||
Returns:
|
||||
AudioDownloadResult with the downloaded file path
|
||||
|
||||
Raises:
|
||||
NoResultsError: If no results are found
|
||||
DownloadError: If download fails
|
||||
"""
|
||||
# Search
|
||||
results = self.search(artist, title)
|
||||
|
||||
# Select best
|
||||
best_index = self.select_best(results)
|
||||
best_result = results[best_index]
|
||||
logger.info(
|
||||
f"Auto-selected result {best_index}: "
|
||||
f"{best_result.artist} - {best_result.title} ({best_result.provider})"
|
||||
)
|
||||
|
||||
# Download
|
||||
return self.download(best_index, output_dir, output_filename, gcs_path)
|
||||
|
||||
async def search_async(self, artist: str, title: str) -> List[AudioSearchResult]:
|
||||
"""
|
||||
Async version of search for use in async routes.
|
||||
|
||||
Note: Currently wraps sync search in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(None, self.search, artist, title)
|
||||
|
||||
async def download_async(
|
||||
self,
|
||||
result_index: int,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
remote_search_id: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download for use in async routes.
|
||||
|
||||
Note: Currently wraps sync download in executor. Future optimization
|
||||
could make this fully async.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download(result_index, output_dir, output_filename, gcs_path, remote_search_id)
|
||||
)
|
||||
|
||||
async def download_by_id_async(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_dir: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> AudioDownloadResult:
|
||||
"""
|
||||
Async version of download_by_id for use in async routes.
|
||||
"""
|
||||
loop = asyncio.get_event_loop()
|
||||
return await loop.run_in_executor(
|
||||
None,
|
||||
lambda: self.download_by_id(
|
||||
source_name, source_id, output_dir, output_filename,
|
||||
target_file, download_url, gcs_path
|
||||
)
|
||||
)
|
||||
|
||||
def is_remote_enabled(self) -> bool:
|
||||
"""Check if remote flacfetch service is configured."""
|
||||
return self._remote_client is not None
|
||||
|
||||
@property
|
||||
def last_remote_search_id(self) -> Optional[str]:
|
||||
"""Get the search_id from the last remote search.
|
||||
|
||||
This should be stored in job state_data after search and passed
|
||||
back to download() to ensure correct remote download handling.
|
||||
"""
|
||||
return self._remote_search_id
|
||||
|
||||
async def check_remote_health(self) -> Optional[dict]:
|
||||
"""
|
||||
Check health of remote flacfetch service.
|
||||
|
||||
Returns:
|
||||
Health status dict if remote service is configured and healthy,
|
||||
None if not configured or unhealthy.
|
||||
"""
|
||||
if not self._remote_client:
|
||||
return None
|
||||
|
||||
try:
|
||||
return await self._remote_client.health_check()
|
||||
except Exception as e:
|
||||
logger.warning(f"Remote flacfetch health check failed: {e}")
|
||||
return None
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_audio_search_service: Optional[AudioSearchService] = None
|
||||
|
||||
|
||||
def get_audio_search_service() -> AudioSearchService:
|
||||
"""Get the singleton AudioSearchService instance."""
|
||||
global _audio_search_service
|
||||
if _audio_search_service is None:
|
||||
_audio_search_service = AudioSearchService()
|
||||
return _audio_search_service
|
||||
@@ -1,630 +0,0 @@
|
||||
"""
|
||||
Authentication service for karaoke backend.
|
||||
|
||||
Provides token-based authentication with support for:
|
||||
- Admin tokens (hardcoded in environment)
|
||||
- Admin by email domain (@nomadkaraoke.com)
|
||||
- User tokens (stored in Firestore auth_tokens collection)
|
||||
- Session tokens (stored in Firestore sessions collection, from magic link auth)
|
||||
- API keys for business users
|
||||
- Usage tracking and limits
|
||||
- Token management API
|
||||
"""
|
||||
import logging
|
||||
import hashlib
|
||||
import time
|
||||
import secrets
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from typing import Optional, Tuple, Dict, Any, List
|
||||
from datetime import datetime
|
||||
|
||||
from backend.services.firestore_service import FirestoreService
|
||||
from backend.config import get_settings
|
||||
from backend.models.user import UserRole
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Admin email domains - users with these domains are automatically admin
|
||||
ADMIN_EMAIL_DOMAINS = ["nomadkaraoke.com"]
|
||||
|
||||
|
||||
class UserType(str, Enum):
|
||||
"""User access levels."""
|
||||
ADMIN = "admin" # Unlimited access, can manage tokens
|
||||
UNLIMITED = "unlimited" # Unlimited karaoke generation
|
||||
LIMITED = "limited" # Limited number of uses
|
||||
STRIPE = "stripe" # Paid access via Stripe
|
||||
API_KEY = "api_key" # Business API key access
|
||||
|
||||
|
||||
@dataclass
|
||||
class AuthResult:
|
||||
"""Result of authentication validation."""
|
||||
is_valid: bool
|
||||
user_type: UserType
|
||||
remaining_uses: int # -1 = unlimited, 0 = exhausted, >0 = remaining
|
||||
message: str
|
||||
user_email: Optional[str] = None # Email if authenticated via session/API key
|
||||
is_admin: bool = False # True if admin token or admin email domain
|
||||
api_key_id: Optional[str] = None # API key ID if authenticated via API key
|
||||
|
||||
def __iter__(self):
|
||||
"""Allow unpacking as tuple for backward compatibility."""
|
||||
return iter((self.is_valid, self.user_type, self.remaining_uses, self.message))
|
||||
|
||||
|
||||
def is_admin_email(email: str) -> bool:
|
||||
"""Check if an email belongs to an admin domain."""
|
||||
if not email:
|
||||
return False
|
||||
email_lower = email.lower()
|
||||
return any(email_lower.endswith(f"@{domain}") for domain in ADMIN_EMAIL_DOMAINS)
|
||||
|
||||
|
||||
class AuthService:
|
||||
"""Service for authentication and token management."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize auth service."""
|
||||
self.firestore = FirestoreService()
|
||||
self.settings = get_settings()
|
||||
self._init_admin_tokens()
|
||||
|
||||
def _init_admin_tokens(self):
|
||||
"""Load admin tokens from environment variables."""
|
||||
# Get admin tokens from secret manager or env vars
|
||||
admin_tokens_str = self.settings.admin_tokens or ""
|
||||
self.admin_tokens = [t.strip() for t in admin_tokens_str.split(",") if t.strip()]
|
||||
|
||||
if self.admin_tokens:
|
||||
logger.info(f"Loaded {len(self.admin_tokens)} admin token(s)")
|
||||
else:
|
||||
logger.warning("No admin tokens configured! Set ADMIN_TOKENS environment variable.")
|
||||
|
||||
def validate_token(self, token: str) -> Tuple[bool, UserType, int, str]:
|
||||
"""
|
||||
Validate an access token.
|
||||
|
||||
Returns:
|
||||
(is_valid, user_type, remaining_uses, message)
|
||||
remaining_uses: -1 = unlimited, 0 = exhausted, >0 = remaining count
|
||||
"""
|
||||
if not token:
|
||||
return False, UserType.LIMITED, 0, "No token provided"
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return True, UserType.ADMIN, -1, "Admin access granted"
|
||||
|
||||
# Debug: Log token comparison info (only first/last few chars for security)
|
||||
if self.admin_tokens:
|
||||
expected_prefix = self.admin_tokens[0][:8] if len(self.admin_tokens[0]) >= 8 else self.admin_tokens[0]
|
||||
provided_prefix = token[:8] if len(token) >= 8 else token
|
||||
logger.debug(
|
||||
f"Token mismatch: expected prefix '{expected_prefix}...', "
|
||||
f"got '{provided_prefix}...', "
|
||||
f"expected len={len(self.admin_tokens[0])}, got len={len(token)}"
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
# Import here to avoid circular dependency
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
# Return their credits as remaining uses
|
||||
credits = user.credits
|
||||
logger.info(f"Session token validated for user {user.email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
# User has no credits but is authenticated
|
||||
return True, UserType.STRIPE, 0, f"Authenticated but no credits remaining"
|
||||
|
||||
return True, UserType.STRIPE, credits, f"Session valid: {credits} credits"
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return False, UserType.LIMITED, 0, "Invalid token"
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return False, UserType(token_data["type"]), 0, "Token has been revoked"
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return True, token_type, -1, "Unlimited access granted"
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return True, token_type, -1, "Limited token with unlimited uses"
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Limited token: {remaining} uses remaining"
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at:
|
||||
# expires_at is stored as timestamp
|
||||
if time.time() > expires_at:
|
||||
return False, token_type, 0, "Token has expired"
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return False, token_type, 0, "Token usage limit exceeded"
|
||||
|
||||
return True, token_type, remaining, f"Stripe token: {remaining} uses remaining"
|
||||
|
||||
return True, token_type, -1, "Stripe access granted"
|
||||
|
||||
return False, UserType.LIMITED, 0, "Unknown token type"
|
||||
|
||||
def validate_token_full(self, token: str) -> AuthResult:
|
||||
"""
|
||||
Validate an access token and return full authentication details.
|
||||
|
||||
This is the preferred method for new code as it returns complete
|
||||
authentication context including user email and admin status.
|
||||
|
||||
Returns:
|
||||
AuthResult with full authentication details
|
||||
"""
|
||||
if not token:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="No token provided"
|
||||
)
|
||||
|
||||
# Check for admin tokens (highest priority)
|
||||
# Admin tokens from env var are associated with the system admin email
|
||||
if token in self.admin_tokens:
|
||||
logger.info("Admin token validated")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted",
|
||||
user_email="admin@nomadkaraoke.com",
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
# Check stored tokens in Firestore (auth_tokens collection)
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# Fall back to checking session tokens (from magic link auth)
|
||||
from backend.services.user_service import get_user_service
|
||||
|
||||
user_service = get_user_service()
|
||||
is_valid, user, _message = user_service.validate_session(token)
|
||||
|
||||
if is_valid and user:
|
||||
# Session is valid - user is authenticated via magic link
|
||||
credits = user.credits
|
||||
user_email = user.email
|
||||
|
||||
# Check if user is admin by email domain or role
|
||||
user_is_admin = is_admin_email(user_email) or user.role == UserRole.ADMIN
|
||||
|
||||
if user_is_admin:
|
||||
logger.info(f"Admin session validated for {user_email}")
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.ADMIN,
|
||||
remaining_uses=-1,
|
||||
message="Admin access granted (domain)",
|
||||
user_email=user_email,
|
||||
is_admin=True
|
||||
)
|
||||
|
||||
logger.info(f"Session token validated for user {user_email} (credits: {credits})")
|
||||
|
||||
if credits <= 0:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=0,
|
||||
message="Authenticated but no credits remaining",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=UserType.STRIPE,
|
||||
remaining_uses=credits,
|
||||
message=f"Session valid: {credits} credits",
|
||||
user_email=user_email,
|
||||
is_admin=False
|
||||
)
|
||||
|
||||
# Neither auth_token nor session found
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Invalid token"
|
||||
)
|
||||
|
||||
# Check if token is active
|
||||
if not token_data.get("active", True):
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType(token_data["type"]),
|
||||
remaining_uses=0,
|
||||
message="Token has been revoked"
|
||||
)
|
||||
|
||||
token_type = UserType(token_data["type"])
|
||||
max_uses = token_data.get("max_uses", -1)
|
||||
# All auth_tokens must have an associated user_email for job ownership
|
||||
token_user_email = token_data.get("user_email")
|
||||
api_key_id = token_data.get("api_key_id")
|
||||
|
||||
# Require user_email on all auth_tokens (no anonymous token auth)
|
||||
if not token_user_email:
|
||||
logger.warning("Auth token missing required user_email field")
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token configuration error: missing user_email. Please contact support."
|
||||
)
|
||||
|
||||
# Check if token's user is an admin (by email domain)
|
||||
token_is_admin = is_admin_email(token_user_email)
|
||||
|
||||
# UNLIMITED tokens: no usage limits
|
||||
if token_type == UserType.UNLIMITED:
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Unlimited access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# LIMITED tokens: check usage count
|
||||
if token_type == UserType.LIMITED:
|
||||
if max_uses <= 0: # -1 means unlimited
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Limited token with unlimited uses",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Limited token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# STRIPE tokens: check expiration and usage
|
||||
if token_type == UserType.STRIPE:
|
||||
expires_at = token_data.get("expires_at")
|
||||
|
||||
if expires_at and time.time() > expires_at:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token has expired",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="Token usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"Stripe token: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="Stripe access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
# API_KEY tokens
|
||||
if token_type == UserType.API_KEY:
|
||||
if max_uses > 0:
|
||||
current_uses = token_data.get("usage_count", 0)
|
||||
remaining = max_uses - current_uses
|
||||
if remaining <= 0:
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=token_type,
|
||||
remaining_uses=0,
|
||||
message="API key usage limit exceeded",
|
||||
user_email=token_user_email,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=remaining,
|
||||
message=f"API key valid: {remaining} uses remaining",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=True,
|
||||
user_type=token_type,
|
||||
remaining_uses=-1,
|
||||
message="API key access granted",
|
||||
user_email=token_user_email,
|
||||
is_admin=token_is_admin,
|
||||
api_key_id=api_key_id
|
||||
)
|
||||
|
||||
return AuthResult(
|
||||
is_valid=False,
|
||||
user_type=UserType.LIMITED,
|
||||
remaining_uses=0,
|
||||
message="Unknown token type"
|
||||
)
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> bool:
|
||||
"""
|
||||
Increment usage count for a token and track the job.
|
||||
|
||||
For session tokens (magic link auth), this deducts a credit from the user.
|
||||
For auth_tokens, this increments the usage count.
|
||||
|
||||
Args:
|
||||
token: The access token
|
||||
job_id: The job ID being created
|
||||
|
||||
Returns:
|
||||
True if usage was tracked, False otherwise
|
||||
"""
|
||||
# Validate token first
|
||||
is_valid, user_type, remaining_uses, message = self.validate_token(token)
|
||||
|
||||
if not is_valid:
|
||||
logger.warning(f"Cannot increment usage for invalid token: {message}")
|
||||
return False
|
||||
|
||||
# Don't track usage for admin or unlimited tokens
|
||||
if user_type in [UserType.ADMIN, UserType.UNLIMITED]:
|
||||
logger.debug(f"Skipping usage tracking for {user_type} token")
|
||||
return True
|
||||
|
||||
# For admin tokens (not in Firestore), no tracking needed
|
||||
if token in self.admin_tokens:
|
||||
return True
|
||||
|
||||
# Check if this is a session token (STRIPE type from validate_token means it's a session)
|
||||
# Session tokens are not in auth_tokens collection
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if not token_data:
|
||||
# This is a session token - deduct credit from user
|
||||
from backend.services.user_service import get_user_service
|
||||
user_service = get_user_service()
|
||||
|
||||
# Get the user email from the session
|
||||
is_valid, user, _ = user_service.validate_session(token)
|
||||
if not is_valid or not user:
|
||||
logger.error(f"Session token validation failed during usage increment")
|
||||
return False
|
||||
|
||||
# Deduct one credit
|
||||
success, new_balance, deduct_message = user_service.deduct_credit(
|
||||
user.email, job_id, reason="job_creation"
|
||||
)
|
||||
|
||||
if success:
|
||||
logger.info(f"Deducted credit for user {user.email} (remaining: {new_balance})")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to deduct credit for user {user.email}: {deduct_message}")
|
||||
return False
|
||||
|
||||
# Regular auth_token - increment usage in Firestore
|
||||
try:
|
||||
self.firestore.increment_token_usage(token, job_id)
|
||||
logger.info(f"Incremented usage for token (remaining: {remaining_uses - 1})")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to increment token usage: {e}")
|
||||
return False
|
||||
|
||||
def create_token(
|
||||
self,
|
||||
token_value: str,
|
||||
token_type: UserType,
|
||||
max_uses: int = -1,
|
||||
expires_at: Optional[float] = None,
|
||||
created_by: Optional[str] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Create a new access token.
|
||||
|
||||
Args:
|
||||
token_value: The actual token string (should be secure random string)
|
||||
token_type: Type of access (admin, unlimited, limited, stripe)
|
||||
max_uses: Maximum number of uses (-1 = unlimited)
|
||||
expires_at: Expiration timestamp (None = no expiration)
|
||||
created_by: Admin token that created this token
|
||||
|
||||
Returns:
|
||||
Token data dictionary
|
||||
"""
|
||||
token_data = {
|
||||
"token": token_value,
|
||||
"type": token_type.value,
|
||||
"max_uses": max_uses,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"created_at": time.time(),
|
||||
"created_by": created_by,
|
||||
"expires_at": expires_at,
|
||||
"jobs": []
|
||||
}
|
||||
|
||||
self.firestore.create_token(token_value, token_data)
|
||||
logger.info(f"Created new {token_type} token with max_uses={max_uses}")
|
||||
|
||||
return token_data
|
||||
|
||||
def revoke_token(self, token: str, revoked_by: Optional[str] = None) -> bool:
|
||||
"""
|
||||
Revoke an access token.
|
||||
|
||||
Args:
|
||||
token: The token to revoke
|
||||
revoked_by: Admin token that revoked this token
|
||||
|
||||
Returns:
|
||||
True if revoked, False if token not found
|
||||
"""
|
||||
try:
|
||||
self.firestore.update_token(token, {
|
||||
"active": False,
|
||||
"revoked_at": time.time(),
|
||||
"revoked_by": revoked_by
|
||||
})
|
||||
logger.info(f"Revoked token")
|
||||
return True
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to revoke token: {e}")
|
||||
return False
|
||||
|
||||
def list_tokens(self, include_inactive: bool = False) -> list:
|
||||
"""
|
||||
List all tokens (admin only).
|
||||
|
||||
Args:
|
||||
include_inactive: Whether to include revoked tokens
|
||||
|
||||
Returns:
|
||||
List of token data dictionaries (with sensitive data masked)
|
||||
"""
|
||||
try:
|
||||
tokens = self.firestore.list_tokens()
|
||||
|
||||
# Filter inactive if requested
|
||||
if not include_inactive:
|
||||
tokens = [t for t in tokens if t.get("active", True)]
|
||||
|
||||
# Mask token values for security (show only first 8 chars)
|
||||
for token in tokens:
|
||||
if "token" in token and len(token["token"]) > 8:
|
||||
token["token"] = token["token"][:8] + "..."
|
||||
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to list tokens: {e}")
|
||||
return []
|
||||
|
||||
def get_token_info(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Get information about a token (for admin dashboard).
|
||||
|
||||
Returns:
|
||||
Token data or None if not found
|
||||
"""
|
||||
# Check if it's an admin token (not stored in Firestore)
|
||||
if token in self.admin_tokens:
|
||||
return {
|
||||
"token": token[:8] + "...",
|
||||
"type": UserType.ADMIN.value,
|
||||
"max_uses": -1,
|
||||
"usage_count": 0,
|
||||
"active": True,
|
||||
"source": "environment"
|
||||
}
|
||||
|
||||
# Get from Firestore
|
||||
token_data = self.firestore.get_token(token)
|
||||
|
||||
if token_data and "token" in token_data:
|
||||
# Mask the full token value
|
||||
token_data["token"] = token_data["token"][:8] + "..."
|
||||
|
||||
return token_data
|
||||
|
||||
|
||||
# Global instance
|
||||
_auth_service = None
|
||||
|
||||
|
||||
def get_auth_service() -> AuthService:
|
||||
"""Get the global auth service instance."""
|
||||
global _auth_service
|
||||
if _auth_service is None:
|
||||
_auth_service = AuthService()
|
||||
return _auth_service
|
||||
|
||||
@@ -1,792 +0,0 @@
|
||||
"""
|
||||
OAuth Credential Manager for validating and refreshing credentials.
|
||||
|
||||
This service provides:
|
||||
1. Credential validation for YouTube, Dropbox, and Google Drive
|
||||
2. Device Authorization Flow for re-authentication
|
||||
3. Proactive monitoring with alerts when credentials expire
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Optional, Dict, Any, Tuple
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from datetime import datetime
|
||||
|
||||
from google.oauth2.credentials import Credentials as GoogleCredentials
|
||||
from google.auth.transport.requests import Request as GoogleAuthRequest
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CredentialStatus(str, Enum):
|
||||
"""Status of OAuth credentials."""
|
||||
VALID = "valid"
|
||||
EXPIRED = "expired" # Token expired but refresh may work
|
||||
INVALID = "invalid" # Refresh failed, re-auth needed
|
||||
NOT_CONFIGURED = "not_configured" # No credentials stored
|
||||
ERROR = "error" # Unknown error during validation
|
||||
|
||||
|
||||
@dataclass
|
||||
class CredentialCheckResult:
|
||||
"""Result of a credential validation check."""
|
||||
service: str
|
||||
status: CredentialStatus
|
||||
message: str
|
||||
last_checked: datetime
|
||||
expires_at: Optional[datetime] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class DeviceAuthInfo:
|
||||
"""Information for device authorization flow."""
|
||||
device_code: str
|
||||
user_code: str
|
||||
verification_url: str
|
||||
expires_in: int
|
||||
interval: int
|
||||
started_at: datetime
|
||||
|
||||
|
||||
class CredentialManager:
|
||||
"""
|
||||
Manages OAuth credentials for all external services.
|
||||
|
||||
Provides validation, refresh, and device authorization flow for:
|
||||
- YouTube (Google OAuth)
|
||||
- Google Drive (Google OAuth)
|
||||
- Dropbox
|
||||
"""
|
||||
|
||||
# Secret names in Secret Manager
|
||||
YOUTUBE_SECRET = "youtube-oauth-credentials"
|
||||
GDRIVE_SECRET = "gdrive-oauth-credentials"
|
||||
DROPBOX_SECRET = "dropbox-oauth-credentials"
|
||||
|
||||
# OAuth client credentials (for device auth flow)
|
||||
YOUTUBE_CLIENT_SECRET = "youtube-client-credentials"
|
||||
GDRIVE_CLIENT_SECRET = "gdrive-client-credentials"
|
||||
|
||||
# Google OAuth endpoints
|
||||
GOOGLE_TOKEN_URI = "https://oauth2.googleapis.com/token"
|
||||
GOOGLE_DEVICE_AUTH_URI = "https://oauth2.googleapis.com/device/code"
|
||||
|
||||
# Dropbox OAuth endpoints
|
||||
DROPBOX_TOKEN_URI = "https://api.dropboxapi.com/oauth2/token"
|
||||
|
||||
# Scopes for each service
|
||||
YOUTUBE_SCOPES = ["https://www.googleapis.com/auth/youtube"]
|
||||
GDRIVE_SCOPES = ["https://www.googleapis.com/auth/drive.file"]
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._pending_device_auths: Dict[str, DeviceAuthInfo] = {}
|
||||
|
||||
def _get_client_credentials(self, secret_name: str) -> Optional[Dict[str, str]]:
|
||||
"""
|
||||
Load OAuth client credentials from Secret Manager.
|
||||
|
||||
The secret should contain:
|
||||
{
|
||||
"client_id": "...",
|
||||
"client_secret": "..."
|
||||
}
|
||||
"""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
if not creds_json:
|
||||
return None
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
if creds.get("client_id") and creds.get("client_secret"):
|
||||
return creds
|
||||
|
||||
logger.warning(f"{secret_name} missing client_id or client_secret")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to load {secret_name}: {e}")
|
||||
return None
|
||||
|
||||
def get_youtube_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get YouTube OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.YOUTUBE_CLIENT_SECRET)
|
||||
|
||||
def get_gdrive_client_credentials(self) -> Optional[Dict[str, str]]:
|
||||
"""Get Google Drive OAuth client credentials from Secret Manager."""
|
||||
return self._get_client_credentials(self.GDRIVE_CLIENT_SECRET)
|
||||
|
||||
# =========================================================================
|
||||
# Credential Validation
|
||||
# =========================================================================
|
||||
|
||||
def check_all_credentials(self) -> Dict[str, CredentialCheckResult]:
|
||||
"""
|
||||
Check validity of all OAuth credentials.
|
||||
|
||||
Returns:
|
||||
Dictionary mapping service name to check result
|
||||
"""
|
||||
results = {}
|
||||
|
||||
results["youtube"] = self.check_youtube_credentials()
|
||||
results["gdrive"] = self.check_gdrive_credentials()
|
||||
results["dropbox"] = self.check_dropbox_credentials()
|
||||
|
||||
return results
|
||||
|
||||
def check_youtube_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if YouTube credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.YOUTUBE_SECRET,
|
||||
service_name="youtube",
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
test_api_call=self._test_youtube_api
|
||||
)
|
||||
|
||||
def check_gdrive_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Google Drive credentials are valid and can be refreshed."""
|
||||
return self._check_google_credentials(
|
||||
secret_name=self.GDRIVE_SECRET,
|
||||
service_name="gdrive",
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
test_api_call=self._test_gdrive_api
|
||||
)
|
||||
|
||||
def check_dropbox_credentials(self) -> CredentialCheckResult:
|
||||
"""Check if Dropbox credentials are valid."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(self.DROPBOX_SECRET)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message="Dropbox credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds = json.loads(creds_json)
|
||||
|
||||
if not creds.get("access_token"):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials missing access_token",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test the credentials with a simple API call
|
||||
if self._test_dropbox_api(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials are valid",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
else:
|
||||
# Try to refresh if we have refresh token
|
||||
if creds.get("refresh_token") and creds.get("app_key") and creds.get("app_secret"):
|
||||
if self._refresh_dropbox_token(creds):
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.VALID,
|
||||
message="Dropbox credentials refreshed successfully",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.INVALID,
|
||||
message="Dropbox credentials invalid and refresh failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking Dropbox credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service="dropbox",
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error checking credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _check_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
service_name: str,
|
||||
scopes: list,
|
||||
test_api_call: callable
|
||||
) -> CredentialCheckResult:
|
||||
"""Generic Google OAuth credential check."""
|
||||
try:
|
||||
creds_json = self.settings.get_secret(secret_name)
|
||||
|
||||
if not creds_json:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.NOT_CONFIGURED,
|
||||
message=f"{service_name} credentials not configured",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
creds_data = json.loads(creds_json)
|
||||
|
||||
# Check for required fields
|
||||
required = ["refresh_token", "client_id", "client_secret"]
|
||||
missing = [f for f in required if not creds_data.get(f)]
|
||||
if missing:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Missing required fields: {missing}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Create credentials object
|
||||
credentials = GoogleCredentials(
|
||||
token=creds_data.get("token"),
|
||||
refresh_token=creds_data.get("refresh_token"),
|
||||
token_uri=creds_data.get("token_uri", self.GOOGLE_TOKEN_URI),
|
||||
client_id=creds_data.get("client_id"),
|
||||
client_secret=creds_data.get("client_secret"),
|
||||
scopes=creds_data.get("scopes", scopes)
|
||||
)
|
||||
|
||||
# Try to refresh if expired
|
||||
if credentials.expired or not credentials.token:
|
||||
try:
|
||||
credentials.refresh(GoogleAuthRequest())
|
||||
# Update stored credentials with new token
|
||||
self._update_google_credentials(secret_name, creds_data, credentials)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to refresh {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Token refresh failed: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Test with API call
|
||||
if test_api_call(credentials):
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.VALID,
|
||||
message=f"{service_name} credentials are valid",
|
||||
last_checked=datetime.utcnow(),
|
||||
expires_at=credentials.expiry
|
||||
)
|
||||
else:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"{service_name} API test failed",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.INVALID,
|
||||
message=f"Invalid JSON in credentials: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking {service_name} credentials: {e}")
|
||||
return CredentialCheckResult(
|
||||
service=service_name,
|
||||
status=CredentialStatus.ERROR,
|
||||
message=f"Error: {str(e)}",
|
||||
last_checked=datetime.utcnow()
|
||||
)
|
||||
|
||||
def _update_google_credentials(
|
||||
self,
|
||||
secret_name: str,
|
||||
original_data: dict,
|
||||
credentials: GoogleCredentials
|
||||
) -> bool:
|
||||
"""Update stored Google credentials with refreshed token."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
updated_data = original_data.copy()
|
||||
updated_data["token"] = credentials.token
|
||||
if credentials.expiry:
|
||||
updated_data["expiry"] = credentials.expiry.isoformat()
|
||||
|
||||
# Add new secret version
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{secret_name}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(updated_data).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info(f"Updated {secret_name} with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update {secret_name}: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# API Test Methods
|
||||
# =========================================================================
|
||||
|
||||
def _test_youtube_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test YouTube credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
youtube = build("youtube", "v3", credentials=credentials)
|
||||
# Just get channel info - minimal API call
|
||||
request = youtube.channels().list(part="id", mine=True)
|
||||
response = request.execute()
|
||||
|
||||
return "items" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"YouTube API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_gdrive_api(self, credentials: GoogleCredentials) -> bool:
|
||||
"""Test Google Drive credentials with a simple API call."""
|
||||
try:
|
||||
from googleapiclient.discovery import build
|
||||
|
||||
drive = build("drive", "v3", credentials=credentials)
|
||||
# Just get about info - minimal API call
|
||||
request = drive.about().get(fields="user")
|
||||
response = request.execute()
|
||||
|
||||
return "user" in response
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Google Drive API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _test_dropbox_api(self, creds: dict) -> bool:
|
||||
"""Test Dropbox credentials with a simple API call."""
|
||||
try:
|
||||
import dropbox
|
||||
|
||||
dbx = dropbox.Dropbox(creds["access_token"])
|
||||
# Just get account info - minimal API call
|
||||
dbx.users_get_current_account()
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox API test failed: {e}")
|
||||
return False
|
||||
|
||||
def _refresh_dropbox_token(self, creds: dict) -> bool:
|
||||
"""Try to refresh Dropbox token."""
|
||||
try:
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
self.DROPBOX_TOKEN_URI,
|
||||
data={
|
||||
"grant_type": "refresh_token",
|
||||
"refresh_token": creds["refresh_token"],
|
||||
"client_id": creds["app_key"],
|
||||
"client_secret": creds["app_secret"],
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
token_data = response.json()
|
||||
creds["access_token"] = token_data["access_token"]
|
||||
|
||||
# Update in Secret Manager
|
||||
self._update_dropbox_credentials(creds)
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Dropbox token refresh failed: {e}")
|
||||
return False
|
||||
|
||||
def _update_dropbox_credentials(self, creds: dict) -> bool:
|
||||
"""Update stored Dropbox credentials."""
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
secret_path = f"projects/{project_id}/secrets/{self.DROPBOX_SECRET}"
|
||||
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(creds).encode("utf-8")}
|
||||
)
|
||||
|
||||
logger.info("Updated Dropbox credentials with refreshed token")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update Dropbox credentials: {e}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Device Authorization Flow
|
||||
# =========================================================================
|
||||
|
||||
def start_youtube_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start YouTube device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_youtube_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"YouTube client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'youtube-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.YOUTUBE_SCOPES,
|
||||
service_name="youtube"
|
||||
)
|
||||
|
||||
def start_gdrive_device_auth(
|
||||
self,
|
||||
client_id: Optional[str] = None,
|
||||
client_secret: Optional[str] = None
|
||||
) -> DeviceAuthInfo:
|
||||
"""
|
||||
Start Google Drive device authorization flow.
|
||||
|
||||
Args:
|
||||
client_id: Google OAuth client ID (optional, reads from Secret Manager if not provided)
|
||||
client_secret: Google OAuth client secret (optional, reads from Secret Manager if not provided)
|
||||
|
||||
Returns:
|
||||
DeviceAuthInfo with user code and verification URL
|
||||
"""
|
||||
# Load from Secret Manager if not provided
|
||||
if not client_id or not client_secret:
|
||||
stored_creds = self.get_gdrive_client_credentials()
|
||||
if not stored_creds:
|
||||
raise Exception(
|
||||
"Google Drive client credentials not found. Either pass client_id/client_secret "
|
||||
"or create the 'gdrive-client-credentials' secret in Secret Manager."
|
||||
)
|
||||
client_id = stored_creds["client_id"]
|
||||
client_secret = stored_creds["client_secret"]
|
||||
|
||||
return self._start_google_device_auth(
|
||||
client_id=client_id,
|
||||
client_secret=client_secret,
|
||||
scopes=self.GDRIVE_SCOPES,
|
||||
service_name="gdrive"
|
||||
)
|
||||
|
||||
def _start_google_device_auth(
|
||||
self,
|
||||
client_id: str,
|
||||
client_secret: str,
|
||||
scopes: list,
|
||||
service_name: str
|
||||
) -> DeviceAuthInfo:
|
||||
"""Start Google device authorization flow."""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Starting device auth flow with scopes: {scopes}")
|
||||
|
||||
response = requests.post(
|
||||
self.GOOGLE_DEVICE_AUTH_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"scope": " ".join(scopes),
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.error(f"[{service_name}] Device auth request failed: {response.status_code} - {response.text}")
|
||||
raise Exception(f"Device auth request failed: {response.text}")
|
||||
|
||||
logger.info(f"[{service_name}] Device auth initiated successfully")
|
||||
|
||||
data = response.json()
|
||||
|
||||
# Google uses 'verification_uri' but some docs show 'verification_url'
|
||||
verification_url = data.get("verification_uri") or data.get("verification_url")
|
||||
|
||||
device_info = DeviceAuthInfo(
|
||||
device_code=data["device_code"],
|
||||
user_code=data["user_code"],
|
||||
verification_url=verification_url,
|
||||
expires_in=data["expires_in"],
|
||||
interval=data.get("interval", 5),
|
||||
started_at=datetime.utcnow()
|
||||
)
|
||||
|
||||
# Store for polling, include client secret for token exchange
|
||||
self._pending_device_auths[f"{service_name}:{data['device_code']}"] = {
|
||||
"info": device_info,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
"service_name": service_name
|
||||
}
|
||||
|
||||
return device_info
|
||||
|
||||
def poll_device_auth(self, service_name: str, device_code: str) -> Tuple[str, Optional[dict]]:
|
||||
"""
|
||||
Poll for device authorization completion.
|
||||
|
||||
This method is STATELESS - it fetches client credentials from Secret Manager
|
||||
and polls Google directly. This works correctly in serverless environments
|
||||
like Cloud Run where in-memory state is not preserved between requests.
|
||||
|
||||
Args:
|
||||
service_name: "youtube" or "gdrive"
|
||||
device_code: The device code from start_*_device_auth
|
||||
|
||||
Returns:
|
||||
Tuple of (status, token_data)
|
||||
status: "pending", "complete", "expired", "error"
|
||||
token_data: Token data if complete, None otherwise
|
||||
"""
|
||||
import requests
|
||||
|
||||
logger.info(f"[{service_name}] Polling device auth for code: {device_code[:20]}...")
|
||||
|
||||
# Get client credentials from Secret Manager (stateless approach)
|
||||
if service_name == "youtube":
|
||||
client_creds = self.get_youtube_client_credentials()
|
||||
scopes = self.YOUTUBE_SCOPES
|
||||
secret_name = self.YOUTUBE_SECRET
|
||||
elif service_name == "gdrive":
|
||||
client_creds = self.get_gdrive_client_credentials()
|
||||
scopes = self.GDRIVE_SCOPES
|
||||
secret_name = self.GDRIVE_SECRET
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unknown service")
|
||||
return ("error", {"message": f"Unknown service: {service_name}"})
|
||||
|
||||
if not client_creds:
|
||||
logger.error(f"[{service_name}] Client credentials not found in Secret Manager")
|
||||
return ("error", {"message": f"Client credentials not found in Secret Manager for {service_name}"})
|
||||
|
||||
client_id = client_creds.get("client_id")
|
||||
client_secret = client_creds.get("client_secret")
|
||||
|
||||
if not client_id or not client_secret:
|
||||
logger.error(f"[{service_name}] Invalid client credentials (missing client_id or client_secret)")
|
||||
return ("error", {"message": f"Invalid client credentials for {service_name}"})
|
||||
|
||||
logger.info(f"[{service_name}] Got client credentials, polling Google token endpoint...")
|
||||
|
||||
# Poll Google token endpoint
|
||||
response = requests.post(
|
||||
self.GOOGLE_TOKEN_URI,
|
||||
data={
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"device_code": device_code,
|
||||
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
|
||||
data = response.json()
|
||||
logger.info(f"[{service_name}] Token endpoint response: status={response.status_code}")
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"[{service_name}] Token exchange successful! Got access token and refresh_token={bool(data.get('refresh_token'))}")
|
||||
|
||||
# Success! Store the credentials
|
||||
token_data = {
|
||||
"token": data["access_token"],
|
||||
"refresh_token": data.get("refresh_token"),
|
||||
"token_uri": self.GOOGLE_TOKEN_URI,
|
||||
"client_id": client_id,
|
||||
"client_secret": client_secret,
|
||||
"scopes": scopes,
|
||||
}
|
||||
|
||||
# Save to Secret Manager
|
||||
logger.info(f"[{service_name}] Saving credentials to secret: {secret_name}")
|
||||
saved = self._save_credentials_to_secret(secret_name, token_data)
|
||||
if not saved:
|
||||
logger.error(f"[{service_name}] Failed to save credentials to Secret Manager!")
|
||||
return ("error", {"message": "Token exchange succeeded but failed to save credentials to Secret Manager"})
|
||||
|
||||
logger.info(f"[{service_name}] Device auth flow COMPLETE - credentials saved successfully")
|
||||
return ("complete", token_data)
|
||||
|
||||
elif "error" in data:
|
||||
error = data["error"]
|
||||
logger.info(f"[{service_name}] Token endpoint returned error: {error}")
|
||||
|
||||
if error == "authorization_pending":
|
||||
return ("pending", {"message": "Waiting for user to authorize. Please visit the verification URL and enter the code."})
|
||||
elif error == "slow_down":
|
||||
return ("pending", {"message": "Please wait a few more seconds before polling again."})
|
||||
elif error == "expired_token":
|
||||
logger.warning(f"[{service_name}] Device code expired")
|
||||
return ("expired", {"message": "Device code expired. Please start a new device auth flow."})
|
||||
elif error == "access_denied":
|
||||
logger.warning(f"[{service_name}] User denied access")
|
||||
return ("error", {"message": "User denied access"})
|
||||
else:
|
||||
logger.error(f"[{service_name}] Unexpected error: {data}")
|
||||
return ("error", {"message": data.get("error_description", error)})
|
||||
|
||||
logger.error(f"[{service_name}] Unknown response from token endpoint: {data}")
|
||||
return ("error", {"message": "Unknown response from token endpoint"})
|
||||
|
||||
def _save_credentials_to_secret(self, secret_name: str, token_data: dict) -> bool:
|
||||
"""Save credentials to Secret Manager, creating the secret if needed."""
|
||||
logger.info(f"[SecretManager] Attempting to save credentials to: {secret_name}")
|
||||
|
||||
try:
|
||||
from google.cloud import secretmanager
|
||||
from google.api_core import exceptions as gcp_exceptions
|
||||
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
project_id = self.settings.google_cloud_project
|
||||
|
||||
if not project_id:
|
||||
logger.error(f"[SecretManager] No GCP project configured!")
|
||||
return False
|
||||
|
||||
parent = f"projects/{project_id}"
|
||||
secret_path = f"{parent}/secrets/{secret_name}"
|
||||
|
||||
logger.info(f"[SecretManager] Using project: {project_id}, secret path: {secret_path}")
|
||||
|
||||
# Try to add a version to existing secret
|
||||
try:
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Added new version to existing secret: {secret_name}")
|
||||
return True
|
||||
|
||||
except gcp_exceptions.NotFound:
|
||||
# Secret doesn't exist, create it first
|
||||
logger.info(f"[SecretManager] Secret {secret_name} not found, creating new secret...")
|
||||
client.create_secret(
|
||||
parent=parent,
|
||||
secret_id=secret_name,
|
||||
secret={"replication": {"automatic": {}}}
|
||||
)
|
||||
logger.info(f"[SecretManager] Created secret: {secret_name}")
|
||||
|
||||
# Now add the version
|
||||
client.add_secret_version(
|
||||
parent=secret_path,
|
||||
payload={"data": json.dumps(token_data).encode("utf-8")}
|
||||
)
|
||||
logger.info(f"[SecretManager] SUCCESS - Created secret and saved credentials to: {secret_name}")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[SecretManager] FAILED to save credentials to {secret_name}: {type(e).__name__}: {e}")
|
||||
import traceback
|
||||
logger.error(f"[SecretManager] Traceback: {traceback.format_exc()}")
|
||||
return False
|
||||
|
||||
# =========================================================================
|
||||
# Alerts
|
||||
# =========================================================================
|
||||
|
||||
def send_credential_alert(
|
||||
self,
|
||||
invalid_services: list[CredentialCheckResult],
|
||||
discord_webhook_url: Optional[str] = None
|
||||
) -> bool:
|
||||
"""
|
||||
Send alert about invalid credentials.
|
||||
|
||||
Args:
|
||||
invalid_services: List of services with invalid credentials
|
||||
discord_webhook_url: Discord webhook URL for notifications
|
||||
|
||||
Returns:
|
||||
True if alert was sent successfully
|
||||
"""
|
||||
if not discord_webhook_url:
|
||||
logger.warning("No Discord webhook URL configured for alerts")
|
||||
return False
|
||||
|
||||
try:
|
||||
import requests
|
||||
|
||||
services_list = "\n".join([
|
||||
f"• **{r.service}**: {r.message}"
|
||||
for r in invalid_services
|
||||
])
|
||||
|
||||
# Get the API base URL for re-auth links
|
||||
api_url = self.settings.api_base_url if hasattr(self.settings, 'api_base_url') else "https://your-api-url"
|
||||
|
||||
message = {
|
||||
"embeds": [{
|
||||
"title": "⚠️ OAuth Credentials Need Attention",
|
||||
"description": f"The following service credentials need re-authorization:\n\n{services_list}",
|
||||
"color": 16744256, # Orange
|
||||
"fields": [
|
||||
{
|
||||
"name": "Re-authorize",
|
||||
"value": f"Visit `{api_url}/api/auth/status` to start re-authorization flow",
|
||||
"inline": False
|
||||
}
|
||||
],
|
||||
"footer": {
|
||||
"text": "karaoke-gen backend"
|
||||
},
|
||||
"timestamp": datetime.utcnow().isoformat()
|
||||
}]
|
||||
}
|
||||
|
||||
response = requests.post(discord_webhook_url, json=message, timeout=30)
|
||||
return response.status_code in (200, 204)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to send credential alert: {e}")
|
||||
return False
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_credential_manager: Optional[CredentialManager] = None
|
||||
|
||||
|
||||
def get_credential_manager() -> CredentialManager:
|
||||
"""Get the singleton credential manager instance."""
|
||||
global _credential_manager
|
||||
if _credential_manager is None:
|
||||
_credential_manager = CredentialManager()
|
||||
return _credential_manager
|
||||
@@ -1,172 +0,0 @@
|
||||
"""
|
||||
Discord Notification Service.
|
||||
|
||||
Provides Discord webhook notification functionality, extracted from KaraokeFinalise
|
||||
for use by both the cloud backend (video_worker) and local CLI.
|
||||
|
||||
This service handles:
|
||||
- Posting messages to Discord webhooks
|
||||
- Video upload notifications
|
||||
- Validation of webhook URLs
|
||||
"""
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DiscordNotificationService:
|
||||
"""
|
||||
Service for sending Discord webhook notifications.
|
||||
|
||||
Supports posting messages to Discord channels via webhooks,
|
||||
commonly used to notify when new karaoke videos are uploaded.
|
||||
"""
|
||||
|
||||
# Discord webhook URL pattern
|
||||
WEBHOOK_URL_PATTERN = re.compile(
|
||||
r"^https://discord\.com/api/webhooks/\d+/[a-zA-Z0-9_-]+$"
|
||||
)
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
webhook_url: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the Discord notification service.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for posting notifications
|
||||
dry_run: If True, log actions without performing them
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.webhook_url = webhook_url
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
|
||||
# Validate webhook URL if provided
|
||||
if webhook_url:
|
||||
self._validate_webhook_url(webhook_url)
|
||||
|
||||
def _validate_webhook_url(self, url: str) -> None:
|
||||
"""
|
||||
Validate that the webhook URL is a valid Discord webhook URL.
|
||||
|
||||
Args:
|
||||
url: URL to validate
|
||||
|
||||
Raises:
|
||||
ValueError: If URL is not a valid Discord webhook URL
|
||||
"""
|
||||
url = url.strip()
|
||||
if not url.startswith("https://discord.com/api/webhooks/"):
|
||||
raise ValueError(
|
||||
f"Invalid Discord webhook URL: {url}. "
|
||||
"URL must start with 'https://discord.com/api/webhooks/'"
|
||||
)
|
||||
|
||||
def post_message(
|
||||
self,
|
||||
message: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a message to a Discord channel via webhook.
|
||||
|
||||
Args:
|
||||
message: The message content to post
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if message was posted successfully, False otherwise
|
||||
|
||||
Raises:
|
||||
ValueError: If no webhook URL is available
|
||||
requests.HTTPError: If the webhook request fails
|
||||
"""
|
||||
url = webhook_url or self.webhook_url
|
||||
if not url:
|
||||
raise ValueError("No Discord webhook URL provided")
|
||||
|
||||
url = url.strip()
|
||||
self._validate_webhook_url(url)
|
||||
|
||||
if self.dry_run:
|
||||
self.logger.info(
|
||||
f"DRY RUN: Would post Discord message: '{message}' to {url}"
|
||||
)
|
||||
return True
|
||||
|
||||
self.logger.info(f"Posting message to Discord webhook...")
|
||||
data = {"content": message}
|
||||
response = requests.post(url, json=data, timeout=30)
|
||||
response.raise_for_status()
|
||||
self.logger.info("Message posted to Discord successfully")
|
||||
return True
|
||||
|
||||
def post_video_notification(
|
||||
self,
|
||||
youtube_url: str,
|
||||
webhook_url: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""
|
||||
Post a notification about a new video upload.
|
||||
|
||||
Args:
|
||||
youtube_url: The YouTube URL of the uploaded video
|
||||
webhook_url: Optional webhook URL (uses instance URL if not provided)
|
||||
|
||||
Returns:
|
||||
True if notification was posted successfully, False otherwise
|
||||
"""
|
||||
if not youtube_url:
|
||||
self.logger.info("Skipping Discord notification - no YouTube URL available")
|
||||
return False
|
||||
|
||||
message = f"New upload: {youtube_url}"
|
||||
return self.post_message(message, webhook_url)
|
||||
|
||||
def is_enabled(self) -> bool:
|
||||
"""
|
||||
Check if Discord notifications are enabled.
|
||||
|
||||
Returns:
|
||||
True if a valid webhook URL is configured
|
||||
"""
|
||||
return bool(self.webhook_url)
|
||||
|
||||
|
||||
# Singleton instance and factory function (following existing service pattern)
|
||||
_discord_notification_service: Optional[DiscordNotificationService] = None
|
||||
|
||||
|
||||
def get_discord_notification_service(
|
||||
webhook_url: Optional[str] = None,
|
||||
**kwargs
|
||||
) -> DiscordNotificationService:
|
||||
"""
|
||||
Get a Discord notification service instance.
|
||||
|
||||
Args:
|
||||
webhook_url: Discord webhook URL for notifications
|
||||
**kwargs: Additional arguments passed to DiscordNotificationService
|
||||
|
||||
Returns:
|
||||
DiscordNotificationService instance
|
||||
"""
|
||||
global _discord_notification_service
|
||||
|
||||
# Create new instance if webhook URL changed or not yet created
|
||||
if _discord_notification_service is None or webhook_url:
|
||||
_discord_notification_service = DiscordNotificationService(
|
||||
webhook_url=webhook_url,
|
||||
**kwargs
|
||||
)
|
||||
|
||||
return _discord_notification_service
|
||||
@@ -1,313 +0,0 @@
|
||||
"""Native Dropbox API service for cloud backend.
|
||||
|
||||
This service provides Dropbox operations using the native Python SDK,
|
||||
replacing rclone for server-side operations. It handles:
|
||||
- Folder listing for brand code calculation
|
||||
- File and folder uploads
|
||||
- Shared link generation
|
||||
|
||||
Credentials are loaded from Google Cloud Secret Manager.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
from google.cloud import secretmanager
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DropboxService:
|
||||
"""Dropbox operations using native Python SDK."""
|
||||
|
||||
def __init__(self):
|
||||
self._client = None
|
||||
self._is_configured = False
|
||||
|
||||
def _load_credentials(self) -> Optional[dict]:
|
||||
"""Load OAuth credentials from Secret Manager."""
|
||||
try:
|
||||
client = secretmanager.SecretManagerServiceClient()
|
||||
# Try to get the project ID from environment or use default
|
||||
project_id = os.environ.get("GOOGLE_CLOUD_PROJECT", "karaoke-gen")
|
||||
name = f"projects/{project_id}/secrets/dropbox-oauth-credentials/versions/latest"
|
||||
response = client.access_secret_version(request={"name": name})
|
||||
return json.loads(response.payload.data.decode("UTF-8"))
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to load Dropbox credentials from Secret Manager: {e}")
|
||||
return None
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if Dropbox credentials are available."""
|
||||
if self._is_configured:
|
||||
return True
|
||||
creds = self._load_credentials()
|
||||
self._is_configured = creds is not None and "access_token" in creds
|
||||
return self._is_configured
|
||||
|
||||
@property
|
||||
def client(self):
|
||||
"""Get or create Dropbox client."""
|
||||
if self._client is None:
|
||||
# Import here to avoid import errors if dropbox is not installed
|
||||
try:
|
||||
from dropbox import Dropbox
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
"dropbox package is not installed. "
|
||||
"Install it with: pip install dropbox"
|
||||
)
|
||||
|
||||
creds = self._load_credentials()
|
||||
if not creds:
|
||||
raise RuntimeError("Dropbox credentials not configured in Secret Manager")
|
||||
|
||||
self._client = Dropbox(
|
||||
oauth2_access_token=creds["access_token"],
|
||||
oauth2_refresh_token=creds.get("refresh_token"),
|
||||
app_key=creds.get("app_key"),
|
||||
app_secret=creds.get("app_secret"),
|
||||
)
|
||||
return self._client
|
||||
|
||||
def list_folders(self, path: str) -> list[str]:
|
||||
"""
|
||||
List folder names at path for brand code calculation.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to list (e.g., "/Karaoke/Tracks-Organized")
|
||||
|
||||
Returns:
|
||||
List of folder names in the path
|
||||
"""
|
||||
from dropbox.files import FolderMetadata
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Listing folders at Dropbox path: {path}")
|
||||
|
||||
result = self.client.files_list_folder(path)
|
||||
folders = []
|
||||
|
||||
# Get all entries (handling pagination)
|
||||
while True:
|
||||
for entry in result.entries:
|
||||
if isinstance(entry, FolderMetadata):
|
||||
folders.append(entry.name)
|
||||
|
||||
if not result.has_more:
|
||||
break
|
||||
result = self.client.files_list_folder_continue(result.cursor)
|
||||
|
||||
logger.info(f"Found {len(folders)} folders in {path}")
|
||||
return folders
|
||||
|
||||
def get_next_brand_code(self, path: str, brand_prefix: str) -> str:
|
||||
"""
|
||||
Calculate next brand code from existing folders, filling gaps in the sequence.
|
||||
|
||||
Scans the folder for existing brand codes like "NOMAD-0001" and
|
||||
returns the first available code, filling any gaps left by deleted outputs.
|
||||
|
||||
Args:
|
||||
path: Dropbox path containing organized folders
|
||||
brand_prefix: Brand prefix (e.g., "NOMAD")
|
||||
|
||||
Returns:
|
||||
Next brand code (e.g., "NOMAD-0003" if 0003 is the first gap)
|
||||
"""
|
||||
folders = self.list_folders(path)
|
||||
pattern = re.compile(rf"^{re.escape(brand_prefix)}-(\d{{4}})")
|
||||
|
||||
existing_nums: set[int] = set()
|
||||
for folder in folders:
|
||||
match = pattern.match(folder)
|
||||
if match:
|
||||
existing_nums.add(int(match.group(1)))
|
||||
|
||||
max_existing = max(existing_nums) if existing_nums else 0
|
||||
|
||||
# Only fill gaps for numbers >= 1001 (preserve legacy gaps below 1000)
|
||||
if max_existing >= 1000:
|
||||
next_num = 1001
|
||||
while next_num in existing_nums:
|
||||
next_num += 1
|
||||
else:
|
||||
next_num = max_existing + 1
|
||||
|
||||
next_code = f"{brand_prefix}-{next_num:04d}"
|
||||
if max_existing >= 1000 and next_num <= max_existing:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (filling gap, max existing: {max_existing})")
|
||||
else:
|
||||
logger.info(f"Next brand code for {brand_prefix}: {next_code} (max existing: {max_existing})")
|
||||
return next_code
|
||||
|
||||
def upload_file(self, local_path: str, remote_path: str) -> None:
|
||||
"""
|
||||
Upload a single file to Dropbox.
|
||||
|
||||
Args:
|
||||
local_path: Local file path
|
||||
remote_path: Dropbox destination path (must start with /)
|
||||
"""
|
||||
from dropbox.files import WriteMode
|
||||
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
file_size = os.path.getsize(local_path)
|
||||
logger.info(f"Uploading {local_path} ({file_size / 1024 / 1024:.1f} MB) to {remote_path}")
|
||||
|
||||
# For large files (>150MB), use upload sessions
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
with open(local_path, "rb") as f:
|
||||
if file_size <= CHUNK_SIZE:
|
||||
# Simple upload for smaller files
|
||||
self.client.files_upload(f.read(), remote_path, mode=WriteMode.overwrite)
|
||||
else:
|
||||
# Chunked upload for large files
|
||||
self._upload_large_file(f, remote_path, file_size)
|
||||
|
||||
logger.info(f"Successfully uploaded to {remote_path}")
|
||||
|
||||
def _upload_large_file(self, file_obj, remote_path: str, file_size: int) -> None:
|
||||
"""Upload a large file using upload sessions."""
|
||||
from dropbox.files import CommitInfo, UploadSessionCursor, WriteMode
|
||||
|
||||
CHUNK_SIZE = 150 * 1024 * 1024 # 150 MB
|
||||
|
||||
# Start upload session
|
||||
session_start = self.client.files_upload_session_start(file_obj.read(CHUNK_SIZE))
|
||||
cursor = UploadSessionCursor(
|
||||
session_id=session_start.session_id,
|
||||
offset=file_obj.tell(),
|
||||
)
|
||||
commit = CommitInfo(path=remote_path, mode=WriteMode.overwrite)
|
||||
|
||||
# Upload remaining chunks
|
||||
while file_obj.tell() < file_size:
|
||||
remaining = file_size - file_obj.tell()
|
||||
if remaining <= CHUNK_SIZE:
|
||||
# Final chunk
|
||||
self.client.files_upload_session_finish(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
commit,
|
||||
)
|
||||
else:
|
||||
# Intermediate chunk
|
||||
self.client.files_upload_session_append_v2(
|
||||
file_obj.read(CHUNK_SIZE),
|
||||
cursor,
|
||||
)
|
||||
cursor.offset = file_obj.tell()
|
||||
|
||||
def upload_folder(self, local_dir: str, remote_path: str) -> None:
|
||||
"""
|
||||
Recursively upload all files and subdirectories to Dropbox folder.
|
||||
|
||||
Args:
|
||||
local_dir: Local directory to upload
|
||||
remote_path: Dropbox destination folder path
|
||||
"""
|
||||
# Ensure remote path starts with /
|
||||
if not remote_path.startswith("/"):
|
||||
remote_path = f"/{remote_path}"
|
||||
|
||||
logger.info(f"Uploading folder {local_dir} to {remote_path}")
|
||||
|
||||
uploaded_count = 0
|
||||
for root, _dirs, files in os.walk(local_dir):
|
||||
# Calculate the relative path from local_dir to current root
|
||||
rel_root = os.path.relpath(root, local_dir)
|
||||
if rel_root == ".":
|
||||
current_remote = remote_path
|
||||
else:
|
||||
current_remote = f"{remote_path}/{rel_root}"
|
||||
|
||||
for filename in files:
|
||||
local_file = os.path.join(root, filename)
|
||||
remote_file = f"{current_remote}/{filename}"
|
||||
self.upload_file(local_file, remote_file)
|
||||
uploaded_count += 1
|
||||
|
||||
logger.info(f"Uploaded {uploaded_count} files to {remote_path}")
|
||||
|
||||
def create_shared_link(self, path: str) -> str:
|
||||
"""
|
||||
Create and return a shared link for the path.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to share
|
||||
|
||||
Returns:
|
||||
Shared link URL
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
from dropbox.sharing import SharedLinkSettings
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Creating shared link for: {path}")
|
||||
|
||||
try:
|
||||
# Try to create a new shared link
|
||||
settings = SharedLinkSettings(requested_visibility=None)
|
||||
link = self.client.sharing_create_shared_link_with_settings(path, settings)
|
||||
logger.info(f"Created new shared link: {link.url}")
|
||||
return link.url
|
||||
except ApiError as e:
|
||||
# Link may already exist - try to get existing link
|
||||
if e.error.is_shared_link_already_exists():
|
||||
logger.info("Shared link already exists, retrieving existing link")
|
||||
links = self.client.sharing_list_shared_links(path=path, direct_only=True)
|
||||
if links.links:
|
||||
logger.info(f"Found existing shared link: {links.links[0].url}")
|
||||
return links.links[0].url
|
||||
raise
|
||||
|
||||
def delete_folder(self, path: str) -> bool:
|
||||
"""
|
||||
Delete a folder and all its contents from Dropbox.
|
||||
|
||||
Args:
|
||||
path: Dropbox path to delete (e.g., "/Karaoke/Tracks-Organized/NOMAD-1234 - Artist - Title")
|
||||
|
||||
Returns:
|
||||
True if deleted successfully, False otherwise
|
||||
"""
|
||||
from dropbox.exceptions import ApiError
|
||||
|
||||
# Ensure path starts with /
|
||||
if not path.startswith("/"):
|
||||
path = f"/{path}"
|
||||
|
||||
logger.info(f"Deleting Dropbox folder: {path}")
|
||||
|
||||
try:
|
||||
self.client.files_delete_v2(path)
|
||||
logger.info(f"Successfully deleted: {path}")
|
||||
return True
|
||||
except ApiError as e:
|
||||
if e.error.is_path_lookup() and e.error.get_path_lookup().is_not_found():
|
||||
logger.warning(f"Folder not found (already deleted?): {path}")
|
||||
return True # Consider it success if already gone
|
||||
logger.error(f"Failed to delete Dropbox folder: {e}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting Dropbox folder: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def get_dropbox_service() -> DropboxService:
|
||||
"""Factory function to get a DropboxService instance."""
|
||||
return DropboxService()
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,646 +0,0 @@
|
||||
"""
|
||||
Email Validation Service.
|
||||
|
||||
Provides email validation, normalization, and blocklist checking
|
||||
to prevent abuse during beta enrollment and other flows.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional, Tuple, Set
|
||||
|
||||
from google.cloud import firestore
|
||||
|
||||
from backend.config import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _mask_email(email: str) -> str:
|
||||
"""Mask an email address for privacy-safe logging."""
|
||||
if not email or "@" not in email:
|
||||
return "***"
|
||||
local, domain = email.split("@", 1)
|
||||
if len(local) <= 2:
|
||||
masked_local = "*" * len(local)
|
||||
else:
|
||||
masked_local = local[0] + "*" * (len(local) - 2) + local[-1]
|
||||
return f"{masked_local}@{domain}"
|
||||
|
||||
|
||||
# Firestore collection for blocklist data
|
||||
BLOCKLISTS_COLLECTION = "blocklists"
|
||||
BLOCKLIST_CONFIG_DOC = "config"
|
||||
|
||||
# Default disposable email domains (can be extended via admin UI)
|
||||
DEFAULT_DISPOSABLE_DOMAINS = {
|
||||
# Popular temporary email services
|
||||
"10minutemail.com",
|
||||
"10minutemail.net",
|
||||
"guerrillamail.com",
|
||||
"guerrillamail.org",
|
||||
"guerrillamail.net",
|
||||
"guerrillamail.biz",
|
||||
"guerrillamailblock.com",
|
||||
"tempmail.com",
|
||||
"tempmail.net",
|
||||
"temp-mail.org",
|
||||
"temp-mail.io",
|
||||
"throwawaymail.com",
|
||||
"mailinator.com",
|
||||
"mailinator.net",
|
||||
"mailinator.org",
|
||||
"mailinator.info",
|
||||
"maildrop.cc",
|
||||
"yopmail.com",
|
||||
"yopmail.fr",
|
||||
"yopmail.net",
|
||||
"fakeinbox.com",
|
||||
"fakemailgenerator.com",
|
||||
"dispostable.com",
|
||||
"getairmail.com",
|
||||
"getnada.com",
|
||||
"mohmal.com",
|
||||
"trashmail.com",
|
||||
"trashmail.net",
|
||||
"trashmail.org",
|
||||
"sharklasers.com",
|
||||
"grr.la",
|
||||
"guerrillamail.de",
|
||||
"pokemail.net",
|
||||
"spam4.me",
|
||||
"spamgourmet.com",
|
||||
"mytrashmail.com",
|
||||
"mailnesia.com",
|
||||
"mailcatch.com",
|
||||
"mintemail.com",
|
||||
"tempr.email",
|
||||
"tempail.com",
|
||||
"emailondeck.com",
|
||||
"incognitomail.com",
|
||||
"inboxalias.com",
|
||||
"33mail.com",
|
||||
"spamex.com",
|
||||
"spamfree24.org",
|
||||
"spamspot.com",
|
||||
"mailnull.com",
|
||||
"mailsac.com",
|
||||
"emailfake.com",
|
||||
"fakemail.fr",
|
||||
"tempinbox.com",
|
||||
"throwaway.email",
|
||||
"burnermail.io",
|
||||
"anonbox.net",
|
||||
"anonymbox.com",
|
||||
"discard.email",
|
||||
"discardmail.com",
|
||||
"discardmail.de",
|
||||
"mailexpire.com",
|
||||
"mailforspam.com",
|
||||
"meltmail.com",
|
||||
"mt2009.com",
|
||||
"mt2014.com",
|
||||
"nospam.ze.tc",
|
||||
"nospamfor.us",
|
||||
"nowmymail.com",
|
||||
"receiveee.com",
|
||||
"safe-mail.net",
|
||||
"spamavert.com",
|
||||
"spambob.com",
|
||||
"spambog.com",
|
||||
"spambox.us",
|
||||
"spamcannon.com",
|
||||
"spamcannon.net",
|
||||
"spamcon.org",
|
||||
"spamcorptastic.com",
|
||||
"spamday.com",
|
||||
"spamfree.eu",
|
||||
"spamherelots.com",
|
||||
"spamhereplease.com",
|
||||
"spamhole.com",
|
||||
"spamify.com",
|
||||
"spaminator.de",
|
||||
"spamkill.info",
|
||||
"spaml.com",
|
||||
"spaml.de",
|
||||
"spamoff.de",
|
||||
"spamobox.com",
|
||||
"spamslicer.com",
|
||||
"spamstack.net",
|
||||
"spamthis.co.uk",
|
||||
"spamthisplease.com",
|
||||
"supergreatmail.com",
|
||||
"suremail.info",
|
||||
"teleworm.us",
|
||||
"tempemail.co.za",
|
||||
"tempemail.net",
|
||||
"tempmailaddress.com",
|
||||
"tempmailo.com",
|
||||
"thankyou2010.com",
|
||||
"thisisnotmyrealemail.com",
|
||||
"tm.slsrs.ru",
|
||||
"tmpeml.info",
|
||||
"trash-mail.at",
|
||||
"trash-mail.de",
|
||||
"trash2009.com",
|
||||
"trashemail.de",
|
||||
"trashmail.at",
|
||||
"trashmailer.com",
|
||||
"wegwerfmail.de",
|
||||
"wegwerfmail.net",
|
||||
"wegwerfmail.org",
|
||||
"wh4f.org",
|
||||
"willhackforfood.biz",
|
||||
"willselfdestruct.com",
|
||||
"xmaily.com",
|
||||
"xyzfree.net",
|
||||
"yep.it",
|
||||
"yogamaven.com",
|
||||
"yuurok.com",
|
||||
"zehnminutenmail.de",
|
||||
"zippymail.info",
|
||||
# Additional common ones
|
||||
"mailnator.com",
|
||||
"bugmenot.com",
|
||||
"dodgeit.com",
|
||||
"dodgit.com",
|
||||
"e4ward.com",
|
||||
"emailsensei.com",
|
||||
"hushmail.com",
|
||||
"jetable.org",
|
||||
"kasmail.com",
|
||||
"mailblock.net",
|
||||
"mailcatch.com",
|
||||
"mymailoasis.com",
|
||||
"nervmich.net",
|
||||
"nervtmansen.de",
|
||||
"oneoffemail.com",
|
||||
"pookmail.com",
|
||||
"shortmail.net",
|
||||
"sneakemail.com",
|
||||
"sogetthis.com",
|
||||
"tempemailaddress.com",
|
||||
"tempomail.fr",
|
||||
"temporaryemail.net",
|
||||
"temporaryforwarding.com",
|
||||
"temporaryinbox.com",
|
||||
"thankyou2010.com",
|
||||
"tyldd.com",
|
||||
"uggsrock.com",
|
||||
"veryrealemail.com",
|
||||
"yourewronghereswhy.com",
|
||||
}
|
||||
|
||||
# Gmail-like domains that support alias normalization
|
||||
GMAIL_LIKE_DOMAINS = {
|
||||
"gmail.com",
|
||||
"googlemail.com",
|
||||
}
|
||||
|
||||
|
||||
class EmailValidationService:
|
||||
"""
|
||||
Service for email validation and abuse prevention.
|
||||
|
||||
Features:
|
||||
- Email normalization (Gmail alias stripping)
|
||||
- Disposable domain detection
|
||||
- Blocklist checking (emails, IPs)
|
||||
- IP-based enrollment rate limiting
|
||||
"""
|
||||
|
||||
_instance: Optional["EmailValidationService"] = None
|
||||
_blocklist_cache: Optional[dict] = None
|
||||
_blocklist_cache_time: Optional[datetime] = None
|
||||
_cache_ttl_seconds: int = 300 # 5 minutes
|
||||
|
||||
def __init__(self, db: Optional[firestore.Client] = None):
|
||||
"""Initialize the email validation service."""
|
||||
if db is None:
|
||||
from backend.services.firestore_service import get_firestore_client
|
||||
db = get_firestore_client()
|
||||
self.db = db
|
||||
|
||||
@classmethod
|
||||
def get_instance(cls, db: Optional[firestore.Client] = None) -> "EmailValidationService":
|
||||
"""Get singleton instance of the service."""
|
||||
if cls._instance is None:
|
||||
cls._instance = cls(db)
|
||||
return cls._instance
|
||||
|
||||
def normalize_email(self, email: str) -> str:
|
||||
"""
|
||||
Normalize an email address.
|
||||
|
||||
For Gmail-like domains:
|
||||
- Removes dots from local part
|
||||
- Removes everything after + in local part
|
||||
- Converts to lowercase
|
||||
|
||||
For other domains:
|
||||
- Only converts to lowercase
|
||||
|
||||
Args:
|
||||
email: Email address to normalize
|
||||
|
||||
Returns:
|
||||
Normalized email address
|
||||
|
||||
Examples:
|
||||
j.o.h.n+spam@gmail.com -> john@gmail.com
|
||||
John.Doe@example.com -> john.doe@example.com
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return email.lower().strip() if email else ""
|
||||
|
||||
email = email.lower().strip()
|
||||
local, domain = email.rsplit("@", 1)
|
||||
|
||||
if domain in GMAIL_LIKE_DOMAINS:
|
||||
# Remove dots from local part
|
||||
local = local.replace(".", "")
|
||||
# Remove everything after +
|
||||
if "+" in local:
|
||||
local = local.split("+")[0]
|
||||
|
||||
return f"{local}@{domain}"
|
||||
|
||||
def get_blocklist_config(self, force_refresh: bool = False) -> dict:
|
||||
"""
|
||||
Get blocklist configuration from Firestore.
|
||||
|
||||
Uses caching to reduce Firestore reads.
|
||||
|
||||
Returns:
|
||||
Dict with disposable_domains, blocked_emails, blocked_ips sets
|
||||
"""
|
||||
now = datetime.now(timezone.utc)
|
||||
|
||||
# Check cache validity
|
||||
if (
|
||||
not force_refresh
|
||||
and self._blocklist_cache is not None
|
||||
and self._blocklist_cache_time is not None
|
||||
and (now - self._blocklist_cache_time).total_seconds() < self._cache_ttl_seconds
|
||||
):
|
||||
return self._blocklist_cache
|
||||
|
||||
# Fetch from Firestore
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
config = {
|
||||
"disposable_domains": set(data.get("disposable_domains", [])) | DEFAULT_DISPOSABLE_DOMAINS,
|
||||
"blocked_emails": set(data.get("blocked_emails", [])),
|
||||
"blocked_ips": set(data.get("blocked_ips", [])),
|
||||
}
|
||||
else:
|
||||
# Use defaults if no config exists
|
||||
config = {
|
||||
"disposable_domains": DEFAULT_DISPOSABLE_DOMAINS.copy(),
|
||||
"blocked_emails": set(),
|
||||
"blocked_ips": set(),
|
||||
}
|
||||
|
||||
# Update cache
|
||||
EmailValidationService._blocklist_cache = config
|
||||
EmailValidationService._blocklist_cache_time = now
|
||||
|
||||
return config
|
||||
|
||||
def is_disposable_domain(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email uses a disposable domain.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the domain is known to be disposable
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False
|
||||
|
||||
domain = email.lower().split("@")[-1]
|
||||
config = self.get_blocklist_config()
|
||||
return domain in config["disposable_domains"]
|
||||
|
||||
def is_email_blocked(self, email: str) -> bool:
|
||||
"""
|
||||
Check if an email is explicitly blocked.
|
||||
|
||||
Checks both the raw email and normalized version.
|
||||
|
||||
Args:
|
||||
email: Email address to check
|
||||
|
||||
Returns:
|
||||
True if the email is blocked
|
||||
"""
|
||||
if not email:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
email_lower = email.lower().strip()
|
||||
email_normalized = self.normalize_email(email)
|
||||
|
||||
return (
|
||||
email_lower in config["blocked_emails"]
|
||||
or email_normalized in config["blocked_emails"]
|
||||
)
|
||||
|
||||
def is_ip_blocked(self, ip_address: str) -> bool:
|
||||
"""
|
||||
Check if an IP address is blocked.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to check
|
||||
|
||||
Returns:
|
||||
True if the IP is blocked
|
||||
"""
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
config = self.get_blocklist_config()
|
||||
return ip_address in config["blocked_ips"]
|
||||
|
||||
def validate_email_for_beta(self, email: str) -> Tuple[bool, str]:
|
||||
"""
|
||||
Validate an email for beta enrollment.
|
||||
|
||||
Performs all validation checks:
|
||||
1. Basic format validation
|
||||
2. Disposable domain check
|
||||
3. Blocked email check
|
||||
|
||||
Args:
|
||||
email: Email address to validate
|
||||
|
||||
Returns:
|
||||
Tuple of (is_valid, error_message)
|
||||
"""
|
||||
if not email or "@" not in email:
|
||||
return False, "Invalid email format"
|
||||
|
||||
# Check disposable domain
|
||||
if self.is_disposable_domain(email):
|
||||
logger.warning(f"Beta enrollment blocked - disposable domain: {_mask_email(email)}")
|
||||
return False, "Disposable email addresses are not allowed"
|
||||
|
||||
# Check blocked email
|
||||
if self.is_email_blocked(email):
|
||||
logger.warning(f"Beta enrollment blocked - email blocked: {_mask_email(email)}")
|
||||
return False, "This email address is not allowed"
|
||||
|
||||
return True, ""
|
||||
|
||||
def hash_ip(self, ip_address: str) -> str:
|
||||
"""
|
||||
Hash an IP address for privacy-preserving storage.
|
||||
|
||||
Args:
|
||||
ip_address: IP address to hash
|
||||
|
||||
Returns:
|
||||
SHA-256 hash of the IP
|
||||
"""
|
||||
return hashlib.sha256(ip_address.encode()).hexdigest()
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Blocklist Management (Admin)
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def add_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Add a domain to the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
if not domain:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
else:
|
||||
data = {}
|
||||
domains = set()
|
||||
|
||||
domains.add(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_disposable_domain(self, domain: str, admin_email: str) -> bool:
|
||||
"""Remove a domain from the disposable domains blocklist."""
|
||||
domain = domain.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
domains = set(data.get("disposable_domains", []))
|
||||
|
||||
if domain not in domains:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
domains.discard(domain)
|
||||
data["disposable_domains"] = list(domains)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed disposable domain: {domain} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Add an email to the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
if not email:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
else:
|
||||
data = {}
|
||||
emails = set()
|
||||
|
||||
emails.add(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_email(self, email: str, admin_email: str) -> bool:
|
||||
"""Remove an email from the blocked emails list."""
|
||||
email = email.lower().strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
emails = set(data.get("blocked_emails", []))
|
||||
|
||||
if email not in emails:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
emails.discard(email)
|
||||
data["blocked_emails"] = list(emails)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked email: {email} by {admin_email}")
|
||||
return True
|
||||
|
||||
def add_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Add an IP address to the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
if not ip_address:
|
||||
return False
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if doc.exists:
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
else:
|
||||
data = {}
|
||||
ips = set()
|
||||
|
||||
ips.add(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Added blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def remove_blocked_ip(self, ip_address: str, admin_email: str) -> bool:
|
||||
"""Remove an IP address from the blocked IPs list."""
|
||||
ip_address = ip_address.strip()
|
||||
|
||||
doc_ref = self.db.collection(BLOCKLISTS_COLLECTION).document(BLOCKLIST_CONFIG_DOC)
|
||||
result = {"found": False}
|
||||
|
||||
@firestore.transactional
|
||||
def remove_in_transaction(transaction, doc_ref):
|
||||
doc = doc_ref.get(transaction=transaction)
|
||||
if not doc.exists:
|
||||
return
|
||||
|
||||
data = doc.to_dict()
|
||||
ips = set(data.get("blocked_ips", []))
|
||||
|
||||
if ip_address not in ips:
|
||||
return
|
||||
|
||||
result["found"] = True
|
||||
ips.discard(ip_address)
|
||||
data["blocked_ips"] = list(ips)
|
||||
data["updated_at"] = datetime.now(timezone.utc)
|
||||
data["updated_by"] = admin_email
|
||||
transaction.set(doc_ref, data, merge=True)
|
||||
|
||||
transaction = self.db.transaction()
|
||||
remove_in_transaction(transaction, doc_ref)
|
||||
|
||||
if not result["found"]:
|
||||
return False
|
||||
|
||||
# Invalidate cache
|
||||
EmailValidationService._blocklist_cache = None
|
||||
|
||||
logger.info(f"Removed blocked IP: {ip_address} by {admin_email}")
|
||||
return True
|
||||
|
||||
def get_blocklist_stats(self) -> dict:
|
||||
"""Get statistics about current blocklists."""
|
||||
config = self.get_blocklist_config(force_refresh=True)
|
||||
return {
|
||||
"disposable_domains_count": len(config["disposable_domains"]),
|
||||
"blocked_emails_count": len(config["blocked_emails"]),
|
||||
"blocked_ips_count": len(config["blocked_ips"]),
|
||||
"default_disposable_domains_count": len(DEFAULT_DISPOSABLE_DOMAINS),
|
||||
}
|
||||
|
||||
|
||||
def get_email_validation_service(
|
||||
db: Optional[firestore.Client] = None,
|
||||
) -> EmailValidationService:
|
||||
"""Get the singleton EmailValidationService instance."""
|
||||
return EmailValidationService.get_instance(db)
|
||||
@@ -1,465 +0,0 @@
|
||||
"""
|
||||
Encoding Interface.
|
||||
|
||||
Defines the abstract interface for video encoding backends, allowing
|
||||
the video worker orchestrator to use either GCE or local encoding
|
||||
interchangeably.
|
||||
|
||||
This follows the Strategy pattern - different encoding implementations
|
||||
can be swapped without changing the orchestration logic.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from abc import ABC, abstractmethod
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Any
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingInput:
|
||||
"""
|
||||
Input configuration for video encoding.
|
||||
|
||||
Contains all the paths and metadata needed to encode a karaoke video.
|
||||
"""
|
||||
# Required input files
|
||||
title_video_path: str # Title card video (MOV)
|
||||
karaoke_video_path: str # Main karaoke video with vocals (MOV/MKV)
|
||||
instrumental_audio_path: str # Instrumental audio track (FLAC)
|
||||
|
||||
# Optional input files
|
||||
end_video_path: Optional[str] = None # End credits video (MOV)
|
||||
|
||||
# Metadata for output naming
|
||||
artist: str = ""
|
||||
title: str = ""
|
||||
brand_code: Optional[str] = None
|
||||
|
||||
# Output directory
|
||||
output_dir: str = ""
|
||||
|
||||
# Instrumental selection (clean, with_backing, or custom)
|
||||
instrumental_selection: str = "clean"
|
||||
|
||||
# Additional options
|
||||
options: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class EncodingOutput:
|
||||
"""
|
||||
Output from video encoding.
|
||||
|
||||
Contains paths to all generated video files and status information.
|
||||
"""
|
||||
success: bool
|
||||
error_message: Optional[str] = None
|
||||
|
||||
# Output file paths (relative to output_dir or absolute)
|
||||
karaoke_mp4_path: Optional[str] = None # Karaoke video with instrumental audio
|
||||
with_vocals_mp4_path: Optional[str] = None # With vocals MP4 version
|
||||
lossless_4k_mp4_path: Optional[str] = None # Final lossless 4K MP4
|
||||
lossy_4k_mp4_path: Optional[str] = None # Final lossy 4K MP4
|
||||
lossless_mkv_path: Optional[str] = None # MKV with FLAC (for YouTube)
|
||||
lossy_720p_mp4_path: Optional[str] = None # 720p web version
|
||||
|
||||
# All output files as a dict for convenience
|
||||
output_files: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
# Encoding metadata
|
||||
encoding_time_seconds: Optional[float] = None
|
||||
encoding_backend: Optional[str] = None # "gce" or "local"
|
||||
|
||||
|
||||
class EncodingBackend(ABC):
|
||||
"""
|
||||
Abstract base class for video encoding backends.
|
||||
|
||||
Implementations:
|
||||
- GCEEncodingBackend: Cloud-based encoding using Google Compute Engine
|
||||
- LocalEncodingBackend: Local FFmpeg-based encoding
|
||||
"""
|
||||
|
||||
@property
|
||||
@abstractmethod
|
||||
def name(self) -> str:
|
||||
"""Return the name of this encoding backend."""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video files according to the input configuration.
|
||||
|
||||
This method should:
|
||||
1. Create karaoke video with instrumental audio
|
||||
2. Convert with-vocals video to MP4 if needed
|
||||
3. Encode lossless 4K MP4 (concatenating title + karaoke + end)
|
||||
4. Encode lossy 4K MP4 with AAC audio
|
||||
5. Create MKV with FLAC audio for YouTube
|
||||
6. Encode 720p version for web
|
||||
|
||||
Args:
|
||||
input_config: EncodingInput with all required paths and options
|
||||
|
||||
Returns:
|
||||
EncodingOutput with paths to generated files and status
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def is_available(self) -> bool:
|
||||
"""
|
||||
Check if this encoding backend is available and configured.
|
||||
|
||||
Returns:
|
||||
True if the backend can be used, False otherwise
|
||||
"""
|
||||
pass
|
||||
|
||||
@abstractmethod
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of the encoding backend.
|
||||
|
||||
Returns:
|
||||
Dict with status information (availability, queue length, etc.)
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class LocalEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
Local FFmpeg-based encoding backend.
|
||||
|
||||
Wraps the LocalEncodingService to implement the EncodingBackend interface.
|
||||
Uses asyncio.to_thread() to run synchronous FFmpeg operations asynchronously.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the local encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: If True, log operations without executing
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "local"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the local encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.local_encoding_service import LocalEncodingService
|
||||
self._service = LocalEncodingService(
|
||||
dry_run=self.dry_run,
|
||||
logger=self.logger
|
||||
)
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using local FFmpeg.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import asyncio
|
||||
import time
|
||||
from backend.services.local_encoding_service import EncodingConfig
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
# Build output file paths
|
||||
base_name = f"{input_config.artist} - {input_config.title}"
|
||||
output_dir = input_config.output_dir or "."
|
||||
|
||||
import os
|
||||
config = EncodingConfig(
|
||||
title_video=input_config.title_video_path,
|
||||
karaoke_video=input_config.karaoke_video_path,
|
||||
instrumental_audio=input_config.instrumental_audio_path,
|
||||
end_video=input_config.end_video_path,
|
||||
output_karaoke_mp4=os.path.join(output_dir, f"{base_name} (Karaoke).mp4"),
|
||||
output_with_vocals_mp4=os.path.join(output_dir, f"{base_name} (With Vocals).mp4"),
|
||||
output_lossless_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mp4"),
|
||||
output_lossy_4k_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 4k).mp4"),
|
||||
output_lossless_mkv=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossless 4k).mkv"),
|
||||
output_720p_mp4=os.path.join(output_dir, f"{base_name} (Final Karaoke Lossy 720p).mp4"),
|
||||
countdown_padding_seconds=input_config.options.get("countdown_padding_seconds"),
|
||||
)
|
||||
|
||||
# Run encoding in thread pool to avoid blocking
|
||||
service = self._get_service()
|
||||
result = await asyncio.to_thread(service.encode_all_formats, config)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
if result.success:
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
karaoke_mp4_path=config.output_karaoke_mp4,
|
||||
with_vocals_mp4_path=config.output_with_vocals_mp4,
|
||||
lossless_4k_mp4_path=config.output_lossless_4k_mp4,
|
||||
lossy_4k_mp4_path=config.output_lossy_4k_mp4,
|
||||
lossless_mkv_path=config.output_lossless_mkv,
|
||||
lossy_720p_mp4_path=config.output_720p_mp4,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
else:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=result.error,
|
||||
output_files=result.output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if local encoding is available (FFmpeg installed)."""
|
||||
import subprocess
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
result = await asyncio.to_thread(
|
||||
subprocess.run,
|
||||
["ffmpeg", "-version"],
|
||||
capture_output=True,
|
||||
timeout=10
|
||||
)
|
||||
return result.returncode == 0
|
||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get local encoding status."""
|
||||
available = await self.is_available()
|
||||
service = self._get_service()
|
||||
|
||||
return {
|
||||
"backend": self.name,
|
||||
"available": available,
|
||||
"hwaccel_available": service.hwaccel_available if available else False,
|
||||
"video_encoder": service.video_encoder if available else None,
|
||||
}
|
||||
|
||||
|
||||
class GCEEncodingBackend(EncodingBackend):
|
||||
"""
|
||||
GCE-based encoding backend.
|
||||
|
||||
Wraps the existing EncodingService to implement the EncodingBackend interface.
|
||||
Submits jobs to a remote GCE worker for high-performance encoding.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
dry_run: bool = False,
|
||||
logger: Optional[logging.Logger] = None,
|
||||
):
|
||||
"""
|
||||
Initialize the GCE encoding backend.
|
||||
|
||||
Args:
|
||||
dry_run: Ignored for GCE backend (remote service doesn't support dry run)
|
||||
logger: Optional logger instance
|
||||
"""
|
||||
self.dry_run = dry_run # Stored but not used (GCE is remote)
|
||||
self.logger = logger or logging.getLogger(__name__)
|
||||
self._service = None
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "gce"
|
||||
|
||||
def _get_service(self):
|
||||
"""Lazy-load the GCE encoding service."""
|
||||
if self._service is None:
|
||||
from backend.services.encoding_service import get_encoding_service
|
||||
self._service = get_encoding_service()
|
||||
return self._service
|
||||
|
||||
async def encode(self, input_config: EncodingInput) -> EncodingOutput:
|
||||
"""
|
||||
Encode video using GCE worker.
|
||||
|
||||
Note: GCE encoding requires files to be in GCS, so this method
|
||||
expects input_config.options to contain 'input_gcs_path' and
|
||||
'output_gcs_path'.
|
||||
|
||||
Args:
|
||||
input_config: Encoding input configuration
|
||||
|
||||
Returns:
|
||||
EncodingOutput with results
|
||||
"""
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
service = self._get_service()
|
||||
|
||||
# GCE requires GCS paths
|
||||
input_gcs_path = input_config.options.get("input_gcs_path")
|
||||
output_gcs_path = input_config.options.get("output_gcs_path")
|
||||
job_id = input_config.options.get("job_id", input_config.brand_code or "unknown")
|
||||
|
||||
if not input_gcs_path or not output_gcs_path:
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message="GCE encoding requires input_gcs_path and output_gcs_path in options",
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
try:
|
||||
# Build encoding config
|
||||
encoding_config = {
|
||||
"formats": ["mp4_4k_lossless", "mp4_4k_lossy", "mkv_4k", "mp4_720p"],
|
||||
"artist": input_config.artist,
|
||||
"title": input_config.title,
|
||||
"instrumental_selection": input_config.instrumental_selection,
|
||||
}
|
||||
|
||||
# Add countdown padding if present (for audio sync with countdown-padded vocals)
|
||||
countdown_padding = input_config.options.get("countdown_padding_seconds")
|
||||
if countdown_padding:
|
||||
encoding_config["countdown_padding_seconds"] = countdown_padding
|
||||
self.logger.info(f"GCE encoding with countdown_padding_seconds={countdown_padding}")
|
||||
|
||||
# Submit and wait for completion
|
||||
result = await service.encode_videos(
|
||||
job_id=job_id,
|
||||
input_gcs_path=input_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
encoding_config=encoding_config,
|
||||
)
|
||||
|
||||
encoding_time = time.time() - start_time
|
||||
|
||||
# Extract output file paths from result
|
||||
# Handle case where GCE worker returns a list or unexpected format
|
||||
if isinstance(result, list):
|
||||
# If result is a list, try to find the output_files in the first dict
|
||||
self.logger.warning(f"GCE returned list instead of dict: {result}")
|
||||
result = result[0] if result and isinstance(result[0], dict) else {}
|
||||
if not isinstance(result, dict):
|
||||
self.logger.error(f"Unexpected GCE result type: {type(result)}")
|
||||
result = {}
|
||||
raw_output_files = result.get("output_files", {})
|
||||
|
||||
# Convert output_files from list of paths to dict
|
||||
# GCE worker returns list like: ["path/Artist - Title (Final Karaoke Lossless 4k).mp4", ...]
|
||||
# We need dict like: {"mp4_4k_lossless": "path/...", "mp4_720p": "path/..."}
|
||||
if isinstance(raw_output_files, list):
|
||||
output_files = {}
|
||||
for path in raw_output_files:
|
||||
if not isinstance(path, str):
|
||||
continue
|
||||
filename = path.split("/")[-1] if "/" in path else path
|
||||
filename_lower = filename.lower()
|
||||
# Map filename patterns to output format keys
|
||||
# Files are named like "Artist - Title (Final Karaoke Lossless 4k).mp4"
|
||||
if "lossless 4k" in filename_lower:
|
||||
if filename.endswith(".mkv"):
|
||||
output_files["mkv_4k"] = path
|
||||
else:
|
||||
output_files["mp4_4k_lossless"] = path
|
||||
elif "lossy 4k" in filename_lower:
|
||||
output_files["mp4_4k_lossy"] = path
|
||||
elif "720p" in filename_lower:
|
||||
output_files["mp4_720p"] = path
|
||||
self.logger.info(f"Converted output_files list to dict: {output_files}")
|
||||
else:
|
||||
output_files = raw_output_files if isinstance(raw_output_files, dict) else {}
|
||||
|
||||
return EncodingOutput(
|
||||
success=True,
|
||||
lossless_4k_mp4_path=output_files.get("mp4_4k_lossless"),
|
||||
lossy_4k_mp4_path=output_files.get("mp4_4k_lossy"),
|
||||
lossless_mkv_path=output_files.get("mkv_4k"),
|
||||
lossy_720p_mp4_path=output_files.get("mp4_720p"),
|
||||
output_files=output_files,
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
encoding_time = time.time() - start_time
|
||||
self.logger.error(f"GCE encoding failed: {e}")
|
||||
return EncodingOutput(
|
||||
success=False,
|
||||
error_message=str(e),
|
||||
encoding_time_seconds=encoding_time,
|
||||
encoding_backend=self.name
|
||||
)
|
||||
|
||||
async def is_available(self) -> bool:
|
||||
"""Check if GCE encoding is available and configured."""
|
||||
service = self._get_service()
|
||||
return service.is_enabled
|
||||
|
||||
async def get_status(self) -> Dict[str, Any]:
|
||||
"""Get GCE encoding status."""
|
||||
service = self._get_service()
|
||||
|
||||
status = {
|
||||
"backend": self.name,
|
||||
"available": service.is_enabled,
|
||||
"configured": service.is_configured,
|
||||
}
|
||||
|
||||
if service.is_configured:
|
||||
try:
|
||||
health = await service.health_check()
|
||||
status["health"] = health
|
||||
except Exception as e:
|
||||
status["health_error"] = str(e)
|
||||
|
||||
return status
|
||||
|
||||
|
||||
# Factory function to get an encoding backend
|
||||
def get_encoding_backend(
|
||||
backend_type: str = "auto",
|
||||
**kwargs
|
||||
) -> EncodingBackend:
|
||||
"""
|
||||
Get an encoding backend instance.
|
||||
|
||||
Args:
|
||||
backend_type: Type of backend - "local", "gce", or "auto"
|
||||
**kwargs: Additional arguments for the backend
|
||||
|
||||
Returns:
|
||||
EncodingBackend instance
|
||||
|
||||
Raises:
|
||||
ValueError: If backend_type is unknown
|
||||
"""
|
||||
if backend_type == "local":
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
elif backend_type == "gce":
|
||||
return GCEEncodingBackend(**kwargs)
|
||||
elif backend_type == "auto":
|
||||
# Check if GCE is available, otherwise use local
|
||||
gce_backend = GCEEncodingBackend(**kwargs)
|
||||
if gce_backend._get_service().is_enabled:
|
||||
return gce_backend
|
||||
return LocalEncodingBackend(**kwargs)
|
||||
else:
|
||||
raise ValueError(f"Unknown encoding backend type: {backend_type}")
|
||||
@@ -1,502 +0,0 @@
|
||||
"""
|
||||
GCE Encoding Worker Service.
|
||||
|
||||
This service dispatches video encoding jobs to a dedicated high-performance
|
||||
GCE instance (C4-standard with Intel Granite Rapids CPU) for faster encoding.
|
||||
|
||||
The GCE worker provides:
|
||||
- 3.9 GHz all-core frequency (vs 3.7 GHz on Cloud Run)
|
||||
- Dedicated vCPUs (no contention)
|
||||
- 2-3x faster FFmpeg libx264 encoding
|
||||
|
||||
Usage:
|
||||
encoding_service = get_encoding_service()
|
||||
if encoding_service.is_configured:
|
||||
result = await encoding_service.encode_videos(job_id, input_gcs_path, config)
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Optional, Dict, Any
|
||||
|
||||
import aiohttp
|
||||
|
||||
from backend.config import get_settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Retry configuration for handling transient failures (e.g., worker restarts)
|
||||
MAX_RETRIES = 3
|
||||
INITIAL_BACKOFF_SECONDS = 2.0
|
||||
MAX_BACKOFF_SECONDS = 10.0
|
||||
|
||||
|
||||
class EncodingService:
|
||||
"""Service for dispatching encoding jobs to GCE worker."""
|
||||
|
||||
def __init__(self):
|
||||
self.settings = get_settings()
|
||||
self._url = None
|
||||
self._api_key = None
|
||||
self._initialized = False
|
||||
|
||||
def _load_credentials(self):
|
||||
"""Load encoding worker URL and API key from config/secrets."""
|
||||
if self._initialized:
|
||||
return
|
||||
|
||||
# Try environment variables first, then Secret Manager
|
||||
self._url = self.settings.encoding_worker_url
|
||||
self._api_key = self.settings.encoding_worker_api_key
|
||||
|
||||
# Fall back to Secret Manager
|
||||
if not self._url:
|
||||
self._url = self.settings.get_secret("encoding-worker-url")
|
||||
if not self._api_key:
|
||||
self._api_key = self.settings.get_secret("encoding-worker-api-key")
|
||||
|
||||
self._initialized = True
|
||||
|
||||
@property
|
||||
def is_configured(self) -> bool:
|
||||
"""Check if encoding service is configured with URL and API key."""
|
||||
self._load_credentials()
|
||||
return bool(self._url and self._api_key)
|
||||
|
||||
@property
|
||||
def is_enabled(self) -> bool:
|
||||
"""Check if GCE encoding is enabled and configured."""
|
||||
return self.settings.use_gce_encoding and self.is_configured
|
||||
|
||||
@property
|
||||
def is_preview_enabled(self) -> bool:
|
||||
"""Check if GCE preview encoding is enabled and configured."""
|
||||
return self.settings.use_gce_preview_encoding and self.is_configured
|
||||
|
||||
async def _request_with_retry(
|
||||
self,
|
||||
method: str,
|
||||
url: str,
|
||||
headers: Dict[str, str],
|
||||
json_payload: Optional[Dict[str, Any]] = None,
|
||||
timeout: float = 30.0,
|
||||
job_id: str = "unknown",
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make an HTTP request with retry logic for transient failures.
|
||||
|
||||
This handles connection errors that occur when the GCE worker is
|
||||
restarting (e.g., during deployments) by retrying with exponential backoff.
|
||||
|
||||
Args:
|
||||
method: HTTP method (GET, POST)
|
||||
url: Request URL
|
||||
headers: Request headers
|
||||
json_payload: JSON body for POST requests
|
||||
timeout: Request timeout in seconds
|
||||
job_id: Job ID for logging
|
||||
|
||||
Returns:
|
||||
Dict with keys:
|
||||
- status (int): HTTP status code
|
||||
- json (Any): Parsed JSON response body (if status 200, else None)
|
||||
- text (str): Raw response text (if status != 200, else None)
|
||||
|
||||
Raises:
|
||||
aiohttp.ClientConnectorError: If all retries fail due to connection errors
|
||||
aiohttp.ServerDisconnectedError: If all retries fail due to server disconnect
|
||||
asyncio.TimeoutError: If all retries fail due to timeout
|
||||
"""
|
||||
last_exception = None
|
||||
backoff = INITIAL_BACKOFF_SECONDS
|
||||
|
||||
for attempt in range(MAX_RETRIES + 1):
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
if method.upper() == "POST":
|
||||
async with session.post(
|
||||
url, json=json_payload, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
# Return a copy of the response data since we exit the context
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
else: # GET
|
||||
async with session.get(
|
||||
url, headers=headers, timeout=timeout
|
||||
) as resp:
|
||||
return {
|
||||
"status": resp.status,
|
||||
"json": await resp.json() if resp.status == 200 else None,
|
||||
"text": await resp.text() if resp.status != 200 else None,
|
||||
}
|
||||
except (aiohttp.ClientConnectorError, aiohttp.ServerDisconnectedError, asyncio.TimeoutError) as e:
|
||||
last_exception = e
|
||||
if attempt < MAX_RETRIES:
|
||||
logger.warning(
|
||||
f"[job:{job_id}] GCE worker connection failed (attempt {attempt + 1}/{MAX_RETRIES + 1}): {e}. "
|
||||
f"Retrying in {backoff:.1f}s..."
|
||||
)
|
||||
await asyncio.sleep(backoff)
|
||||
backoff = min(backoff * 2, MAX_BACKOFF_SECONDS)
|
||||
else:
|
||||
logger.error(
|
||||
f"[job:{job_id}] GCE worker connection failed after {MAX_RETRIES + 1} attempts: {e}"
|
||||
)
|
||||
|
||||
raise last_exception
|
||||
|
||||
async def submit_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Dict[str, Any],
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit an encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files (gs://bucket/path/)
|
||||
output_gcs_path: GCS path for output files (gs://bucket/path/)
|
||||
encoding_config: Configuration for encoding (formats, quality, etc.)
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"input_gcs_path": input_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"encoding_config": encoding_config,
|
||||
}
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def get_job_status(self, job_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the status of an encoding job.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
|
||||
Returns:
|
||||
Job status including: status, progress, error, output_files
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/status/{job_id}"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="GET",
|
||||
url=url,
|
||||
headers=headers,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 404:
|
||||
raise RuntimeError(f"Encoding job {job_id} not found")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to get job status: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def wait_for_completion(
|
||||
self,
|
||||
job_id: str,
|
||||
poll_interval: float = 10.0,
|
||||
timeout: float = 3600.0,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Poll for encoding job completion.
|
||||
|
||||
Args:
|
||||
job_id: Job identifier
|
||||
poll_interval: Seconds between status checks
|
||||
timeout: Maximum time to wait (default 1 hour)
|
||||
progress_callback: Optional callback(progress: int) for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
|
||||
Raises:
|
||||
TimeoutError: If job doesn't complete within timeout
|
||||
RuntimeError: If job fails
|
||||
"""
|
||||
logger.info(f"[job:{job_id}] Waiting for GCE encoding to complete...")
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
last_progress = 0
|
||||
|
||||
while True:
|
||||
elapsed = asyncio.get_event_loop().time() - start_time
|
||||
if elapsed > timeout:
|
||||
raise TimeoutError(f"Encoding job {job_id} timed out after {timeout}s")
|
||||
|
||||
status = await self.get_job_status(job_id)
|
||||
|
||||
# Handle case where GCE worker returns a list instead of dict
|
||||
if isinstance(status, list):
|
||||
logger.warning(f"[job:{job_id}] GCE returned list instead of dict: {status}")
|
||||
status = status[0] if status and isinstance(status[0], dict) else {}
|
||||
if not isinstance(status, dict):
|
||||
logger.error(f"[job:{job_id}] Unexpected status type: {type(status)}")
|
||||
status = {}
|
||||
|
||||
job_status = status.get("status", "unknown")
|
||||
progress = status.get("progress", 0)
|
||||
|
||||
# Report progress
|
||||
if progress != last_progress:
|
||||
logger.info(f"[job:{job_id}] Encoding progress: {progress}%")
|
||||
last_progress = progress
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(progress)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback failed: {e}")
|
||||
|
||||
if job_status == "complete":
|
||||
logger.info(f"[job:{job_id}] GCE encoding complete in {elapsed:.1f}s")
|
||||
return status
|
||||
|
||||
if job_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise RuntimeError(f"Encoding job {job_id} failed: {error}")
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
|
||||
async def encode_videos(
|
||||
self,
|
||||
job_id: str,
|
||||
input_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
encoding_config: Optional[Dict[str, Any]] = None,
|
||||
progress_callback=None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
input_gcs_path: GCS path to input files
|
||||
output_gcs_path: GCS path for output files
|
||||
encoding_config: Optional encoding configuration
|
||||
progress_callback: Optional callback for progress updates
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
config = encoding_config or {"formats": ["mp4_4k", "mp4_720p"]}
|
||||
|
||||
# Submit the job
|
||||
await self.submit_encoding_job(job_id, input_gcs_path, output_gcs_path, config)
|
||||
|
||||
# Wait for completion
|
||||
return await self.wait_for_completion(
|
||||
job_id, progress_callback=progress_callback
|
||||
)
|
||||
|
||||
async def submit_preview_encoding_job(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit a preview video encoding job to the GCE worker.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file (gs://bucket/path/file.ass)
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
|
||||
Returns:
|
||||
Response from the encoding worker
|
||||
|
||||
Raises:
|
||||
Exception: If submission fails
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
raise RuntimeError("Encoding service not configured")
|
||||
|
||||
url = f"{self._url}/encode-preview"
|
||||
headers = {"X-API-Key": self._api_key, "Content-Type": "application/json"}
|
||||
payload = {
|
||||
"job_id": job_id,
|
||||
"ass_gcs_path": ass_gcs_path,
|
||||
"audio_gcs_path": audio_gcs_path,
|
||||
"output_gcs_path": output_gcs_path,
|
||||
"background_color": background_color,
|
||||
}
|
||||
if background_image_gcs_path:
|
||||
payload["background_image_gcs_path"] = background_image_gcs_path
|
||||
if font_gcs_path:
|
||||
payload["font_gcs_path"] = font_gcs_path
|
||||
|
||||
logger.info(f"[job:{job_id}] Submitting preview encoding job to GCE worker: {url}")
|
||||
|
||||
resp = await self._request_with_retry(
|
||||
method="POST",
|
||||
url=url,
|
||||
headers=headers,
|
||||
json_payload=payload,
|
||||
timeout=30.0,
|
||||
job_id=job_id,
|
||||
)
|
||||
|
||||
if resp["status"] == 401:
|
||||
raise RuntimeError("Invalid API key for encoding worker")
|
||||
if resp["status"] == 409:
|
||||
raise RuntimeError(f"Preview encoding job {job_id} already exists")
|
||||
if resp["status"] != 200:
|
||||
raise RuntimeError(f"Failed to submit preview encoding job: {resp['status']} - {resp['text']}")
|
||||
|
||||
return resp["json"]
|
||||
|
||||
async def encode_preview_video(
|
||||
self,
|
||||
job_id: str,
|
||||
ass_gcs_path: str,
|
||||
audio_gcs_path: str,
|
||||
output_gcs_path: str,
|
||||
background_color: str = "black",
|
||||
background_image_gcs_path: Optional[str] = None,
|
||||
font_gcs_path: Optional[str] = None,
|
||||
timeout: float = 90.0,
|
||||
poll_interval: float = 2.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Submit preview encoding job and wait for completion.
|
||||
|
||||
This is a convenience method that combines submit + wait with shorter
|
||||
timeouts suitable for preview videos.
|
||||
|
||||
Args:
|
||||
job_id: Unique job identifier
|
||||
ass_gcs_path: GCS path to ASS subtitles file
|
||||
audio_gcs_path: GCS path to audio file
|
||||
output_gcs_path: GCS path for output video
|
||||
background_color: Background color (default: black)
|
||||
background_image_gcs_path: Optional GCS path to background image
|
||||
font_gcs_path: Optional GCS path to custom font file
|
||||
timeout: Maximum time to wait (default 90s for preview)
|
||||
poll_interval: Seconds between status checks (default 2s)
|
||||
|
||||
Returns:
|
||||
Final job status with output files
|
||||
"""
|
||||
# Submit the job
|
||||
submit_result = await self.submit_preview_encoding_job(
|
||||
job_id=job_id,
|
||||
ass_gcs_path=ass_gcs_path,
|
||||
audio_gcs_path=audio_gcs_path,
|
||||
output_gcs_path=output_gcs_path,
|
||||
background_color=background_color,
|
||||
background_image_gcs_path=background_image_gcs_path,
|
||||
font_gcs_path=font_gcs_path,
|
||||
)
|
||||
|
||||
# If cached, return immediately - video already exists in GCS
|
||||
submit_status = submit_result.get("status")
|
||||
if submit_status == "cached":
|
||||
logger.info(f"[job:{job_id}] Preview already cached, returning immediately")
|
||||
return {"status": "complete", "output_path": submit_result.get("output_path")}
|
||||
|
||||
# If in_progress, another request is encoding it - just wait for that
|
||||
if submit_status == "in_progress":
|
||||
logger.info(f"[job:{job_id}] Preview encoding already in progress, waiting")
|
||||
|
||||
# Wait for completion with shorter timeout
|
||||
return await self.wait_for_completion(
|
||||
job_id=job_id,
|
||||
poll_interval=poll_interval,
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check the health of the encoding worker.
|
||||
|
||||
Returns:
|
||||
Health status including active jobs and FFmpeg version
|
||||
"""
|
||||
self._load_credentials()
|
||||
|
||||
if not self.is_configured:
|
||||
return {"status": "not_configured"}
|
||||
|
||||
url = f"{self._url}/health"
|
||||
headers = {"X-API-Key": self._api_key}
|
||||
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=headers, timeout=10) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.json()
|
||||
return {"status": "error", "code": resp.status}
|
||||
except Exception as e:
|
||||
return {"status": "error", "error": str(e)}
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_encoding_service: Optional[EncodingService] = None
|
||||
|
||||
|
||||
def get_encoding_service() -> EncodingService:
|
||||
"""Get the singleton encoding service instance."""
|
||||
global _encoding_service
|
||||
if _encoding_service is None:
|
||||
_encoding_service = EncodingService()
|
||||
return _encoding_service
|
||||
@@ -1,539 +0,0 @@
|
||||
"""
|
||||
Firestore database operations for job management.
|
||||
"""
|
||||
import logging
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
from google.cloud import firestore
|
||||
from google.cloud.firestore_v1 import FieldFilter
|
||||
|
||||
from backend.config import settings
|
||||
from backend.models.job import Job, JobStatus, TimelineEvent
|
||||
from backend.models.worker_log import WorkerLogEntry
|
||||
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FirestoreService:
|
||||
"""Service for Firestore database operations."""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize Firestore client."""
|
||||
self.db = firestore.Client(project=settings.google_cloud_project)
|
||||
self.collection = settings.firestore_collection
|
||||
self.tokens_collection = "auth_tokens" # Collection for access tokens
|
||||
|
||||
def create_job(self, job: Job) -> None:
|
||||
"""Create a new job in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job.job_id)
|
||||
doc_ref.set(job.model_dump(mode='json'))
|
||||
logger.info(f"Created job {job.job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating job {job.job_id}: {e}")
|
||||
raise
|
||||
|
||||
def get_job(self, job_id: str) -> Optional[Job]:
|
||||
"""Get a job by ID."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc = doc_ref.get()
|
||||
|
||||
logger.debug(f"Fetching job {job_id} from collection {self.collection}")
|
||||
logger.debug(f"Document exists: {doc.exists}")
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
data = doc.to_dict()
|
||||
logger.debug(f"Document data keys: {list(data.keys()) if data else 'None'}")
|
||||
logger.debug(f"Document data: {data}")
|
||||
|
||||
return Job(**data)
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job(self, job_id: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update a job with partial data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Add updated_at timestamp
|
||||
updates['updated_at'] = datetime.utcnow()
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def update_job_status(
|
||||
self,
|
||||
job_id: str,
|
||||
status: JobStatus,
|
||||
progress: Optional[int] = None,
|
||||
message: Optional[str] = None,
|
||||
**additional_fields
|
||||
) -> None:
|
||||
"""Update job status and add timeline event."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
|
||||
# Create timeline event
|
||||
timeline_event = TimelineEvent(
|
||||
status=status.value,
|
||||
timestamp=datetime.utcnow().isoformat(),
|
||||
progress=progress,
|
||||
message=message
|
||||
)
|
||||
|
||||
# Prepare updates
|
||||
updates = {
|
||||
'status': status.value,
|
||||
'updated_at': datetime.utcnow(),
|
||||
'timeline': firestore.ArrayUnion([timeline_event.model_dump(mode='json')])
|
||||
}
|
||||
|
||||
if progress is not None:
|
||||
updates['progress'] = progress
|
||||
|
||||
# Add any additional fields
|
||||
updates.update(additional_fields)
|
||||
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated job {job_id} status to {status.value}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating job status {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def list_jobs(
|
||||
self,
|
||||
status: Optional[JobStatus] = None,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
created_after: Optional[datetime] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
user_email: Optional[str] = None,
|
||||
tenant_id: Optional[str] = None,
|
||||
limit: int = 100
|
||||
) -> List[Job]:
|
||||
"""
|
||||
List jobs with optional filters.
|
||||
|
||||
Args:
|
||||
status: Filter by job status
|
||||
environment: Filter by request_metadata.environment (test/production/development)
|
||||
client_id: Filter by request_metadata.client_id
|
||||
created_after: Filter jobs created after this datetime
|
||||
created_before: Filter jobs created before this datetime
|
||||
user_email: Filter by user_email (owner of the job)
|
||||
tenant_id: Filter by tenant_id (white-label portal scoping)
|
||||
limit: Maximum number of jobs to return
|
||||
|
||||
Returns:
|
||||
List of Job objects matching filters, ordered by created_at descending
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
# Filter by request_metadata fields using dot notation
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
# Filter by user_email (job owner)
|
||||
if user_email:
|
||||
query = query.where(filter=FieldFilter('user_email', '==', user_email.lower()))
|
||||
|
||||
# Filter by tenant_id (white-label portal scoping)
|
||||
if tenant_id:
|
||||
query = query.where(filter=FieldFilter('tenant_id', '==', tenant_id))
|
||||
|
||||
# Date range filters
|
||||
if created_after:
|
||||
query = query.where(filter=FieldFilter('created_at', '>=', created_after))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
query = query.order_by('created_at', direction=firestore.Query.DESCENDING).limit(limit)
|
||||
|
||||
docs = query.stream()
|
||||
jobs = [Job(**doc.to_dict()) for doc in docs]
|
||||
|
||||
return jobs
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing jobs: {e}")
|
||||
raise
|
||||
|
||||
def delete_jobs_by_filter(
|
||||
self,
|
||||
environment: Optional[str] = None,
|
||||
client_id: Optional[str] = None,
|
||||
status: Optional[JobStatus] = None,
|
||||
created_before: Optional[datetime] = None,
|
||||
) -> int:
|
||||
"""
|
||||
Delete multiple jobs matching filter criteria.
|
||||
|
||||
CAUTION: This is a destructive operation. Use carefully.
|
||||
|
||||
Args:
|
||||
environment: Delete jobs with this environment (e.g., "test")
|
||||
client_id: Delete jobs from this client
|
||||
status: Delete jobs with this status
|
||||
created_before: Delete jobs created before this datetime
|
||||
|
||||
Returns:
|
||||
Number of jobs deleted
|
||||
"""
|
||||
try:
|
||||
query = self.db.collection(self.collection)
|
||||
|
||||
if environment:
|
||||
query = query.where(filter=FieldFilter('request_metadata.environment', '==', environment))
|
||||
|
||||
if client_id:
|
||||
query = query.where(filter=FieldFilter('request_metadata.client_id', '==', client_id))
|
||||
|
||||
if status:
|
||||
query = query.where(filter=FieldFilter('status', '==', status.value))
|
||||
|
||||
if created_before:
|
||||
query = query.where(filter=FieldFilter('created_at', '<=', created_before))
|
||||
|
||||
# Get matching documents
|
||||
docs = list(query.stream())
|
||||
deleted_count = 0
|
||||
|
||||
# Delete in batches
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
batch_count += 1
|
||||
deleted_count += 1
|
||||
|
||||
# Firestore batch limit is 500
|
||||
if batch_count >= 500:
|
||||
batch.commit()
|
||||
batch = self.db.batch()
|
||||
batch_count = 0
|
||||
|
||||
# Commit any remaining deletes
|
||||
if batch_count > 0:
|
||||
batch.commit()
|
||||
|
||||
logger.info(f"Deleted {deleted_count} jobs matching filter criteria")
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting jobs by filter: {e}")
|
||||
raise
|
||||
|
||||
def delete_job(self, job_id: str) -> None:
|
||||
"""Delete a job from Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.delete()
|
||||
logger.info(f"Deleted job {job_id} from Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting job {job_id}: {e}")
|
||||
raise
|
||||
|
||||
def append_worker_log(self, job_id: str, log_entry: Dict[str, Any]) -> None:
|
||||
"""
|
||||
Atomically append a log entry to worker_logs using ArrayUnion.
|
||||
|
||||
This avoids the race condition of read-modify-write when multiple
|
||||
workers are logging concurrently.
|
||||
|
||||
DEPRECATED: Use append_log_to_subcollection() instead to avoid
|
||||
the 1MB document size limit.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: Log entry dict with timestamp, level, worker, message
|
||||
"""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.collection).document(job_id)
|
||||
doc_ref.update({
|
||||
'worker_logs': firestore.ArrayUnion([log_entry]),
|
||||
'updated_at': datetime.utcnow()
|
||||
})
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending worker log for job {job_id}: {e}")
|
||||
|
||||
# ============================================
|
||||
# Worker Log Subcollection Methods
|
||||
# ============================================
|
||||
# These methods store logs in a subcollection (jobs/{job_id}/logs)
|
||||
# instead of an embedded array to avoid the 1MB document size limit.
|
||||
|
||||
def append_log_to_subcollection(self, job_id: str, log_entry: WorkerLogEntry) -> None:
|
||||
"""
|
||||
Append a log entry to the logs subcollection.
|
||||
|
||||
Stores logs at: jobs/{job_id}/logs/{log_id}
|
||||
|
||||
This approach avoids the 1MB document size limit by storing each
|
||||
log entry as a separate document in a subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
log_entry: WorkerLogEntry instance
|
||||
"""
|
||||
try:
|
||||
# Ensure job_id is set on the log entry
|
||||
log_entry.job_id = job_id
|
||||
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Add document with auto-generated ID or use log_entry.id
|
||||
doc_ref = logs_ref.document(log_entry.id)
|
||||
doc_ref.set(log_entry.to_dict())
|
||||
|
||||
# Don't log every append - too spammy
|
||||
except Exception as e:
|
||||
# Log but don't raise - logging shouldn't break workers
|
||||
logger.debug(f"Error appending log to subcollection for job {job_id}: {e}")
|
||||
|
||||
def get_logs_from_subcollection(
|
||||
self,
|
||||
job_id: str,
|
||||
limit: int = 500,
|
||||
since_timestamp: Optional[datetime] = None,
|
||||
worker: Optional[str] = None,
|
||||
offset: int = 0
|
||||
) -> List[WorkerLogEntry]:
|
||||
"""
|
||||
Get log entries from the logs subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
limit: Maximum number of logs to return (default 500)
|
||||
since_timestamp: Return only logs after this timestamp
|
||||
worker: Filter by worker name (optional)
|
||||
offset: Number of logs to skip (for pagination)
|
||||
|
||||
Returns:
|
||||
List of WorkerLogEntry instances, ordered by timestamp ascending
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Build query
|
||||
query = logs_ref.order_by("timestamp", direction=firestore.Query.ASCENDING)
|
||||
|
||||
if since_timestamp:
|
||||
query = query.where(filter=FieldFilter("timestamp", ">", since_timestamp))
|
||||
|
||||
if worker:
|
||||
query = query.where(filter=FieldFilter("worker", "==", worker))
|
||||
|
||||
# Apply offset and limit
|
||||
if offset > 0:
|
||||
query = query.offset(offset)
|
||||
|
||||
query = query.limit(limit)
|
||||
|
||||
# Execute query
|
||||
docs = query.stream()
|
||||
logs = [WorkerLogEntry.from_dict(doc.to_dict()) for doc in docs]
|
||||
|
||||
return logs
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting logs from subcollection for job {job_id}: {e}")
|
||||
return []
|
||||
|
||||
def get_logs_count_from_subcollection(self, job_id: str) -> int:
|
||||
"""
|
||||
Get the total count of log entries in the subcollection.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
|
||||
Returns:
|
||||
Total count of log entries
|
||||
"""
|
||||
try:
|
||||
# Get subcollection reference
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
|
||||
# Use aggregation query for efficient counting
|
||||
count_query = logs_ref.count()
|
||||
result = count_query.get()
|
||||
|
||||
# Result is a list of AggregationResult, we want the first one's count
|
||||
if result and len(result) > 0:
|
||||
return result[0][0].value
|
||||
return 0
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error counting logs for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
def delete_logs_subcollection(self, job_id: str, batch_size: int = 500) -> int:
|
||||
"""
|
||||
Delete all log entries in the logs subcollection.
|
||||
|
||||
This is used when deleting a job to clean up its logs.
|
||||
|
||||
Args:
|
||||
job_id: Job ID
|
||||
batch_size: Number of documents to delete per batch
|
||||
|
||||
Returns:
|
||||
Number of logs deleted
|
||||
"""
|
||||
try:
|
||||
logs_ref = self.db.collection(self.collection).document(job_id).collection("logs")
|
||||
deleted_count = 0
|
||||
|
||||
while True:
|
||||
# Get a batch of documents
|
||||
docs = logs_ref.limit(batch_size).stream()
|
||||
deleted_in_batch = 0
|
||||
|
||||
# Delete in a batch
|
||||
batch = self.db.batch()
|
||||
for doc in docs:
|
||||
batch.delete(doc.reference)
|
||||
deleted_in_batch += 1
|
||||
|
||||
if deleted_in_batch == 0:
|
||||
break
|
||||
|
||||
batch.commit()
|
||||
deleted_count += deleted_in_batch
|
||||
|
||||
# If we deleted less than batch_size, we're done
|
||||
if deleted_in_batch < batch_size:
|
||||
break
|
||||
|
||||
if deleted_count > 0:
|
||||
logger.info(f"Deleted {deleted_count} logs for job {job_id}")
|
||||
|
||||
return deleted_count
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting logs subcollection for job {job_id}: {e}")
|
||||
return 0
|
||||
|
||||
# ============================================
|
||||
# Token Management Methods
|
||||
# ============================================
|
||||
|
||||
def create_token(self, token: str, token_data: Dict[str, Any]) -> None:
|
||||
"""Create a new access token in Firestore."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.set(token_data)
|
||||
logger.info(f"Created token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error creating token: {e}")
|
||||
raise
|
||||
|
||||
def get_token(self, token: str) -> Optional[Dict[str, Any]]:
|
||||
"""Get token data by token string."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc = doc_ref.get()
|
||||
|
||||
if not doc.exists:
|
||||
return None
|
||||
|
||||
return doc.to_dict()
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting token: {e}")
|
||||
return None
|
||||
|
||||
def update_token(self, token: str, updates: Dict[str, Any]) -> None:
|
||||
"""Update token data."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
doc_ref.update(updates)
|
||||
logger.info(f"Updated token in Firestore")
|
||||
except Exception as e:
|
||||
logger.error(f"Error updating token: {e}")
|
||||
raise
|
||||
|
||||
def increment_token_usage(self, token: str, job_id: str) -> None:
|
||||
"""Increment token usage count and add job to history."""
|
||||
try:
|
||||
doc_ref = self.db.collection(self.tokens_collection).document(token)
|
||||
|
||||
# Use Firestore transaction to ensure atomic increment
|
||||
@firestore.transactional
|
||||
def update_in_transaction(transaction, doc_ref):
|
||||
snapshot = doc_ref.get(transaction=transaction)
|
||||
if not snapshot.exists:
|
||||
raise ValueError("Token not found")
|
||||
|
||||
data = snapshot.to_dict()
|
||||
current_usage = data.get("usage_count", 0)
|
||||
jobs = data.get("jobs", [])
|
||||
|
||||
# Increment usage
|
||||
transaction.update(doc_ref, {
|
||||
"usage_count": current_usage + 1,
|
||||
"last_used": datetime.utcnow(),
|
||||
"jobs": firestore.ArrayUnion([{
|
||||
"job_id": job_id,
|
||||
"created_at": datetime.utcnow()
|
||||
}])
|
||||
})
|
||||
|
||||
transaction = self.db.transaction()
|
||||
update_in_transaction(transaction, doc_ref)
|
||||
|
||||
logger.info(f"Incremented token usage for job {job_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error incrementing token usage: {e}")
|
||||
raise
|
||||
|
||||
def list_tokens(self) -> List[Dict[str, Any]]:
|
||||
"""List all tokens (admin only)."""
|
||||
try:
|
||||
docs = self.db.collection(self.tokens_collection).stream()
|
||||
tokens = [doc.to_dict() for doc in docs]
|
||||
logger.info(f"Retrieved {len(tokens)} tokens from Firestore")
|
||||
return tokens
|
||||
except Exception as e:
|
||||
logger.error(f"Error listing tokens: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_firestore_client: Optional[firestore.Client] = None
|
||||
|
||||
|
||||
def get_firestore_client() -> firestore.Client:
|
||||
"""
|
||||
Get a shared Firestore client instance.
|
||||
|
||||
This returns a raw Firestore client (not the FirestoreService) for use
|
||||
in services that need direct Firestore access without the job-specific
|
||||
abstractions provided by FirestoreService.
|
||||
|
||||
Returns:
|
||||
Firestore client instance
|
||||
"""
|
||||
global _firestore_client
|
||||
if _firestore_client is None:
|
||||
_firestore_client = firestore.Client(project=settings.google_cloud_project)
|
||||
return _firestore_client
|
||||
|
||||
@@ -1,573 +0,0 @@
|
||||
"""
|
||||
Client for the remote flacfetch HTTP API service.
|
||||
|
||||
This client communicates with a dedicated flacfetch VM that handles:
|
||||
- BitTorrent downloads from private trackers (RED, OPS)
|
||||
- YouTube downloads
|
||||
- GCS uploads of downloaded files
|
||||
|
||||
The flacfetch service provides:
|
||||
- Full peer connectivity for torrents (not possible in Cloud Run)
|
||||
- Indefinite seeding of completed torrents
|
||||
- Automatic disk cleanup
|
||||
|
||||
Usage:
|
||||
client = get_flacfetch_client()
|
||||
if client:
|
||||
# Search for audio
|
||||
search_result = await client.search("ABBA", "Waterloo")
|
||||
|
||||
# Download the best result
|
||||
download_id = await client.download(
|
||||
search_id=search_result["search_id"],
|
||||
result_index=0,
|
||||
gcs_path="uploads/job123/audio/",
|
||||
)
|
||||
|
||||
# Wait for completion
|
||||
result = await client.wait_for_download(download_id)
|
||||
print(f"Downloaded to: {result['gcs_path']}")
|
||||
"""
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class FlacfetchServiceError(Exception):
|
||||
"""Error communicating with or returned from flacfetch service."""
|
||||
pass
|
||||
|
||||
|
||||
class FlacfetchClient:
|
||||
"""
|
||||
Client for remote flacfetch HTTP API.
|
||||
|
||||
All endpoints require authentication via X-API-Key header.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
api_key: str,
|
||||
timeout: int = 60,
|
||||
):
|
||||
"""
|
||||
Initialize flacfetch client.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of flacfetch service (e.g., http://10.0.0.5:8080)
|
||||
api_key: API key for authentication
|
||||
timeout: Default request timeout in seconds
|
||||
"""
|
||||
self.base_url = base_url.rstrip('/')
|
||||
self.api_key = api_key
|
||||
self.timeout = timeout
|
||||
logger.info(f"FlacfetchClient initialized with base_url={self.base_url}")
|
||||
|
||||
def _headers(self) -> Dict[str, str]:
|
||||
"""Get request headers with authentication."""
|
||||
return {
|
||||
"X-API-Key": self.api_key,
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
|
||||
async def health_check(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if flacfetch service is healthy.
|
||||
|
||||
Returns:
|
||||
Health status dict with transmission, disk, and provider info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: If service is unhealthy or unreachable
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/health",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
|
||||
if data.get("status") not in ["healthy", "degraded"]:
|
||||
raise FlacfetchServiceError(f"Service unhealthy: {data}")
|
||||
|
||||
return data
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cannot reach flacfetch service: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Health check failed: {e.response.status_code}")
|
||||
|
||||
async def search(
|
||||
self,
|
||||
artist: str,
|
||||
title: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Search for audio matching artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
Search response dict with search_id and results list
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On search failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=self.timeout,
|
||||
)
|
||||
|
||||
if resp.status_code == 404:
|
||||
# No results found - return empty results
|
||||
return {
|
||||
"search_id": None,
|
||||
"artist": artist,
|
||||
"title": title,
|
||||
"results": [],
|
||||
"results_count": 0,
|
||||
}
|
||||
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Search request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Search failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download(
|
||||
self,
|
||||
search_id: str,
|
||||
result_index: int,
|
||||
output_filename: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading an audio file.
|
||||
|
||||
Args:
|
||||
search_id: Search ID from previous search
|
||||
result_index: Index of result to download
|
||||
output_filename: Optional custom filename (without extension)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"search_id": search_id,
|
||||
"result_index": result_index,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download start failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def download_by_id(
|
||||
self,
|
||||
source_name: str,
|
||||
source_id: str,
|
||||
output_filename: Optional[str] = None,
|
||||
target_file: Optional[str] = None,
|
||||
download_url: Optional[str] = None,
|
||||
gcs_path: Optional[str] = None,
|
||||
) -> str:
|
||||
"""
|
||||
Start downloading directly by source ID (no prior search required).
|
||||
|
||||
This is useful when you have stored the source_id from a previous search
|
||||
and want to download later without re-searching.
|
||||
|
||||
Args:
|
||||
source_name: Provider name (RED, OPS, YouTube, Spotify)
|
||||
source_id: Source-specific ID (torrent ID, video ID, track ID)
|
||||
output_filename: Optional custom filename (without extension)
|
||||
target_file: For torrents, specific file to extract
|
||||
download_url: For YouTube/Spotify, direct URL (optional)
|
||||
gcs_path: GCS path for upload (e.g., "uploads/job123/audio/")
|
||||
|
||||
Returns:
|
||||
Download ID for tracking progress
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download start failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
payload = {
|
||||
"source_name": source_name,
|
||||
"source_id": source_id,
|
||||
}
|
||||
if output_filename:
|
||||
payload["output_filename"] = output_filename
|
||||
if target_file:
|
||||
payload["target_file"] = target_file
|
||||
if download_url:
|
||||
payload["download_url"] = download_url
|
||||
if gcs_path:
|
||||
payload["upload_to_gcs"] = True
|
||||
payload["gcs_path"] = gcs_path
|
||||
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/download-by-id",
|
||||
headers=self._headers(),
|
||||
json=payload,
|
||||
timeout=self.timeout,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
|
||||
data = resp.json()
|
||||
return data["download_id"]
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Download by ID failed: {e.response.status_code} - {e.response.text}")
|
||||
|
||||
async def get_download_status(self, download_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get the current status of a download.
|
||||
|
||||
Args:
|
||||
download_id: Download ID from start_download
|
||||
|
||||
Returns:
|
||||
Status dict with progress, speed, path info
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On status check failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/download/{download_id}/status",
|
||||
headers=self._headers(),
|
||||
timeout=10,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Status request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
if e.response.status_code == 404:
|
||||
raise FlacfetchServiceError(f"Download not found: {download_id}")
|
||||
raise FlacfetchServiceError(f"Status check failed: {e.response.status_code}")
|
||||
|
||||
async def wait_for_download(
|
||||
self,
|
||||
download_id: str,
|
||||
timeout: int = 600,
|
||||
poll_interval: float = 2.0,
|
||||
progress_callback: Optional[callable] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Wait for a download to complete.
|
||||
|
||||
Args:
|
||||
download_id: Download ID to wait for
|
||||
timeout: Maximum wait time in seconds
|
||||
poll_interval: Time between status checks
|
||||
progress_callback: Optional callback(status_dict) for progress updates
|
||||
|
||||
Returns:
|
||||
Final status dict with output_path or gcs_path
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On download failure or timeout
|
||||
"""
|
||||
elapsed = 0.0
|
||||
|
||||
while elapsed < timeout:
|
||||
status = await self.get_download_status(download_id)
|
||||
|
||||
if progress_callback:
|
||||
try:
|
||||
progress_callback(status)
|
||||
except Exception as e:
|
||||
logger.warning(f"Progress callback error: {e}")
|
||||
|
||||
download_status = status.get("status")
|
||||
|
||||
if download_status in ["complete", "seeding"]:
|
||||
logger.info(f"Download {download_id} complete: {status.get('gcs_path') or status.get('output_path')}")
|
||||
return status
|
||||
elif download_status == "failed":
|
||||
error = status.get("error", "Unknown error")
|
||||
raise FlacfetchServiceError(f"Download failed: {error}")
|
||||
elif download_status == "cancelled":
|
||||
raise FlacfetchServiceError("Download was cancelled")
|
||||
|
||||
# Log progress
|
||||
progress = status.get("progress", 0)
|
||||
speed = status.get("download_speed_kbps", 0)
|
||||
peers = status.get("peers", 0)
|
||||
logger.debug(
|
||||
f"Download {download_id}: {progress:.1f}% "
|
||||
f"({speed:.1f} KB/s, {peers} peers)"
|
||||
)
|
||||
|
||||
await asyncio.sleep(poll_interval)
|
||||
elapsed += poll_interval
|
||||
|
||||
raise FlacfetchServiceError(f"Download timed out after {timeout}s")
|
||||
|
||||
async def list_torrents(self) -> Dict[str, Any]:
|
||||
"""
|
||||
List all torrents in Transmission.
|
||||
|
||||
Returns:
|
||||
Torrent list response with torrent info and totals
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/torrents",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"List torrents failed: {e.response.status_code}")
|
||||
|
||||
async def delete_torrent(
|
||||
self,
|
||||
torrent_id: int,
|
||||
delete_data: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Delete a torrent from Transmission.
|
||||
|
||||
Args:
|
||||
torrent_id: Transmission torrent ID
|
||||
delete_data: Also delete downloaded files
|
||||
|
||||
Returns:
|
||||
Delete response
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/torrents/{torrent_id}",
|
||||
headers=self._headers(),
|
||||
params={"delete_data": str(delete_data).lower()},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Delete torrent failed: {e.response.status_code}")
|
||||
|
||||
async def cleanup_torrents(
|
||||
self,
|
||||
strategy: str = "oldest",
|
||||
target_free_gb: float = 10.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Trigger disk cleanup.
|
||||
|
||||
Args:
|
||||
strategy: Cleanup strategy (oldest, largest, lowest_ratio)
|
||||
target_free_gb: Target free space to achieve
|
||||
|
||||
Returns:
|
||||
Cleanup response with removed count and freed space
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.post(
|
||||
f"{self.base_url}/torrents/cleanup",
|
||||
headers=self._headers(),
|
||||
json={
|
||||
"strategy": strategy,
|
||||
"target_free_gb": target_free_gb,
|
||||
},
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(f"Cleanup failed: {e.response.status_code}")
|
||||
|
||||
# =========================================================================
|
||||
# Cache Management
|
||||
# =========================================================================
|
||||
|
||||
async def clear_search_cache(self, artist: str, title: str) -> bool:
|
||||
"""
|
||||
Clear a specific cached search result by artist and title.
|
||||
|
||||
Args:
|
||||
artist: Artist name
|
||||
title: Track title
|
||||
|
||||
Returns:
|
||||
True if a cache entry was deleted, False if no entry existed
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.request(
|
||||
"DELETE",
|
||||
f"{self.base_url}/cache/search",
|
||||
headers=self._headers(),
|
||||
json={"artist": artist, "title": title},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted = data.get("deleted", False)
|
||||
logger.info(
|
||||
f"Cleared flacfetch cache for '{artist}' - '{title}': "
|
||||
f"{'deleted' if deleted else 'no entry found'}"
|
||||
)
|
||||
return deleted
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear search cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear search cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def clear_all_cache(self) -> int:
|
||||
"""
|
||||
Clear all cached search results.
|
||||
|
||||
Returns:
|
||||
Number of cache entries deleted
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.delete(
|
||||
f"{self.base_url}/cache",
|
||||
headers=self._headers(),
|
||||
timeout=60,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
deleted_count = data.get("deleted_count", 0)
|
||||
logger.info(f"Cleared all flacfetch cache: {deleted_count} entries deleted")
|
||||
return deleted_count
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Clear all cache request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Clear all cache failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
async def get_cache_stats(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get statistics about the cache.
|
||||
|
||||
Returns:
|
||||
Dict with count, total_size_bytes, oldest_entry, newest_entry, configured
|
||||
|
||||
Raises:
|
||||
FlacfetchServiceError: On request failure
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient() as client:
|
||||
resp = await client.get(
|
||||
f"{self.base_url}/cache/stats",
|
||||
headers=self._headers(),
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json()
|
||||
|
||||
except httpx.RequestError as e:
|
||||
raise FlacfetchServiceError(f"Get cache stats request failed: {e}")
|
||||
except httpx.HTTPStatusError as e:
|
||||
raise FlacfetchServiceError(
|
||||
f"Get cache stats failed: {e.response.status_code} - {e.response.text}"
|
||||
)
|
||||
|
||||
|
||||
# Singleton client instance
|
||||
_client: Optional[FlacfetchClient] = None
|
||||
|
||||
|
||||
def get_flacfetch_client() -> Optional[FlacfetchClient]:
|
||||
"""
|
||||
Get the flacfetch client if configured.
|
||||
|
||||
Returns:
|
||||
FlacfetchClient instance if FLACFETCH_API_URL and FLACFETCH_API_KEY are set,
|
||||
None otherwise (indicating local-only mode).
|
||||
"""
|
||||
global _client
|
||||
|
||||
if _client is None:
|
||||
from backend.config import get_settings
|
||||
settings = get_settings()
|
||||
|
||||
if settings.flacfetch_api_url and settings.flacfetch_api_key:
|
||||
_client = FlacfetchClient(
|
||||
base_url=settings.flacfetch_api_url,
|
||||
api_key=settings.flacfetch_api_key,
|
||||
)
|
||||
logger.info("FlacfetchClient initialized for remote service")
|
||||
else:
|
||||
logger.debug("FlacfetchClient not configured (missing URL or API key)")
|
||||
|
||||
return _client
|
||||
|
||||
|
||||
def reset_flacfetch_client():
|
||||
"""Reset the singleton client (for testing)."""
|
||||
global _client
|
||||
_client = None
|
||||
|
||||
Loaded 100 of 1392 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user